r/programming • u/pimterry • Nov 10 '21
The Invisible JavaScript Backdoor
https://certitude.consulting/blog/en/invisible-backdoor/139
u/mindbleach Nov 10 '21
Banning unicode would be silly - but highlighting unicode would be just as easy. If you can detect it then you can flag it. Editors can already force the display of unprintable characters like whitespace and CR / LF. Just make it a warning, not an error.
A whitelist of non-confusing characters would avoid desensitizing people to that warning. No English speaker is going to see a variable named Einbahnstraße and think it's trying to pull a fast one. So you'd be free to throw an evil invisible character at the front of it. The double-S double-bluff.
56
u/darthwalsh Nov 10 '21
There's already been a lot of security work going into Unicode characters in URL hostnames that are pixel-for-pixel matches for ASCII characters, like some eastern european "e" that's not an e allowing for phishing at google.com.
Throwing up a big warning for invisible characters seems trivial in comparison.
1
u/Celestial_Blu3 Nov 12 '21
Why are those pixel-for-pixel identical characters even allowed?
2
u/darthwalsh Nov 12 '21
Imagine you're from eastlandia and you want to put the name of your school in your website domain. Would be pretty obnoxious if you could put most of your Unicode character alphabet into the name, except for one vowel which happens to match up with English...
But you're right, I think the result of the security fixes was to not allow the mixing of lookalike characters with English characters. Works great unless you find out you can spell out a-p-p-l-e completely with lookalikes...
5
Nov 11 '21
No English speaker is going to see a variable named Einbahnstraße and think it's trying to pull a fast one.
I would ask why the programmer wouldnt just use ss for esset
6
3
u/mindbleach Nov 12 '21
Because that's how it's fucking spelled.
Why did you write "programmer" when the Hawaiian alphabet has no R?
1
u/bloody-albatross Nov 12 '21
I would ask why the programmer isn't just using
oneway? Less to type, too.1
u/XeonProductions Nov 12 '21
Perhaps banning invisible unicode characters, or adding detection that will flag certain use case scenarios for further review.
-83
u/PL_Design Nov 10 '21 edited Nov 10 '21
Banning unicode is not silly. Unicode is dreadful, and most programs will never be translated. 99% of the time it is literally pointless and people would be better served by using local character encodings.
EDIT: Isn't it interesting how saying you dislike unicode causes everyone to dogpile you? It feels like all of you have been brainwashed. It is startlingly creepy. I suggest you freaks go to therapy.
51
u/CartmansEvilTwin Nov 10 '21
No. We had that already with all those ISO encodings and it's hell.
What is the local encoding for Germany for example? We have our own Umlaut-characters, but what if some spaniard called Piñera wants to live here? And what about André, Çem, etc.?
So you end up with an encoding that looks almost identical to Unicode/UTF-8 anyway.
7
u/naasking Nov 11 '21
What is the local encoding for Germany for example? We have our own Umlaut-characters, but what if some spaniard called Piñera wants to live here? And what about André, Çem, etc.?
There's a middle ground here: only permit full Unicode between a programming language's string delimiters, ie. typically between two " characters, and the rest of the grammar must use only printable ASCII characters. This takes care of all input/output issues like the example you mention, without introducing homoglyph and invisible character vulnerabilities into a language's grammar.
9
u/auxiliary-character Nov 11 '21
This takes care of all input/output issues like the example you mention
Except for when you want to credit a programmer named Piñera in a comment, since comments exist outside string delimiters.
→ More replies (33)1
u/marinuso Nov 11 '21
Code isn't the same as data. You can have Mr. Piñera living on the Einbahnstraße but you name the columns
lastnameandstreet. (In English, because code should be written in English anyway.)It's perfectly sane to restrict identifiers to ASCII, or preferably even a subset of that. Even APL of all languages restricts identifiers to letters, numbers, and a handful of whitelisted punctuation characters.
(Of course you shouldn't ban Unicode entirely.)
30
u/mindbleach Nov 10 '21
In which the programming subreddit tries to solve the underhanded C competition by saying a compiler should shit the bed if you add Tools > Preferences > Language > 日本語.
And if I try to copy-paste code from a StackOverflow user in Russia, I guess I can go fuck myself.
→ More replies (17)21
u/ClassicPart Nov 10 '21
99% of the time it is literally pointless
Sit down for this one, but it might shock you to learn that there are other countries on this planet. It's "literally pointless" for you. Get it right.
→ More replies (1)13
u/DethRaid Nov 11 '21
Isn't it interesting that you have a bad idea and everyone is downvoting that because it's a bad idea?
→ More replies (5)7
u/scratchisthebest Nov 11 '21
i agree also everyone on the planet should speak english. i am very smart. i love to use "code pages"
→ More replies (1)3
u/wankthisway Nov 11 '21
saying you dislike unicode
is not the same as you actually saying
Unicode is dreadful,
Less victim mentality, please.
1
→ More replies (4)1
u/Sag0Sag0 Nov 11 '21
Yes, you are right. This is just one big conspiracy by big Unicode.
→ More replies (3)
46
u/f0rtytw0 Nov 10 '21
7
u/tjpalmer Nov 11 '21
Yeah, the topic has almost nothing to do with js specifically.
6
u/f0rtytw0 Nov 11 '21
Yeah, the take away is don't trust your eyes when visually inspecting something that can use unicode.
207
u/KaiAusBerlin Nov 10 '21
eval(myWholeBundledProjectCode.replaceAll(hackingChars, ''))
wait 1 hour and there will be an npm package for that
/s
64
u/Zaphoidx Nov 10 '21
I do wonder how Github and other online repositories deal with this sort of stuff.
Do they render the character normally, or do they special-case it to ensure that stuff like this doesn't slip through?
Never come across it myself in the wild so have no clue.
67
u/MathWizz94 Nov 10 '21
One of the links in the article leads to a Gist with hidden characters that GitHub shows a warning about: https://gist.github.com/jupenur/f4c10dce1b2824cd1273f6b518fd968b
25
u/FVMAzalea Nov 10 '21
The warnings are new after the Cambridge researchers released the CVE a couple weeks ago.
30
u/StabbyPants Nov 10 '21
wait 2 hours and it will also mine btc and send the proceeds to some .ru address
3
u/auxiliary-character Nov 11 '21
Or you could use a git hook to do it instead of doing the check at runtime like a maniac
100
u/chalks777 Nov 10 '21 edited Nov 10 '21
Very cool exploit and I like the idea. Ideally this should be caught at least two ways:
1. Lint would almost certainly catch this. In particular this should give an error for improper formatting:
const checkCommands = [
'ping -c 1 google.com',
'curl -s http://example.com/',ㅤ\u3164
];because (based on the patterns in this example) it should be:
const checkCommands = [
'ping -c 1 google.com',
'curl -s http://example.com/',ㅤ
\u3164,
];and if(environmentǃ=ENV_PROD){ violates no-cond-assign
2. PR review. Yes, it's hard to see visually, but the cardinal sin here is putting ANY user input into exec. That's insane.
42
u/Wacov Nov 10 '21
the cardinal sin here is putting ANY user input into exec. That's insane.
You mean the timeout? Without the hidden var the checkCommands array doesn't contain user input
11
u/chalks777 Nov 10 '21
You mean the timeout?
Yes. Granted, it's almost certainly fine to put a timeout direct from
req.queryin the call from a security/exploit standpoint (see documentation). I would definitely object to anybody doing that normally because it's a really bad habit to get into, even in this case. I would hope that when scrutinized a little harder you would find something weird going on.I wouldn't expect a normal reviewer to actually notice the
\u3164though without the help of some automated tool.9
u/kenman Nov 10 '21
Granted, it's almost certainly fine to put a timeout direct from
req.queryin the call from a security/exploit standpoint (see documentation).Are you speaking only to the injection vector? Because setting a timeout of
0(or some exceptionally high value), coupled with a massive number of requests, would create a self-inflicted DoS. The code should at least provide a window of acceptable values.4
4
u/Fatalist_m Nov 11 '21 edited Nov 11 '21
It does not put
timeoutdirectly into exec though, "+timeout || 5_000" will always return a number. You could add range checks or any other checks but the exploit would be just as hard to notice.1
u/Phobos15 Nov 12 '21
I wouldn't expect a normal reviewer to actually notice the \u3164 though without the help of some automated tool.
I think most people who care about formatting would have noticed it 99% of the time and asked for the trailing "," and space to be removed. But stuff can slip through. So you still want other checks.
I noticed it before reading what the article was about. Granted, I think most people would have only realized it was a variable and not a formatting mistake if they had been reading about the browser url exploits and if they commonly use lint which has forced them to care about trailing commas.
39
u/buncle Nov 10 '21
but the cardinal sin here is putting ANY user input into exec.
I think the clever part of this exploit is that it appears, at first glance, that there isn’t any user input going I to exec (it would look like
cmdis a fixed array).Definitely pretty clever.
I would say this is an issue that lays with the editors, more than anything else. Allowing invisible Unicode to sit within an open source file is unpleasant for a number of reasons (not just exploits, but making it hard to locate copy/paste errors). I think the obvious answer here would be for IDEs to make ‘invisible’ characters visible while editing.
6
u/chalks777 Nov 10 '21
Agreed completely. My only point with the
execis that it might get more attention in a PR review because it's putting user input (timeout) directly into the function call options.4
u/ShinyHappyREM Nov 10 '21
I would say this is an issue that lays with the editors, more than anything else
Or it's languages that allow non-ASCII characters outside of strings and comments...
5
u/buncle Nov 10 '21
I think Unicode should be acceptable, for non-English speaking coders, but going down this route would require a specific subset of Unicode (which could be a can of worms, and add complexity to the language).
It’s hard to say what the ideal solution here would be, but I agree that ideally invisible characters should not be parsed by the language outside of strings/comments at all (or should throw an error).
8
u/ShinyHappyREM Nov 10 '21
I think Unicode should be acceptable, for non-English speaking coders
Even as a non-native speaker I have to say it'd be effectively useless.
Have you ever tried to read code with identifiers in a language you didn't understand? It may as well be obfuscated. Adding non-latin characters would make matters even worse.
1
u/buncle Nov 10 '21
That’s fair. It’s not something I have any personal experience with, so my thought/opinions on non-Latin chars at a language level have zero weight.
1
u/Programmdude Nov 11 '21
In some countries (india, china and likely japan) come to mind, using english identifiers would also be like reading obfuscated code. If the software company is entirely local to that country, not all the employees will be able to speak english with any degree of proficiency.
I still think ascii should be used for identifiers instead of unicode, china can use pinyin and japan can use romaji.
5
u/SureFudge Nov 10 '21
but the cardinal sin here is putting ANY user input into exec. That's insane.
Came here to say this. Don't use exec, eval and the likes ever.
3
u/Doctor_McKay Nov 11 '21
exec is completely different from eval. Sometimes you need to invoke an external process.
3
u/Magzter Nov 11 '21
Regarding point 2 it's not really the cardinal sin here. The point is it's a backdoor, even if timeout was sanitised and mapped to a range of acceptable values before being passed to exec, the backdoor still exists.
2
u/ubernostrum Nov 11 '21
This is a thing you already had to be watching out for if you were doing stuff like user signups; people can do bad things in usernames if you let them.
22
u/ambirdsall Nov 10 '21
If the invisible variable definition were formatted like
const {
timeout, ㅤ
}then the whole thing would be visually indistinguishable from ordinary code using trailing commas style.
15
-1
u/kenman Nov 10 '21
I've always hated the
comma-danglerule anyways.25
u/chalks777 Nov 10 '21
I like using the
always-multilineoption.Valid:
{ foo, bar, baz }{ foo, bar, baz, }Invalid:
{ foo, bar, baz, }{ foo, bar, baz }3
31
2
u/Kwantuum Nov 11 '21
and now any time you add a line at the end of an object, you get two lines of diff instead of one.
1
u/p4y Nov 11 '21
If somebody else adds a line to the same object, you get a merge conflict and your editor wants to solve it by duplicating the line where you added a comma
9
u/ProgramTheWorld Nov 10 '21
There are more tricks with Unicode like flipping arguments order with the writing direction characters. Fun stuff.
57
u/theoldboy Nov 10 '21
Obviously I'm very biased as an English speaker, but allowing arbitrary Unicode in source code by default (especially in identifiers) just causes too many problems these days. It'd be a lot safer if the default was to allow only the ASCII code points and you had to explicitly enable anything else.
24
u/lood9phee2Ri Nov 10 '21 edited Nov 10 '21
well, indeed arbitrary unicode as bare identifiers may be questionable I suppose?
Even if desired to write source code identifiers in a different writing system for whatever social/cultural/political/ideological/plain-contrariness-and-obfuscation reasons, you could perhaps just allow a different subset of unicode, yet one that's that's still small and not too ambiguous like ascii.
e.g. like that corresponding to russian koi8r (cyrillic, for glorious motherland comrade), i.s.434:1999 (coding in something normally written two thousand years ago on large rocks is the sort of thing the irish would do because it's funny), or whatever.
I'm not saying actually use the old national encodings, just it would be possible to limit identifiers in given compilation units to being from particular subsets of unicode that are kind of like the old 8-bit national encodings in the grammar, i.e. there is a medium between "ascii ...that actually doesn't even work fully for most european languages arguably including proper english though we're used to that" and "arbitrary unicode" that is "non-arbitrary unicode limited in various ways, perhaps to subsets corresponding to particular scripts".
At interface boundaries you could allow controlled importation i.e. identifiers outside the subset have to be explicitly imported (so that your delightfully incomprehensible all-ogham codebase can still link against stdlib) - because it would all be still unicode and not actually national 8-bity encodings, that would still work.
9
u/MrJohz Nov 10 '21
I think browsers have come up with a reasonable solution for URLs — you can use characters from certain character sets, but you've got to remain in the same character set in the same URL. For example, you can use as many Unicode characters as you like based on the Latin alphabet (accents, digraphs, etc), but if you combine a character from the Latin alphabet with one from the Cyrillic alphabet, you'll get an error (or at least for most browsers, the "raw" punycode representation will be shown). There are a bunch of other rules that help here, such as banning invisible characters, banning a list of known dangerous characters, etc.
I think these sorts of rules are probably a bit restrictive for defining identifier rules, particularly because subtle changes in these rules can have big effects on whether a program is valid or not. However, as linting rules (ideally ones that block builds by default), they would work very well. I know that the Rust compiler does a lot of this sort of stuff — if there are confusables in identifiers, or the "trojan source" characters mentioned at the top of this article — and by default prevents the code from compiling (although this is only a lint, and therefore can be disabled manually if desired).
Unfortunately, there's not much standardised in the JavaScript ecosystem, but I do think developer tools like ESLint and editors/code viewers like GitHub should be showing these sorts of warnings by default.
2
u/StabbyPants Nov 10 '21
what about having modules declare their codepoints? so, if you want to name a variable кофи, you declare your module as using cyrillic, the linter allows ansi + cyrillic, and your dep mgmt rolls up a list of all subsets currently declared. so, if your footprint is russian, euro, ascii, fine. if it's got akkadian in it, be suspicious
11
u/mindbleach Nov 10 '21
Anything unusual should be highlighted and warned about. That's sufficient.
It's extensible to other spoken languages - someone editing in Japan can expect to see ASCII alongside all three of their native alphabets, but Hangul would still be kinda weird. It should show up as a unicode error block � in addition to having its intended effect. Like how missing stuff in video games tends to show up as giant glowing checkerboards: you can't miss it. Making anything unexpected, visible, lets you reason about what the fuck it's doing, and what the fuck it's doing in your code.
And if it causes headaches for anyone using emoji in their Javascript... good.
4
u/1337Gandalf Nov 10 '21
C and C++ got that right.
13
u/theoldboy Nov 10 '21 edited Nov 11 '21
C and C++ don't allow Unicode in identifiers, which stops many obvious exploits, but most compilers do allow it elsewhere (in literal strings and comments).That can be exploited too.EDIT I'm wrong. it's implementation-defined I think but gcc and clang do allow Unicode identifiers for both C and C++.
2
Nov 11 '21
That doesn't fool the compiler or even the editor syntax highlighting:
2
u/theoldboy Nov 11 '21
Works for me with the examples from https://github.com/nickboucher/trojan-source
trojan-source/C/commenting-out.c
trojan-source/C++/commenting-out.cpp
Yes, the syntax highlighting isn't fooled. Not sure what Godbolt is using for that but many editors have been patched since that paper was published.
1
Nov 11 '21
That is good to know, the version that can be compiled no longer looks deceiving in editors like Notepad++ or MSVC, and the code that still looks deceiving doesn't compile.
1
Nov 11 '21
Not sure about that:
https://godbolt.org/z/fxb34jE8G1
u/1337Gandalf Nov 11 '21
tbf, it's pretty obvious that something is up.
for (unsigned ㅤ, ㅤㅤ, ㅤㅤㅤ; std::cin >> ㅤ >> ㅤㅤ >> ㅤㅤㅤ;) {2
4
u/mcilrain Nov 10 '21
Isn’t that how Python does it? You need to specify encoding at the top of the file or it’s ASCII or Latin-1 or something by default.
6
15
Nov 10 '21
Strongly disagree, comments should be in the language of the programmers and those who will read the code. Most people you are going to see on reddit already speak English well, so they are obviously not going to be bothered by English only.
Because banning non ascii-characters basically means that, denying people the ability to write code in their language.
2
Nov 10 '21
English is the language of international collaboration. You're effectively stopping your code from scaling out by not writing it in English.
17
Nov 10 '21
Yes and ? The website I built for a French political party is not going to scale to millions of users in a grand display of international collaboration. It's going to be read and maintained by three blokes who all speak French.
4
u/exploding_cat_wizard Nov 11 '21
And if they attempt to use French in the syntax, it will be harder to maintain than if they sensibly restrict themselves to using French strings and comments.
There are no reasons for a language to allow non-ASCII identifiers and keywords, a charset every language on earth has an official transliteration to, that trump programmers easily seeing what exactly was written.
2
Nov 11 '21
Still a PITA. Hopefully all of them will use the same encoding, otherwise it will be a lot of fun fixing bugs!
4
u/vytah Nov 10 '21
Most code is never going to scale out, so writing comments and user-facing string literals in a language that represents the problem domain accurately is the way to go.
1
u/exploding_cat_wizard Nov 11 '21
That doesn't require the language to stupidly allow UTF-8 encoding in it's syntax. Just in strings and comments, which is where it belongs.
1
u/Shautieh Nov 13 '21
How are you supposed to name your fucking variables? Of course the coder will use the language he knows.
1
u/exploding_cat_wizard Nov 13 '21
And you don't need extra characters to name them, since every language has a transliteration to Latin.
-1
u/blobjim Nov 11 '21
It's the language of "we invaded your country and imposed our language on you, now we'll impose it again in computer source code!"
9
u/MrSqueezles Nov 10 '21
I understand wanting to code in a native language. We don't expect the entire world population to learn English. I'm no expert, but based on the description, it may be the "!" used in the second example is for commonly used multi-directional languages that require extra clearance on either side of punctuation. Maybe the correct restriction is "Unicode word characters only".
13
u/nitrohigito Nov 10 '21 edited Nov 10 '21
The only time people use the native language here for code is when teaching/studying, or for crappy single-use code nobody else will probably read. It's a tremendous red flag.
It's a bit like Latin used to be. It's sad, annoying, but you really just gotta put up with it, cause it's a numbers game, and boy are we outweighed.
It also doesn't help that the syntax of virtually every programming language I've encountered so far simply meshes unwell with the grammar of the native natural language here, so even for identifiers, it's sometimes just not the greatest.
7
u/wasdninja Nov 10 '21 edited Nov 11 '21
We don't expect the entire world population to learn English
We pretty much do if they want to become programmers. The official documentation of many things are in English only as far as I can tell. Not to mention that the programming languages themselves are literally in English.
1
u/Shautieh Nov 13 '21
That's far from true. Many docs are available in multiple languages, and when they aren't there are unofficial docs which are. It's hard enough to learn to program, English doesn't have to be a part of it.
1
u/blobjim Nov 11 '21
That should probably change.
4
u/wasdninja Nov 11 '21
Programming languages should definitely not be translated. That is really dumb. Having documentation in more languages would be good but documentation is hard enough as it is to keep up with in a single language.
Anyone who doesn't know English is going to have a very rough time learning programming for the foreseeable future.
3
u/bloody-albatross Nov 12 '21
Programming languages should definitely not be translated. That is really dumb.
It is. It is also what Excel and other spreadsheet software already does! And it causes problems when in the German version of Excel a decimal number uses comma instead of the decimal point and then some badly hand crafted VBA script creates invalid CSV files or SQL queries or similar.
28
u/AttackOfTheThumbs Nov 10 '21
As a German, no, everyone should code in English. Coding in other languages is stupid. The field is English and as such, everyone should adjust to it.
23
u/kaashif-h Nov 10 '21
Having had to read a codebase where Indian programmers had used Hindi naming conventions or something...I agree.
11
u/QuotheFan Nov 10 '21
That would have been hilarious!
kaksha pustak {
junta:
pustak(); sankhya prishtha_sankhya; vakya lekhak;};
Comments be like:
// mujhe nahi pata yeh code kyun kaam karta hai. Likhne waala ya toh bhagwaan tha ya chutiya.. :P
7
u/eattherichnow Nov 10 '21
kaksha pustak
Pole reading the above: not english? WTF. Immediately correct:
porridge brick7
u/AttackOfTheThumbs Nov 10 '21
I have read German code, dutch, danish, and others I didn't recognize. It's just a silly thing to do, and entirely pointless.
3
11
u/CartmansEvilTwin Nov 10 '21
And yet, many organisations use tons of native language comments, business lingo or interface definitions.
A good example I encountered a few years ago is Schufa. Their entire interface is German XML.
6
u/AttackOfTheThumbs Nov 10 '21
And yet, many organisations use tons of native language comments, business lingo or interface definitions.
Not everyone can make the right decisions all the time. Comments in code I'm pretty ambivalent to myself. The other too are bad. It would be interesting to see when they decided to use the native tongue.
I work with ERP systems. I have seen a mix of many languages, and in general, when it's not in English, the business ends up losing, because the support becomes more costly. Most of the time I found they made that decisions x years/decades ago and it has been carried forward ever since. Sometimes they end up deciding to transition, other times they start mixing.
I think Schufa is probably big enough to get away with it, but that doesn't mean it was smart. I kind of assume they don't expand past the German speaking space, but I don't even know, since I've never worked with them directly.
It's all based on personal experience anyway. I would just say it's typically bad when things other than English are used.
3
Nov 10 '21
I'm a native Spanish speaker, fan of foreign languages. I definitely prefer to code in English.
Although I created once a toy language with Spanish keywords
2
u/DrayanoX Nov 11 '21
That's easy for us to say when we are already fluent in English. The majority of the world population isn't, or do have some rudimentary English knowledge but aren't comfortable or good enough to use it.
There's no reason to prevent anyone who doesn't speak English from getting into programming this is elitism at its finest.
Exploits can easily be prevented by just blocking specifically confusing and invisible characters from being used. There's no reason why characters such as "ß ç ñ ē ب" cannot be used by people who speak such languages using these.
Blocking all of Unicode is like cutting off your entire leg because you stepped on a Lego.
0
0
u/Shautieh Nov 13 '21
As a German you got no say in this for two reasons : 1 English is easy to learn for you so of course you don't care about others troubles 2 your parents had no other options than to accept that the USA were superior. That's not the case everywhere
1
u/AttackOfTheThumbs Nov 14 '21
This comment is on a whole different level of stupid. Thank you for making me laugh with your idiocy.
USA is third world.
3
u/vytah Nov 10 '21
it may be the "!" used in the second example is for commonly used multi-directional languages that require extra clearance on either side of punctuation
No, it's a letter, U+01C3. But since it's used only in minority languages in Namibia and RSA, like ǃKung, ǃXóõ or Khoekhoe, it's very unlikely to appear in code (in either code proper, comments, or literals) at all.
8
u/AttackOfTheThumbs Nov 10 '21
No, you are correct. Programming should only use a default ascii set. Anything else is stupid. Limit the tools to limit the exploits. There's zero issue with this.
4
u/ThirdEncounter Nov 10 '21
I'll have agree with /u/beached on this one. Telling about 80% of the population who speaks a language other than English "use ascii, because anything else is stupid" is, well, misinformed.
Let's reverse the roles, and say that the "one true character set" is "Japanese ascii" (kanji-scii?) Now you can't use variables such as "loopCounter" because it's not kanji-scii. You have to use ループカウンター because "using loopCounter is stupid."
There's gotta be a way to mitigate the risks, I agree. But "ascii only!" is not it. This is not the 70s anymore.
2
u/Shautieh Nov 13 '21
Exactly. Redditors are so backwards about that. I'm fluent in English but we can't expect people to open a dictionary every time they need to write and read a variable.
→ More replies (1)1
u/exploding_cat_wizard Nov 11 '21
The programming language already forces the use of English, your example doesn't make sense. It's "static public void", not whatever the kanji version of that would be, in Java, and similarly in every language that's actually used in prod.
If these Japanese speakers so beset upon that JavaScript has an English syntax invent their own JapanScript that uses only kanjis, that wouldn't be a problem ( except for whomever thought that would be a good idea, but I'm not one to forbid you to take on whatever problem you want to make for yourself ). It means nobody outside of Japan will be able to use it, and these people will severely limit their community, but at least the whole rest of the world won't have to fight an entirely new sneaky class of bugs because making programming even more complicated is the cool thing to do.
And it's not like anyone outside Japanese readers can even help you with your JavaScript written in kanji, so the actual advantage for you, the UTF-8-kanji-JS writer, is minimal compared to just using kanji-script from the get go.
3
u/DrayanoX Nov 11 '21
The number of programming keywords is limited, it's easy for a non-english speaker to learn them by heart.
Expecting him to learn the entire English language just so he can write code is stupid.
1
u/exploding_cat_wizard Nov 11 '21
That's not at all what anyone here said, wherever did you get that from? You can write any language on this planet in the lingua franca of scripts, Latin. No need to learn English, just use ASCII to write in your language. Less problems for everyone involved, and if you really can't, make your own programming language and at least be explicit that you're doing your own thing, instead of pretending it could be part of a worldwide ecosystem.
3
u/DrayanoX Nov 11 '21
ASCII doesn't allow billions of people to write their native scripts. Russian, Chinese, Japanese, Arabic and many other scripts can't be written in ASCII.
It's unreasonable to expect someone to learn the latin script just so he could name his variables and write his comments.
It's easy enough to learn specific keywords such as const, float, function and class. It's a whole different game to learn enough of a latin language just to get started with programming. We shouldn't be advocating for more barriers to get into programming.
1
u/exploding_cat_wizard Nov 11 '21
It's a whole different game to learn enough of a latin language just to get started with programming.
Nobody needs to learn a Latin language, except those words you already conceded.
And again, if you want to create, e.g., Hindi script, a JS clone that uses Hindi characters, go ahead. Explicit is better than implicit, so admit that you're using a different programming language and stop pretending that you're part of the same programming language community when you are taking yourself out of international conversations by using local scripts. Despite being an optimal solution for everyone involved, and massively reducing actual barriers to programming like programmers not actually being able see what code is actually written thanks to UTF character fun, this option isn't ever really adopted, because it shows clearly why mixing scripts is a bad idea.
2
u/DrayanoX Nov 11 '21
Why the fuck would someone need to create a completely new language for this ? Programming languages are tools used to make the computer do stuff, nothing dictates the way someone chose to use these tools to write his own programs. No one is going to reinvent JS or Python just so he can write comments or name variables in local scripts.
Why should a group of local developers write comments in broken English/[Insert any language written in ASCII] to document their code instead of whatever language they are most familiar with.
stop pretending that you're part of the same programming language community when you are taking yourself out of international conversations by using local scripts.
No one is "pretending" anything, not everything has to revolve around the English language, at the end of the day, people just want to write code that make their computers do stuff, no one should be expected to learn a whole new script just to get started with programming, that's ridiculous.
Why are you expecting that you should be able to read their code and understand it ? Do you too get mad when you come across a book written in Mandarin or something and expect it to be written with Latin characters ?
And just in Europe you have people using languages derived from latin that have characters not available in ASCII such as à ê ç and many others, how do you expect to handle cases like first and last names written with some of these if you aren't allowed to use anything other than ASCII in your code, and that's just to give a basic example.
The solution to this problem isn't to nuke Unicode from programming, blacklisting confusing and invisible characters is easy enough without having to remove every other non-ASCII character.
→ More replies (3)1
u/Shautieh Nov 13 '21
Naming variables is one of the most fundamental work a coder does, and you can't expect non English speakers to use a dictionary every time they want to read of write a variable.
1
u/exploding_cat_wizard Nov 13 '21
My comment literally says you can and should use your language, please take the time to read it instead of jumping to conclusions and arguing strawmen
2
Nov 10 '21
Another advantage of this would be a bit of compile time or runtime performance depending on language, because comparing ascii strings is probably faster than utf8 or utf16 strings when linking identifiers.
2
u/vytah Nov 10 '21
because comparing ascii strings is probably faster than utf8 or utf16 strings when linking identifiers.
Normalization is not performed, it's just matching opaque bytestrings, so the speed is the same.
One could argue that for better speed, you should name everything in Chinese, as it's denser than English.
1
1
1
u/nerd4code Nov 10 '21
IMO it’s potentially still useful to embed Unicode text in a program for various purposes like templating, NLS, or use of fancy punctuators, operators, and symbols, it should be enabled implicitly only for comments, and explicitly for quoted §s where it’s needed, with stringent limits on layout (no mirroring, no full-line RTL, no embedding controls other than RLE, LRE, and PDF) should be permitted in those contexts.
The rest of the code can still be coded as UTF-8, but anything outside the wossis, G0? range I think it’s called? should trigger an error—so U+0020…U+007E’d be permitted, plus C0 ctrls HT, LF, VT, FF, CR as syntactic markers outside quoted regions, maybe +LSEP, PSEP, maybe +(C1) NEL, maaayyybe +(C0) NUL (as 00 or C0,80) and DEL for chars to ignore entirely. Unicode’d potentially still cause problems where permitted, but at least the scope would be bounded and relatively easy to scan for, sorta like an
unsaferegion.0
Nov 10 '21
What makes you think that ASCII would be the one true set of codepoints? Just because it was that way, doesn't mean it would have to continue. We live in a world with many more languages than English and English is not the dominant written or spoken language. Also, we have tools for this already.
2
u/AttackOfTheThumbs Nov 10 '21
English is the dominant language for software development.
1
Nov 10 '21
You should look at the source code for a tonne of device drivers. I've had to use google translate when looking through source code to get a better understanding. But, any move from unicode will result in an bunch of new non-english languages/forks. It will be worse for our perceived comforting warm blanket where everyone speaks what we speak. As I said, there are tools out there now to normalize text and it's the IDE's/language/tool writers that need to update and only accept the normalize forms and to stop homoglyph attacks.
There is also http://www.unicode.org/reports/tr31/
0
u/danweber Nov 10 '21
Yes. If you need other languages, fine, all your user-displayable strings are in a separate file, and treated as hostile.
5
u/jazd Nov 10 '21
You think English speakers don't use Unicode characters?
23
u/emperor000 Nov 10 '21
For identifiers? If you are using Unicode characters for identifiers then that's probably a problem.
34
u/balefrost Nov 10 '21
p̵̛̪̺̟̫̂̒͛͗̌̒̈́͐͂̿͒͝͝͝ḛ̷̩̮̣̭̠͎̪̩̂̏͒̿̇̊̍̆͑̋͠͝ͅř̴̡̛̏f̷͓̬̆̽̀͐̆͛͗̃̑͠͝ẹ̴̜̙͚̬̮̜̙͙͇̪̾͋͊c̶̝̣̖̼̆̔͛̎̈͆͊̊͆̕ṫ̸̨̢̯͈͔̩̤̌͗l̴̥̬̝̥̆͠ý̸͍̿̎̈́͌̃͐̉͐͋̇̾̚N̸͙͔͍̠̜̺͎̩̩̳̝̲͗̍͒̒́̄̇̎̚͜ǫ̶̡̨͙͕͈̞̝̺̦̠͙̲̩̯̅͗̐̿̏̉̄̑̇̉͘r̴̡̢̘̱͖̘̪̝̭̪̦͈̆͑͒̆̾͑̉͊̕̕̕ͅͅm̵̧̯͕̯͙̣̹̪̱͖̠̬͔̩̪̀̔̓ä̴͚ļ̸̧͕̙͖̳͖͚̣̭͕͐͗͑ͅV̷̡̢͔͍̻͚̭̘̖̦͍̠̖̝́́̋̑̋ͅa̶̰̙̝̦̗͚̯̠̞̭̎̓̋r̸̛͓͍͍͙̟̼̬̮̫̩͎̗̯̩͗̑͋́́̊͝i̶̡̩̤̜͉̻̟̹̙̗̱͆̑̉́͐̂͊̍ͅȁ̴̟b̷̧̙̙̞̥́̄̊̊̿̀̈́͂̈́͆͒̕͘l̵̝̜͙͉̦̮͐̒͒̑́͘͝ę̴̧̪̖̬̲̻͔̫͇͎͖̈́̊͐̑̈͂͌̉̆͗͝ = true
5
6
u/StabbyPants Nov 10 '21
figure out how to have 100 variables that are visually identical, call it hate-coding
2
u/Cuauhtemoc-1 Nov 11 '21
Don't need fancy encodings for that.
Just make all your identifiers 8 character string using upper case I and lower case l.
function (IIII, llll, llII, IIll) { ... }
Have fun ...
2
u/StabbyPants Nov 11 '21
It’s all fun and games until I figure out how to make your ide display comic sans
1
u/Cuauhtemoc-1 Nov 11 '21
Of cause. By the way, how do those Unicode homoglyphs look like in Comic sans? I've never tried that ...
1
1
0
Nov 10 '21
[deleted]
-6
Nov 10 '21
Disagreed. Comments and strings should be written in english as well in most cases, especially where international collaboration is required.
2
u/vytah Nov 10 '21
especially
What do you mean "especially"? Should the entire team that speaks a language X write comments in broken English, awkwardly translating terminology related to the problem domain (which is usually limited to their own country) into random English words just so it's in English for sake of being in English?
There's no value in that. No, scratch that, there's negative value in that.
0
4
u/d8f312 Nov 10 '21
I think Github already shows a warning if there are higher-number unicode characters in a file. I recently had to work with EU digital covid certificates which require a subject name to be in ICAO 9303 machine readable format. When I opened the character map file in Github I got a warning.
4
u/auxiliary-character Nov 11 '21
This is how it appears on my screen.
Broken unicode rendering FTW. Very obvious when it shows up as a ▯ instead of pseudo-whitespace.
5
u/dinominant Nov 10 '21 edited Nov 10 '21
The programming language should explicitly list all valid characters and their uses. Explicitly enumerate them in the definition. Allowing "classes" or "ranges" grants external bodies to change a standard or definition and then retroactively modify the behavior of code and programs.
For the case of unicode characters, escape them inside a string. Otherwise they are invalid syntax. This is how it is implemented in international domain names via punycode.
I used a trick like this many many many years ago to force bots and spammers to contact their local police instead of me when they scraped my resume.
Until recently, the entire KDE desktop and QT toolkit could be brought to it's knees if it failed to decode a unicode string in a real filename that exists on disk. I had to inject a hidden problematic file inside a zip file in the bug report to get some attention and even then some developers were completely unreasonable about the security issue of these types of attacks. It probebly took them a few months to find out where that file in their trash folder came from and then figure out why they can't empty their trash.
4
u/Lafreakshow Nov 10 '21
It probebly took them a few months to find out where that file in their trash folder came from and then figure out why they can't empty their trash.
This reminds me of that time we (kids in school) found out about these couple special filenames on Windows that explorer.exe can't deal with so we'd put them all over the school. Computers and it would take them Months to get rid of. I think in the end more or less gave up and formatted the drives. Unfortunately I can't remember what exactly it was but I think the reason it worked had something to do with how early versions of Windows used to handle physical devices.
8
u/dinominant Nov 10 '21
It's funny you mention that, because I wrote a script specifically to deal with these types of files that need to be moved from one operating system to another: https://github.com/nathanshearer/mvregex
In older versions of Windows, 98 for certain and possibly XP, if you modified a shortcut file to reference another shortcut, then pointed the 2nd shortcut at the first, it would cause Explorer.exe to enter an infinite loop when it tried to show the thumbnail of the file. Opening the folder would case the entire shell to freeze ;)
1
1
1
0
u/Worth_Trust_3825 Nov 10 '21
See you in two years when this is a fad again.
8
u/UncleMeat11 Nov 10 '21
Yeah I know. I feel like I am taking crazy pills for this whole discussion.
"Weird unicode characters used to evade code review" was first shown to me in like 2013 and none of the people involved claimed it was novel at the time. The authors of this paper just took an old idea and gave it a sexy name and are reaping the media rewards.
3
u/Worth_Trust_3825 Nov 10 '21
They're not even the first. It's being spammed for about a month now.
1
u/exploding_cat_wizard Nov 11 '21
And so it should be until languages stop accepting Unicode identifiers.
2
u/vytah Nov 10 '21
I remember when the 1337-est trick was to type 400 spaces to hide code very far to the right.
0
u/wasdninja Nov 10 '21
If it was discussed before 2013 I completely missed it. I'm not going to dig through decades old news just to discover what's already discussed.
1
u/rabid-carpenter-8 Nov 10 '21
How do I protect an open source project from Unicode attacks on github?
4
u/caakmaster Nov 11 '21
You could add a linter that checks source code and ensures that only ASCII characters are present. You could also allow your own subset of Unicode characters, too. Just have it fail if it detects any characters other than those you've explicitly allowed.
-38
u/shevy-ruby Nov 10 '21
Honestly at this point I consider all of JavaScript a Backdoor.
The particular constraint here is related to Unicode, evidently, and the sneaky invisible characters that keep on causing issues in general, even aside from Javascript:
Our approach for creating the backdoor was to first, find an invisible Unicode character that can be interpreted as an identifier/variable in JavaScript.
JavaScript in itself is not at fault directly for Unicode being sneaky. But the sheer amount of code available out there by random developers hijacking my browser scares me. These issues keep on coming back again and again - usually via NPM (thank goodness everyone else can now joke about NPM - imagine a world without NPM, we wouldn't have that many points to point out, not just left-pad only) but it's a common factor that JavaScript causes issues. I don't trust Google being the de-facto designer here either - too many conflicts of interest there.
To the exploit:
const checkCommands = [ 'ping -c 1 google.com', 'curl -s http://example.com/',\u3164 ];
While this is quite creative, I am a little bit confused. Isn't JavaScript typically more limited than other "scripting" languages in what you can call? Or can you now call ping, curl and everything else as well? I remember having had a hard time with JavaScript in this regard in the past. But my knowledge may be outdated - only recently did I find out that I can ad-hoc load the content of files via JavaScript now; I used this to have ruby generate json files which can then be read by javascript easily (or actually free format files, so json is only one of the target files; a simple text format is the default I use).
Unicode is scary too. I still haven't worked through all emojis yet!
10
u/chalks777 Nov 10 '21
isn't JavaScript typically more limited than other "scripting" languages in what you can call? Or can you now call ping, curl and everything else as well?
All the code written in the article is javascript that is executed server side running Express on a Node server. Yes, you can run commands in the shell with javascript in that setup. The example code is... not really industry standard or good practice, but it's not completely unrealistic. (Anybody who allows query input to be executed in
execshould look real hard at their review practices)Your comments are kinda out of date with the modern state of javascript. Things have changed and the language does a LOT more than it used to.
2
1
u/DeliciousIncident Nov 10 '21
Anybody who allows query input to be executed in
execshould look real hard at their review practicesRe-read the code again. There is no query input being executed in
exec. Both ping and curl commands are static strings.1
u/chalks777 Nov 10 '21
const { timeout,ㅤ} = req.query; // ... exec(cmd, { timeout: +timeout || 5_000 })
timeoutis being passed toexecstraight fromreq.query.1
u/DeliciousIncident Nov 11 '21
timeoutis indeed passed, but you said "query input to be executed inexec.timeoutis not being executed in theexec(),cmdis.19
u/strongdoctor Nov 10 '21
Your post gave me a little a headache. You seem to have misunderstood quite a few key things about what JS is, and how it's used.
21
0
Nov 10 '21
[deleted]
16
u/robin-m Nov 10 '21
It was fixed for rust.
0
Nov 10 '21
[deleted]
17
u/usr_bin_nya Nov 10 '21
The lint is a part of the compiler itself, not a tool like clippy; and it is deny by default, so code with directionality overrides will not compile unless the lint explicitly disabled with
#![allow(text_direction_codepoint_in_literal)]and/or#![allow(text_direction_codepoint_in_comment)]. Here are the lints' implementations in the compiler.→ More replies (3)
0
0
Nov 10 '21
[deleted]
5
u/ShinyHappyREM Nov 10 '21
I just tried it in Firefox on several distros - it's invisible in the code blocks (even when selecting the text), but it appears in the "A destructuring assignment is used to" and the "Similarly, when the checkCommands array is constructed" paragraphs.
0
Nov 10 '21
[deleted]
1
u/MountainAlps582 Nov 10 '21
Try grabbing the noto font package. I forget which one has emoji. I use arch btw.
251
u/drink_with_me_to_day Nov 10 '21
So we just need github/gitlab/etc to render non-ascii characters in a obvious way? Or just have a IDE running a plugin that renders atypical Unicode chars in red