r/programming • • Nov 10 '21

The Invisible JavaScript Backdoor

https://certitude.consulting/blog/en/invisible-backdoor/
1.4k Upvotes

295 comments sorted by

251

u/drink_with_me_to_day Nov 10 '21

So we just need github/gitlab/etc to render non-ascii characters in a obvious way? Or just have a IDE running a plugin that renders atypical Unicode chars in red

117

u/IsleOfOne Nov 10 '21

No, this is not something that humans need to be mitigating personally by “watching out” for these characters during code review. Half of our industry doesn’t even do code reviews consistently.

This is easily mitigated by SAST solutions in the CI pipeline. There are virtually zero legitimate uses of these characters in source code. Simply have your SAST step fail if any are detected.

17

u/mhink Nov 11 '21

Out of curiosity, what is SAST?

To be perfectly honest, in JS/TS, you could probably get away with a fairly simple eslint rule that checks identifier names for unusual characters and fails the lint.

14

u/CoderHawk Nov 11 '21

SAST

Static Application Security Testing

51

u/[deleted] Nov 10 '21

Also who does code reviews on all their NPM packages?

-54

u/[deleted] Nov 10 '21

Competent developers don't add NPM packages willy-nilly. If you have more than 15 dependencies on a medium sized project, you're probably doing something wrong.

But also, just configure your linter to include node_modules and you're all set.

52

u/LetterBoxSnatch Nov 10 '21

I have only one dependency, create-react-app /s

-7

u/[deleted] Nov 10 '21

Some of the people responding here probably actually use create-react-app in production lmao

16

u/aniforprez Nov 11 '21 edited Nov 11 '21

I don't know why this is downvoted so hard. It's a pox and the dependency tree on this is so insane it's a massive vector for literally any vulnerability that could be discovered. Please try to migrate away from this on production to literally anything else

Look at all the crap it adds to your system and dependency tree. That graph literally doesn't run on my gaming PC if I let it finish

Of course I don't agree with "competent developers audit packages individually and if you don't you're a loser moron" cause projects will be big and will need a lot of stuff but please be mindful of what you're adding

25

u/MatthewMob Nov 10 '21

You must not have a job or either you're about to get fired because wasting hundreds of hours auditing thousands of packages is not a feasible thing to do.

Fact that you didn't know: Packages install other packages, it doesn't matter if you have one or fifty, you probably have too many to go through manually.

5

u/HumbledB4TheMasses Nov 11 '21

Depends entirely on your job bud. I work for a bank right now, they have their own internal package repo for all tools they use, which have been combed through manually. Any updates to those tools (which they basically never download) also are looked over manually again. The only time external code is trusted is if its contracted out, with clear responsability falling on the 3rd party, and even then the internal security team conducts pentests and presents audits to 3rd parties.

You don't fuck around with security when it matters because, "wAsTiNg HuNdReDs Of HoUrS" is way fucking cheaper than going out of business/to jail after you're criminally negligent.

→ More replies (9)

2

u/HumbledB4TheMasses Nov 11 '21

Careful there, all the webshit devs who don't care about security are butthurt in your replies...but you're right.

You shouldn't be exposing your users to hundreds of different sources of code which you haven't combed through for malicious scripts. It can jack session tokens for sites not even related to yours, don't be an asshole, check your dependencies. It's not even that much effort, if it is hard, you probably do have too many dependencies. Dependencies are meant to handle the long/hard stuff, not do your job for you.

0

u/[deleted] Nov 10 '21

[deleted]

→ More replies (10)

31

u/jorge1209 Nov 11 '21 edited Nov 11 '21

Half? Are you just making up facts to support your position, and thinking nobody to call you on it?

You think half the industry doesn't do code reviews?!

More like 2/3rds.

3

u/GaianNeuron Nov 11 '21

The industry does code reviews, but this is a problem that ought to be solved with automation, not reliance on human perception.

12

u/[deleted] Nov 11 '21

You missed the joke.

→ More replies (1)

1

u/Phobos15 Nov 12 '21

More like 2/3rds.

I picked the wrong employer, I would love one of these easy jobs where no one cares.

2

u/jorge1209 Nov 12 '21

They actually suck. Code reviews are good.

Without reviews, without standards, bad programs at allowed to stay, poor techniques gain root, and bugs flourish.

In the end you spend all your time putting out fires your coworkers started, and their version of helping is to throw gasoline on the fire.

The question is not should you have code reviews, but do you do them in a way that builds people up and develops skills, or do they tear people down with them.

5

u/nightcracker Nov 11 '21

The Rust programming language has long disallowed homoglyph characters in the source code in the first place. The linked paper in the article that uses bidirectional overrides is also mitigated now, since Nov 1: https://blog.rust-lang.org/2021/11/01/cve-2021-42574.html

There is no legitimate reason for these characters to appear unescaped in source code. Your tools should automatically reject them.

-39

u/mcilrain Nov 10 '21

GitHub is too woke to “other” a certain set of characters.

0

u/[deleted] Nov 10 '21

[deleted]

0

u/[deleted] Nov 10 '21

[deleted]

4

u/recycled_ideas Nov 11 '21

HL7 was designed as a wire format running down a constantly open socket.

As such it has to be really, really anal about when a message or section of a message has completed.

On top of that it's one of those standards that is basically a giant ball of edge cases and a lot of developers write code without the foggiest idea that those edge cases even exist.

So it's a complex spec with a lot of piss poor implementation by people who saw three messages and thought the grokked it.

139

u/mindbleach Nov 10 '21

Banning unicode would be silly - but highlighting unicode would be just as easy. If you can detect it then you can flag it. Editors can already force the display of unprintable characters like whitespace and CR / LF. Just make it a warning, not an error.

A whitelist of non-confusing characters would avoid desensitizing people to that warning. No English speaker is going to see a variable named Einbahnstraße and think it's trying to pull a fast one. So you'd be free to throw an evil invisible character at the front of it. The double-S double-bluff.

56

u/darthwalsh Nov 10 '21

There's already been a lot of security work going into Unicode characters in URL hostnames that are pixel-for-pixel matches for ASCII characters, like some eastern european "e" that's not an e allowing for phishing at google.com.

Throwing up a big warning for invisible characters seems trivial in comparison.

1

u/Celestial_Blu3 Nov 12 '21

Why are those pixel-for-pixel identical characters even allowed?

2

u/darthwalsh Nov 12 '21

Imagine you're from eastlandia and you want to put the name of your school in your website domain. Would be pretty obnoxious if you could put most of your Unicode character alphabet into the name, except for one vowel which happens to match up with English...

But you're right, I think the result of the security fixes was to not allow the mixing of lookalike characters with English characters. Works great unless you find out you can spell out a-p-p-l-e completely with lookalikes...

5

u/[deleted] Nov 11 '21

No English speaker is going to see a variable named Einbahnstraße and think it's trying to pull a fast one.

I would ask why the programmer wouldnt just use ss for esset

6

u/Godd2 Nov 11 '21

Sometimes you just gotta go old school, weißen Sie?

3

u/mindbleach Nov 12 '21

Because that's how it's fucking spelled.

Why did you write "programmer" when the Hawaiian alphabet has no R?

1

u/bloody-albatross Nov 12 '21

I would ask why the programmer isn't just using oneway? Less to type, too.

1

u/XeonProductions Nov 12 '21

Perhaps banning invisible unicode characters, or adding detection that will flag certain use case scenarios for further review.

-83

u/PL_Design Nov 10 '21 edited Nov 10 '21

Banning unicode is not silly. Unicode is dreadful, and most programs will never be translated. 99% of the time it is literally pointless and people would be better served by using local character encodings.

EDIT: Isn't it interesting how saying you dislike unicode causes everyone to dogpile you? It feels like all of you have been brainwashed. It is startlingly creepy. I suggest you freaks go to therapy.

51

u/CartmansEvilTwin Nov 10 '21

No. We had that already with all those ISO encodings and it's hell.

What is the local encoding for Germany for example? We have our own Umlaut-characters, but what if some spaniard called Piñera wants to live here? And what about André, Çem, etc.?

So you end up with an encoding that looks almost identical to Unicode/UTF-8 anyway.

7

u/naasking Nov 11 '21

What is the local encoding for Germany for example? We have our own Umlaut-characters, but what if some spaniard called Piñera wants to live here? And what about André, Çem, etc.?

There's a middle ground here: only permit full Unicode between a programming language's string delimiters, ie. typically between two " characters, and the rest of the grammar must use only printable ASCII characters. This takes care of all input/output issues like the example you mention, without introducing homoglyph and invisible character vulnerabilities into a language's grammar.

9

u/auxiliary-character Nov 11 '21

This takes care of all input/output issues like the example you mention

Except for when you want to credit a programmer named Piñera in a comment, since comments exist outside string delimiters.

1

u/marinuso Nov 11 '21

Code isn't the same as data. You can have Mr. Piñera living on the Einbahnstraße but you name the columns lastname and street. (In English, because code should be written in English anyway.)

It's perfectly sane to restrict identifiers to ASCII, or preferably even a subset of that. Even APL of all languages restricts identifiers to letters, numbers, and a handful of whitelisted punctuation characters.

(Of course you shouldn't ban Unicode entirely.)

→ More replies (33)

30

u/mindbleach Nov 10 '21

In which the programming subreddit tries to solve the underhanded C competition by saying a compiler should shit the bed if you add Tools > Preferences > Language > 日本語.

And if I try to copy-paste code from a StackOverflow user in Russia, I guess I can go fuck myself.

→ More replies (17)

21

u/ClassicPart Nov 10 '21

99% of the time it is literally pointless

Sit down for this one, but it might shock you to learn that there are other countries on this planet. It's "literally pointless" for you. Get it right.

→ More replies (1)

13

u/DethRaid Nov 11 '21

Isn't it interesting that you have a bad idea and everyone is downvoting that because it's a bad idea?

→ More replies (5)

7

u/scratchisthebest Nov 11 '21

i agree also everyone on the planet should speak english. i am very smart. i love to use "code pages"

→ More replies (1)

3

u/wankthisway Nov 11 '21

saying you dislike unicode

is not the same as you actually saying

Unicode is dreadful,

Less victim mentality, please.

1

u/PL_Design Nov 11 '21

I'll call a piece of shit a piece of shit, thank you very much.

1

u/Sag0Sag0 Nov 11 '21

Yes, you are right. This is just one big conspiracy by big Unicode.

→ More replies (3)
→ More replies (4)

46

u/f0rtytw0 Nov 10 '21

7

u/tjpalmer Nov 11 '21

Yeah, the topic has almost nothing to do with js specifically.

6

u/f0rtytw0 Nov 11 '21

Yeah, the take away is don't trust your eyes when visually inspecting something that can use unicode.

207

u/KaiAusBerlin Nov 10 '21

eval(myWholeBundledProjectCode.replaceAll(hackingChars, ''))

wait 1 hour and there will be an npm package for that

/s

64

u/Zaphoidx Nov 10 '21

I do wonder how Github and other online repositories deal with this sort of stuff.

Do they render the character normally, or do they special-case it to ensure that stuff like this doesn't slip through?

Never come across it myself in the wild so have no clue.

67

u/MathWizz94 Nov 10 '21

One of the links in the article leads to a Gist with hidden characters that GitHub shows a warning about: https://gist.github.com/jupenur/f4c10dce1b2824cd1273f6b518fd968b

25

u/FVMAzalea Nov 10 '21

The warnings are new after the Cambridge researchers released the CVE a couple weeks ago.

30

u/StabbyPants Nov 10 '21

wait 2 hours and it will also mine btc and send the proceeds to some .ru address

3

u/auxiliary-character Nov 11 '21

Or you could use a git hook to do it instead of doing the check at runtime like a maniac

100

u/chalks777 Nov 10 '21 edited Nov 10 '21

Very cool exploit and I like the idea. Ideally this should be caught at least two ways:

1. Lint would almost certainly catch this. In particular this should give an error for improper formatting:

const checkCommands = [
    'ping -c 1 google.com',
    'curl -s http://example.com/',ㅤ\u3164
];

because (based on the patterns in this example) it should be:

const checkCommands = [
    'ping -c 1 google.com',
    'curl -s http://example.com/',ㅤ
    \u3164,
];

and if(environmentǃ=ENV_PROD){ violates no-cond-assign

2. PR review. Yes, it's hard to see visually, but the cardinal sin here is putting ANY user input into exec. That's insane.

42

u/Wacov Nov 10 '21

the cardinal sin here is putting ANY user input into exec. That's insane.

You mean the timeout? Without the hidden var the checkCommands array doesn't contain user input

11

u/chalks777 Nov 10 '21

You mean the timeout?

Yes. Granted, it's almost certainly fine to put a timeout direct from req.query in the call from a security/exploit standpoint (see documentation). I would definitely object to anybody doing that normally because it's a really bad habit to get into, even in this case. I would hope that when scrutinized a little harder you would find something weird going on.

I wouldn't expect a normal reviewer to actually notice the \u3164 though without the help of some automated tool.

9

u/kenman Nov 10 '21

Granted, it's almost certainly fine to put a timeout direct from req.query in the call from a security/exploit standpoint (see documentation).

Are you speaking only to the injection vector? Because setting a timeout of 0 (or some exceptionally high value), coupled with a massive number of requests, would create a self-inflicted DoS. The code should at least provide a window of acceptable values.

4

u/chalks777 Nov 10 '21

The point stands, eh? Don't put user input into exec. :)

4

u/Fatalist_m Nov 11 '21 edited Nov 11 '21

It does not put timeout directly into exec though, "+timeout || 5_000" will always return a number. You could add range checks or any other checks but the exploit would be just as hard to notice.

1

u/Phobos15 Nov 12 '21

I wouldn't expect a normal reviewer to actually notice the \u3164 though without the help of some automated tool.

I think most people who care about formatting would have noticed it 99% of the time and asked for the trailing "," and space to be removed. But stuff can slip through. So you still want other checks.

I noticed it before reading what the article was about. Granted, I think most people would have only realized it was a variable and not a formatting mistake if they had been reading about the browser url exploits and if they commonly use lint which has forced them to care about trailing commas.

39

u/buncle Nov 10 '21

but the cardinal sin here is putting ANY user input into exec.

I think the clever part of this exploit is that it appears, at first glance, that there isn’t any user input going I to exec (it would look like cmd is a fixed array).

Definitely pretty clever.

I would say this is an issue that lays with the editors, more than anything else. Allowing invisible Unicode to sit within an open source file is unpleasant for a number of reasons (not just exploits, but making it hard to locate copy/paste errors). I think the obvious answer here would be for IDEs to make ‘invisible’ characters visible while editing.

6

u/chalks777 Nov 10 '21

Agreed completely. My only point with the exec is that it might get more attention in a PR review because it's putting user input (timeout) directly into the function call options.

4

u/ShinyHappyREM Nov 10 '21

I would say this is an issue that lays with the editors, more than anything else

Or it's languages that allow non-ASCII characters outside of strings and comments...

5

u/buncle Nov 10 '21

I think Unicode should be acceptable, for non-English speaking coders, but going down this route would require a specific subset of Unicode (which could be a can of worms, and add complexity to the language).

It’s hard to say what the ideal solution here would be, but I agree that ideally invisible characters should not be parsed by the language outside of strings/comments at all (or should throw an error).

8

u/ShinyHappyREM Nov 10 '21

I think Unicode should be acceptable, for non-English speaking coders

Even as a non-native speaker I have to say it'd be effectively useless.

Have you ever tried to read code with identifiers in a language you didn't understand? It may as well be obfuscated. Adding non-latin characters would make matters even worse.

1

u/buncle Nov 10 '21

That’s fair. It’s not something I have any personal experience with, so my thought/opinions on non-Latin chars at a language level have zero weight.

1

u/Programmdude Nov 11 '21

In some countries (india, china and likely japan) come to mind, using english identifiers would also be like reading obfuscated code. If the software company is entirely local to that country, not all the employees will be able to speak english with any degree of proficiency.

I still think ascii should be used for identifiers instead of unicode, china can use pinyin and japan can use romaji.

5

u/SureFudge Nov 10 '21

but the cardinal sin here is putting ANY user input into exec. That's insane.

Came here to say this. Don't use exec, eval and the likes ever.

3

u/Doctor_McKay Nov 11 '21

exec is completely different from eval. Sometimes you need to invoke an external process.

3

u/Magzter Nov 11 '21

Regarding point 2 it's not really the cardinal sin here. The point is it's a backdoor, even if timeout was sanitised and mapped to a range of acceptable values before being passed to exec, the backdoor still exists.

2

u/ubernostrum Nov 11 '21

This is a thing you already had to be watching out for if you were doing stuff like user signups; people can do bad things in usernames if you let them.

22

u/ambirdsall Nov 10 '21

If the invisible variable definition were formatted like

const {
    timeout, ㅤ
}

then the whole thing would be visually indistinguishable from ordinary code using trailing commas style.

15

u/lazyl Nov 10 '21

But the linter would still complain.

-1

u/kenman Nov 10 '21

I've always hated the comma-dangle rule anyways.

25

u/chalks777 Nov 10 '21

I like using the always-multiline option.

Valid:

{ foo, bar, baz }

{
    foo,
    bar,
    baz,
}

Invalid:

{ foo, bar, baz, }

{
    foo,
    bar,
    baz
}

31

u/lazyl Nov 10 '21 edited Nov 10 '21

I like the way it keeps the commit diffs clean.

2

u/Kwantuum Nov 11 '21

and now any time you add a line at the end of an object, you get two lines of diff instead of one.

1

u/p4y Nov 11 '21

If somebody else adds a line to the same object, you get a merge conflict and your editor wants to solve it by duplicating the line where you added a comma

9

u/ProgramTheWorld Nov 10 '21

There are more tricks with Unicode like flipping arguments order with the writing direction characters. Fun stuff.

57

u/theoldboy Nov 10 '21

Obviously I'm very biased as an English speaker, but allowing arbitrary Unicode in source code by default (especially in identifiers) just causes too many problems these days. It'd be a lot safer if the default was to allow only the ASCII code points and you had to explicitly enable anything else.

24

u/lood9phee2Ri Nov 10 '21 edited Nov 10 '21

well, indeed arbitrary unicode as bare identifiers may be questionable I suppose?

Even if desired to write source code identifiers in a different writing system for whatever social/cultural/political/ideological/plain-contrariness-and-obfuscation reasons, you could perhaps just allow a different subset of unicode, yet one that's that's still small and not too ambiguous like ascii.

e.g. like that corresponding to russian koi8r (cyrillic, for glorious motherland comrade), i.s.434:1999 (coding in something normally written two thousand years ago on large rocks is the sort of thing the irish would do because it's funny), or whatever.

I'm not saying actually use the old national encodings, just it would be possible to limit identifiers in given compilation units to being from particular subsets of unicode that are kind of like the old 8-bit national encodings in the grammar, i.e. there is a medium between "ascii ...that actually doesn't even work fully for most european languages arguably including proper english though we're used to that" and "arbitrary unicode" that is "non-arbitrary unicode limited in various ways, perhaps to subsets corresponding to particular scripts".

At interface boundaries you could allow controlled importation i.e. identifiers outside the subset have to be explicitly imported (so that your delightfully incomprehensible all-ogham codebase can still link against stdlib) - because it would all be still unicode and not actually national 8-bity encodings, that would still work.

9

u/MrJohz Nov 10 '21

I think browsers have come up with a reasonable solution for URLs — you can use characters from certain character sets, but you've got to remain in the same character set in the same URL. For example, you can use as many Unicode characters as you like based on the Latin alphabet (accents, digraphs, etc), but if you combine a character from the Latin alphabet with one from the Cyrillic alphabet, you'll get an error (or at least for most browsers, the "raw" punycode representation will be shown). There are a bunch of other rules that help here, such as banning invisible characters, banning a list of known dangerous characters, etc.

I think these sorts of rules are probably a bit restrictive for defining identifier rules, particularly because subtle changes in these rules can have big effects on whether a program is valid or not. However, as linting rules (ideally ones that block builds by default), they would work very well. I know that the Rust compiler does a lot of this sort of stuff — if there are confusables in identifiers, or the "trojan source" characters mentioned at the top of this article — and by default prevents the code from compiling (although this is only a lint, and therefore can be disabled manually if desired).

Unfortunately, there's not much standardised in the JavaScript ecosystem, but I do think developer tools like ESLint and editors/code viewers like GitHub should be showing these sorts of warnings by default.

2

u/StabbyPants Nov 10 '21

what about having modules declare their codepoints? so, if you want to name a variable кофи, you declare your module as using cyrillic, the linter allows ansi + cyrillic, and your dep mgmt rolls up a list of all subsets currently declared. so, if your footprint is russian, euro, ascii, fine. if it's got akkadian in it, be suspicious

11

u/mindbleach Nov 10 '21

Anything unusual should be highlighted and warned about. That's sufficient.

It's extensible to other spoken languages - someone editing in Japan can expect to see ASCII alongside all three of their native alphabets, but Hangul would still be kinda weird. It should show up as a unicode error block � in addition to having its intended effect. Like how missing stuff in video games tends to show up as giant glowing checkerboards: you can't miss it. Making anything unexpected, visible, lets you reason about what the fuck it's doing, and what the fuck it's doing in your code.

And if it causes headaches for anyone using emoji in their Javascript... good.

4

u/1337Gandalf Nov 10 '21

C and C++ got that right.

13

u/theoldboy Nov 10 '21 edited Nov 11 '21

C and C++ don't allow Unicode in identifiers, which stops many obvious exploits, but most compilers do allow it elsewhere (in literal strings and comments). That can be exploited too.

EDIT I'm wrong. it's implementation-defined I think but gcc and clang do allow Unicode identifiers for both C and C++.

2

u/[deleted] Nov 11 '21

That doesn't fool the compiler or even the editor syntax highlighting:

https://godbolt.org/z/9desTsdec

2

u/theoldboy Nov 11 '21

Works for me with the examples from https://github.com/nickboucher/trojan-source

trojan-source/C/commenting-out.c

trojan-source/C++/commenting-out.cpp

Yes, the syntax highlighting isn't fooled. Not sure what Godbolt is using for that but many editors have been patched since that paper was published.

1

u/[deleted] Nov 11 '21

That is good to know, the version that can be compiled no longer looks deceiving in editors like Notepad++ or MSVC, and the code that still looks deceiving doesn't compile.

1

u/[deleted] Nov 11 '21

1

u/1337Gandalf Nov 11 '21

tbf, it's pretty obvious that something is up.

for (unsigned ㅤ, ㅤㅤ, ㅤㅤㅤ; std::cin >> ㅤ >> ㅤㅤ >> ㅤㅤㅤ;) {

2

u/[deleted] Nov 11 '21

True, but it compiled without a single warning. :(

4

u/mcilrain Nov 10 '21

Isn’t that how Python does it? You need to specify encoding at the top of the file or it’s ASCII or Latin-1 or something by default.

6

u/theoldboy Nov 10 '21

Used to be, but Python 3 changed the default encoding from ASCII to UTF-8.

15

u/[deleted] Nov 10 '21

Strongly disagree, comments should be in the language of the programmers and those who will read the code. Most people you are going to see on reddit already speak English well, so they are obviously not going to be bothered by English only.

Because banning non ascii-characters basically means that, denying people the ability to write code in their language.

2

u/[deleted] Nov 10 '21

English is the language of international collaboration. You're effectively stopping your code from scaling out by not writing it in English.

17

u/[deleted] Nov 10 '21

Yes and ? The website I built for a French political party is not going to scale to millions of users in a grand display of international collaboration. It's going to be read and maintained by three blokes who all speak French.

4

u/exploding_cat_wizard Nov 11 '21

And if they attempt to use French in the syntax, it will be harder to maintain than if they sensibly restrict themselves to using French strings and comments.

There are no reasons for a language to allow non-ASCII identifiers and keywords, a charset every language on earth has an official transliteration to, that trump programmers easily seeing what exactly was written.

2

u/[deleted] Nov 11 '21

Still a PITA. Hopefully all of them will use the same encoding, otherwise it will be a lot of fun fixing bugs!

4

u/vytah Nov 10 '21

Most code is never going to scale out, so writing comments and user-facing string literals in a language that represents the problem domain accurately is the way to go.

1

u/exploding_cat_wizard Nov 11 '21

That doesn't require the language to stupidly allow UTF-8 encoding in it's syntax. Just in strings and comments, which is where it belongs.

1

u/Shautieh Nov 13 '21

How are you supposed to name your fucking variables? Of course the coder will use the language he knows.

1

u/exploding_cat_wizard Nov 13 '21

And you don't need extra characters to name them, since every language has a transliteration to Latin.

-1

u/blobjim Nov 11 '21

It's the language of "we invaded your country and imposed our language on you, now we'll impose it again in computer source code!"

9

u/MrSqueezles Nov 10 '21

I understand wanting to code in a native language. We don't expect the entire world population to learn English. I'm no expert, but based on the description, it may be the "!" used in the second example is for commonly used multi-directional languages that require extra clearance on either side of punctuation. Maybe the correct restriction is "Unicode word characters only".

13

u/nitrohigito Nov 10 '21 edited Nov 10 '21

The only time people use the native language here for code is when teaching/studying, or for crappy single-use code nobody else will probably read. It's a tremendous red flag.

It's a bit like Latin used to be. It's sad, annoying, but you really just gotta put up with it, cause it's a numbers game, and boy are we outweighed.

It also doesn't help that the syntax of virtually every programming language I've encountered so far simply meshes unwell with the grammar of the native natural language here, so even for identifiers, it's sometimes just not the greatest.

7

u/wasdninja Nov 10 '21 edited Nov 11 '21

We don't expect the entire world population to learn English

We pretty much do if they want to become programmers. The official documentation of many things are in English only as far as I can tell. Not to mention that the programming languages themselves are literally in English.

1

u/Shautieh Nov 13 '21

That's far from true. Many docs are available in multiple languages, and when they aren't there are unofficial docs which are. It's hard enough to learn to program, English doesn't have to be a part of it.

1

u/blobjim Nov 11 '21

That should probably change.

4

u/wasdninja Nov 11 '21

Programming languages should definitely not be translated. That is really dumb. Having documentation in more languages would be good but documentation is hard enough as it is to keep up with in a single language.

Anyone who doesn't know English is going to have a very rough time learning programming for the foreseeable future.

3

u/bloody-albatross Nov 12 '21

Programming languages should definitely not be translated. That is really dumb.

It is. It is also what Excel and other spreadsheet software already does! And it causes problems when in the German version of Excel a decimal number uses comma instead of the decimal point and then some badly hand crafted VBA script creates invalid CSV files or SQL queries or similar.

28

u/AttackOfTheThumbs Nov 10 '21

As a German, no, everyone should code in English. Coding in other languages is stupid. The field is English and as such, everyone should adjust to it.

23

u/kaashif-h Nov 10 '21

Having had to read a codebase where Indian programmers had used Hindi naming conventions or something...I agree.

11

u/QuotheFan Nov 10 '21

That would have been hilarious!

kaksha pustak {

junta:

 pustak();

 sankhya prishtha_sankhya;

 vakya lekhak;

};

Comments be like:

// mujhe nahi pata yeh code kyun kaam karta hai. Likhne waala ya toh bhagwaan tha ya chutiya.. :P

7

u/eattherichnow Nov 10 '21

kaksha pustak

Pole reading the above: not english? WTF. Immediately correct:

porridge brick

7

u/AttackOfTheThumbs Nov 10 '21

I have read German code, dutch, danish, and others I didn't recognize. It's just a silly thing to do, and entirely pointless.

3

u/[deleted] Nov 10 '21

As a non-english speaking person, I do agree, reading non-english identifiers is pain

11

u/CartmansEvilTwin Nov 10 '21

And yet, many organisations use tons of native language comments, business lingo or interface definitions.

A good example I encountered a few years ago is Schufa. Their entire interface is German XML.

6

u/AttackOfTheThumbs Nov 10 '21

And yet, many organisations use tons of native language comments, business lingo or interface definitions.

Not everyone can make the right decisions all the time. Comments in code I'm pretty ambivalent to myself. The other too are bad. It would be interesting to see when they decided to use the native tongue.

I work with ERP systems. I have seen a mix of many languages, and in general, when it's not in English, the business ends up losing, because the support becomes more costly. Most of the time I found they made that decisions x years/decades ago and it has been carried forward ever since. Sometimes they end up deciding to transition, other times they start mixing.

I think Schufa is probably big enough to get away with it, but that doesn't mean it was smart. I kind of assume they don't expand past the German speaking space, but I don't even know, since I've never worked with them directly.

It's all based on personal experience anyway. I would just say it's typically bad when things other than English are used.

3

u/[deleted] Nov 10 '21

I'm a native Spanish speaker, fan of foreign languages. I definitely prefer to code in English.

Although I created once a toy language with Spanish keywords

2

u/DrayanoX Nov 11 '21

That's easy for us to say when we are already fluent in English. The majority of the world population isn't, or do have some rudimentary English knowledge but aren't comfortable or good enough to use it.

There's no reason to prevent anyone who doesn't speak English from getting into programming this is elitism at its finest.

Exploits can easily be prevented by just blocking specifically confusing and invisible characters from being used. There's no reason why characters such as "ß ç ñ ē ب" cannot be used by people who speak such languages using these.

Blocking all of Unicode is like cutting off your entire leg because you stepped on a Lego.

0

u/Retrofire-Pink Nov 10 '21

disagree strongly as an American and native-english speaker

2

u/nitrohigito Nov 10 '21

makes sense

0

u/Shautieh Nov 13 '21

As a German you got no say in this for two reasons : 1 English is easy to learn for you so of course you don't care about others troubles 2 your parents had no other options than to accept that the USA were superior. That's not the case everywhere

1

u/AttackOfTheThumbs Nov 14 '21

This comment is on a whole different level of stupid. Thank you for making me laugh with your idiocy.

USA is third world.

3

u/vytah Nov 10 '21

it may be the "!" used in the second example is for commonly used multi-directional languages that require extra clearance on either side of punctuation

No, it's a letter, U+01C3. But since it's used only in minority languages in Namibia and RSA, like ǃKung, ǃXóõ or Khoekhoe, it's very unlikely to appear in code (in either code proper, comments, or literals) at all.

8

u/AttackOfTheThumbs Nov 10 '21

No, you are correct. Programming should only use a default ascii set. Anything else is stupid. Limit the tools to limit the exploits. There's zero issue with this.

4

u/ThirdEncounter Nov 10 '21

I'll have agree with /u/beached on this one. Telling about 80% of the population who speaks a language other than English "use ascii, because anything else is stupid" is, well, misinformed.

Let's reverse the roles, and say that the "one true character set" is "Japanese ascii" (kanji-scii?) Now you can't use variables such as "loopCounter" because it's not kanji-scii. You have to use ループカウンター because "using loopCounter is stupid."

There's gotta be a way to mitigate the risks, I agree. But "ascii only!" is not it. This is not the 70s anymore.

2

u/Shautieh Nov 13 '21

Exactly. Redditors are so backwards about that. I'm fluent in English but we can't expect people to open a dictionary every time they need to write and read a variable.

1

u/exploding_cat_wizard Nov 11 '21

The programming language already forces the use of English, your example doesn't make sense. It's "static public void", not whatever the kanji version of that would be, in Java, and similarly in every language that's actually used in prod.

If these Japanese speakers so beset upon that JavaScript has an English syntax invent their own JapanScript that uses only kanjis, that wouldn't be a problem ( except for whomever thought that would be a good idea, but I'm not one to forbid you to take on whatever problem you want to make for yourself ). It means nobody outside of Japan will be able to use it, and these people will severely limit their community, but at least the whole rest of the world won't have to fight an entirely new sneaky class of bugs because making programming even more complicated is the cool thing to do.

And it's not like anyone outside Japanese readers can even help you with your JavaScript written in kanji, so the actual advantage for you, the UTF-8-kanji-JS writer, is minimal compared to just using kanji-script from the get go.

3

u/DrayanoX Nov 11 '21

The number of programming keywords is limited, it's easy for a non-english speaker to learn them by heart.

Expecting him to learn the entire English language just so he can write code is stupid.

1

u/exploding_cat_wizard Nov 11 '21

That's not at all what anyone here said, wherever did you get that from? You can write any language on this planet in the lingua franca of scripts, Latin. No need to learn English, just use ASCII to write in your language. Less problems for everyone involved, and if you really can't, make your own programming language and at least be explicit that you're doing your own thing, instead of pretending it could be part of a worldwide ecosystem.

3

u/DrayanoX Nov 11 '21

ASCII doesn't allow billions of people to write their native scripts. Russian, Chinese, Japanese, Arabic and many other scripts can't be written in ASCII.

It's unreasonable to expect someone to learn the latin script just so he could name his variables and write his comments.

It's easy enough to learn specific keywords such as const, float, function and class. It's a whole different game to learn enough of a latin language just to get started with programming. We shouldn't be advocating for more barriers to get into programming.

1

u/exploding_cat_wizard Nov 11 '21

It's a whole different game to learn enough of a latin language just to get started with programming.

Nobody needs to learn a Latin language, except those words you already conceded.

And again, if you want to create, e.g., Hindi script, a JS clone that uses Hindi characters, go ahead. Explicit is better than implicit, so admit that you're using a different programming language and stop pretending that you're part of the same programming language community when you are taking yourself out of international conversations by using local scripts. Despite being an optimal solution for everyone involved, and massively reducing actual barriers to programming like programmers not actually being able see what code is actually written thanks to UTF character fun, this option isn't ever really adopted, because it shows clearly why mixing scripts is a bad idea.

2

u/DrayanoX Nov 11 '21

Why the fuck would someone need to create a completely new language for this ? Programming languages are tools used to make the computer do stuff, nothing dictates the way someone chose to use these tools to write his own programs. No one is going to reinvent JS or Python just so he can write comments or name variables in local scripts.

Why should a group of local developers write comments in broken English/[Insert any language written in ASCII] to document their code instead of whatever language they are most familiar with.

stop pretending that you're part of the same programming language community when you are taking yourself out of international conversations by using local scripts.

No one is "pretending" anything, not everything has to revolve around the English language, at the end of the day, people just want to write code that make their computers do stuff, no one should be expected to learn a whole new script just to get started with programming, that's ridiculous.

Why are you expecting that you should be able to read their code and understand it ? Do you too get mad when you come across a book written in Mandarin or something and expect it to be written with Latin characters ?

And just in Europe you have people using languages derived from latin that have characters not available in ASCII such as à ê ç and many others, how do you expect to handle cases like first and last names written with some of these if you aren't allowed to use anything other than ASCII in your code, and that's just to give a basic example.

The solution to this problem isn't to nuke Unicode from programming, blacklisting confusing and invisible characters is easy enough without having to remove every other non-ASCII character.

→ More replies (3)

1

u/Shautieh Nov 13 '21

Naming variables is one of the most fundamental work a coder does, and you can't expect non English speakers to use a dictionary every time they want to read of write a variable.

1

u/exploding_cat_wizard Nov 13 '21

My comment literally says you can and should use your language, please take the time to read it instead of jumping to conclusions and arguing strawmen

→ More replies (1)

2

u/[deleted] Nov 10 '21

Another advantage of this would be a bit of compile time or runtime performance depending on language, because comparing ascii strings is probably faster than utf8 or utf16 strings when linking identifiers.

2

u/vytah Nov 10 '21

because comparing ascii strings is probably faster than utf8 or utf16 strings when linking identifiers.

Normalization is not performed, it's just matching opaque bytestrings, so the speed is the same.

One could argue that for better speed, you should name everything in Chinese, as it's denser than English.

1

u/ShinyHappyREM Nov 11 '21

name everything in Chinese, as it's denser than English

Even with UTF8?

1

u/caagr98 Nov 11 '21

Pretty sure Python normalizes identifiers.

1

u/vytah Nov 12 '21

TIL.

But I guess it's in the minority, most languages don't bother.

1

u/nerd4code Nov 10 '21

IMO it’s potentially still useful to embed Unicode text in a program for various purposes like templating, NLS, or use of fancy punctuators, operators, and symbols, it should be enabled implicitly only for comments, and explicitly for quoted §s where it’s needed, with stringent limits on layout (no mirroring, no full-line RTL, no embedding controls other than RLE, LRE, and PDF) should be permitted in those contexts.

The rest of the code can still be coded as UTF-8, but anything outside the wossis, G0? range I think it’s called? should trigger an error—so U+0020…U+007E’d be permitted, plus C0 ctrls HT, LF, VT, FF, CR as syntactic markers outside quoted regions, maybe +LSEP, PSEP, maybe +(C1) NEL, maaayyybe +(C0) NUL (as 00 or C0,80) and DEL for chars to ignore entirely. Unicode’d potentially still cause problems where permitted, but at least the scope would be bounded and relatively easy to scan for, sorta like an unsafe region.

0

u/[deleted] Nov 10 '21

What makes you think that ASCII would be the one true set of codepoints? Just because it was that way, doesn't mean it would have to continue. We live in a world with many more languages than English and English is not the dominant written or spoken language. Also, we have tools for this already.

2

u/AttackOfTheThumbs Nov 10 '21

English is the dominant language for software development.

1

u/[deleted] Nov 10 '21

You should look at the source code for a tonne of device drivers. I've had to use google translate when looking through source code to get a better understanding. But, any move from unicode will result in an bunch of new non-english languages/forks. It will be worse for our perceived comforting warm blanket where everyone speaks what we speak. As I said, there are tools out there now to normalize text and it's the IDE's/language/tool writers that need to update and only accept the normalize forms and to stop homoglyph attacks.

There is also http://www.unicode.org/reports/tr31/

0

u/danweber Nov 10 '21

Yes. If you need other languages, fine, all your user-displayable strings are in a separate file, and treated as hostile.

5

u/jazd Nov 10 '21

You think English speakers don't use Unicode characters?

23

u/emperor000 Nov 10 '21

For identifiers? If you are using Unicode characters for identifiers then that's probably a problem.

34

u/balefrost Nov 10 '21

p̵̛̪̺̟̫̂̒͛͗̌̒̈́͐͂̿͒͝͝͝ḛ̷̩̮̣̭̠͎̪̩̂̏͒̿̇̊̍̆͑̋͠͝ͅř̴̡̛̏f̷͓̬̆̽̀͐̆͛͗̃̑͠͝ẹ̴̜̙͚̬̮̜̙͙͇̪̾͋͊c̶̝̣̖̼̆̔͛̎̈͆͊̊͆̕ṫ̸̨̢̯͈͔̩̤̌͗l̴̥̬̝̥̆͠ý̸͍̿̎̈́͌̃͐̉͐͋̇̾̚N̸͙͔͍̠̜̺͎̩̩̳̝̲͗̍͒̒́̄̇̎̚͜ǫ̶̡̨͙͕͈̞̝̺̦̠͙̲̩̯̅͗̐̿̏̉̄̑̇̉͘r̴̡̢̘̱͖̘̪̝̭̪̦͈̆͑͒̆̾͑̉͊̕̕̕ͅͅm̵̧̯͕̯͙̣̹̪̱͖̠̬͔̩̪̀̔̓ä̴͚ļ̸̧͕̙͖̳͖͚̣̭͕͐͗͑ͅV̷̡̢͔͍̻͚̭̘̖̦͍̠̖̝́́̋̑̋ͅa̶̰̙̝̦̗͚̯̠̞̭̎̓̋r̸̛͓͍͍͙̟̼̬̮̫̩͎̗̯̩͗̑͋́́̊͝i̶̡̩̤̜͉̻̟̹̙̗̱͆̑̉́͐̂͊̍ͅȁ̴̟b̷̧̙̙̞̥́̄̊̊̿̀̈́͂̈́͆͒̕͘l̵̝̜͙͉̦̮͐̒͒̑́͘͝ę̴̧̪̖̬̲̻͔̫͇͎͖̈́̊͐̑̈͂͌̉̆͗͝ = true

5

u/emperor000 Nov 10 '21

Exactly. That's awesome.

6

u/StabbyPants Nov 10 '21

figure out how to have 100 variables that are visually identical, call it hate-coding

2

u/Cuauhtemoc-1 Nov 11 '21

Don't need fancy encodings for that.

Just make all your identifiers 8 character string using upper case I and lower case l.

function (IIII, llll, llII, IIll) { ... }

Have fun ...

2

u/StabbyPants Nov 11 '21

It’s all fun and games until I figure out how to make your ide display comic sans

1

u/Cuauhtemoc-1 Nov 11 '21

Of cause. By the way, how do those Unicode homoglyphs look like in Comic sans? I've never tried that ...

1

u/[deleted] Nov 11 '21

Old fashioned job security.

1

u/agumonkey Nov 11 '21

another point for haskeller single letter variables /s

0

u/[deleted] Nov 10 '21

[deleted]

-6

u/[deleted] Nov 10 '21

Disagreed. Comments and strings should be written in english as well in most cases, especially where international collaboration is required.

2

u/vytah Nov 10 '21

especially

What do you mean "especially"? Should the entire team that speaks a language X write comments in broken English, awkwardly translating terminology related to the problem domain (which is usually limited to their own country) into random English words just so it's in English for sake of being in English?

There's no value in that. No, scratch that, there's negative value in that.

0

u/[deleted] Nov 11 '21

Yes. Technical English is much more easier than regular English.

4

u/d8f312 Nov 10 '21

I think Github already shows a warning if there are higher-number unicode characters in a file. I recently had to work with EU digital covid certificates which require a subject name to be in ICAO 9303 machine readable format. When I opened the character map file in Github I got a warning.

4

u/auxiliary-character Nov 11 '21

This is how it appears on my screen.

Broken unicode rendering FTW. Very obvious when it shows up as a ▯ instead of pseudo-whitespace.

5

u/dinominant Nov 10 '21 edited Nov 10 '21

The programming language should explicitly list all valid characters and their uses. Explicitly enumerate them in the definition. Allowing "classes" or "ranges" grants external bodies to change a standard or definition and then retroactively modify the behavior of code and programs.

For the case of unicode characters, escape them inside a string. Otherwise they are invalid syntax. This is how it is implemented in international domain names via punycode.

I used a trick like this many many many years ago to force bots and spammers to contact their local police instead of me when they scraped my resume.

Until recently, the entire KDE desktop and QT toolkit could be brought to it's knees if it failed to decode a unicode string in a real filename that exists on disk. I had to inject a hidden problematic file inside a zip file in the bug report to get some attention and even then some developers were completely unreasonable about the security issue of these types of attacks. It probebly took them a few months to find out where that file in their trash folder came from and then figure out why they can't empty their trash.

4

u/Lafreakshow Nov 10 '21

It probebly took them a few months to find out where that file in their trash folder came from and then figure out why they can't empty their trash.

This reminds me of that time we (kids in school) found out about these couple special filenames on Windows that explorer.exe can't deal with so we'd put them all over the school. Computers and it would take them Months to get rid of. I think in the end more or less gave up and formatted the drives. Unfortunately I can't remember what exactly it was but I think the reason it worked had something to do with how early versions of Windows used to handle physical devices.

8

u/dinominant Nov 10 '21

It's funny you mention that, because I wrote a script specifically to deal with these types of files that need to be moved from one operating system to another: https://github.com/nathanshearer/mvregex

In older versions of Windows, 98 for certain and possibly XP, if you modified a shortcut file to reference another shortcut, then pointed the 2nd shortcut at the first, it would cause Explorer.exe to enter an infinite loop when it tried to show the thumbnail of the file. Opening the folder would case the entire shell to freeze ;)

1

u/xesenix Nov 11 '21

create lint rule for caching it

1

u/CloudsOfMagellan Nov 11 '21

Good to be using a screen reader in this case

1

u/horreum_construere Nov 11 '21

And GitHub Copilot is turned off...

0

u/Worth_Trust_3825 Nov 10 '21

See you in two years when this is a fad again.

8

u/UncleMeat11 Nov 10 '21

Yeah I know. I feel like I am taking crazy pills for this whole discussion.

"Weird unicode characters used to evade code review" was first shown to me in like 2013 and none of the people involved claimed it was novel at the time. The authors of this paper just took an old idea and gave it a sexy name and are reaping the media rewards.

3

u/Worth_Trust_3825 Nov 10 '21

They're not even the first. It's being spammed for about a month now.

1

u/exploding_cat_wizard Nov 11 '21

And so it should be until languages stop accepting Unicode identifiers.

2

u/vytah Nov 10 '21

I remember when the 1337-est trick was to type 400 spaces to hide code very far to the right.

0

u/wasdninja Nov 10 '21

If it was discussed before 2013 I completely missed it. I'm not going to dig through decades old news just to discover what's already discussed.

1

u/rabid-carpenter-8 Nov 10 '21

How do I protect an open source project from Unicode attacks on github?

4

u/caakmaster Nov 11 '21

You could add a linter that checks source code and ensures that only ASCII characters are present. You could also allow your own subset of Unicode characters, too. Just have it fail if it detects any characters other than those you've explicitly allowed.

-38

u/shevy-ruby Nov 10 '21

Honestly at this point I consider all of JavaScript a Backdoor.

The particular constraint here is related to Unicode, evidently, and the sneaky invisible characters that keep on causing issues in general, even aside from Javascript:

Our approach for creating the backdoor was to first, find an invisible Unicode character that can be interpreted as an identifier/variable in JavaScript.

JavaScript in itself is not at fault directly for Unicode being sneaky. But the sheer amount of code available out there by random developers hijacking my browser scares me. These issues keep on coming back again and again - usually via NPM (thank goodness everyone else can now joke about NPM - imagine a world without NPM, we wouldn't have that many points to point out, not just left-pad only) but it's a common factor that JavaScript causes issues. I don't trust Google being the de-facto designer here either - too many conflicts of interest there.

To the exploit:

const checkCommands = [ 'ping -c 1 google.com', 'curl -s http://example.com/',\u3164 ];

While this is quite creative, I am a little bit confused. Isn't JavaScript typically more limited than other "scripting" languages in what you can call? Or can you now call ping, curl and everything else as well? I remember having had a hard time with JavaScript in this regard in the past. But my knowledge may be outdated - only recently did I find out that I can ad-hoc load the content of files via JavaScript now; I used this to have ruby generate json files which can then be read by javascript easily (or actually free format files, so json is only one of the target files; a simple text format is the default I use).

Unicode is scary too. I still haven't worked through all emojis yet!

10

u/chalks777 Nov 10 '21

isn't JavaScript typically more limited than other "scripting" languages in what you can call? Or can you now call ping, curl and everything else as well?

All the code written in the article is javascript that is executed server side running Express on a Node server. Yes, you can run commands in the shell with javascript in that setup. The example code is... not really industry standard or good practice, but it's not completely unrealistic. (Anybody who allows query input to be executed in exec should look real hard at their review practices)

Your comments are kinda out of date with the modern state of javascript. Things have changed and the language does a LOT more than it used to.

2

u/MountainAlps582 Nov 10 '21

You're replying to a well known troll

2

u/chalks777 Nov 10 '21

womp. Now I know.

1

u/DeliciousIncident Nov 10 '21

Anybody who allows query input to be executed in exec should look real hard at their review practices

Re-read the code again. There is no query input being executed in exec. Both ping and curl commands are static strings.

1

u/chalks777 Nov 10 '21
const { timeout,ㅤ} = req.query;
// ...
exec(cmd, { timeout: +timeout || 5_000 })

timeout is being passed to exec straight from req.query.

1

u/DeliciousIncident Nov 11 '21

timeout is indeed passed, but you said "query input to be executed in exec. timeout is not being executed in the exec(), cmd is.

19

u/strongdoctor Nov 10 '21

Your post gave me a little a headache. You seem to have misunderstood quite a few key things about what JS is, and how it's used.

21

u/[deleted] Nov 10 '21

shevy-ruby is a high effort troll, respect the art but don't feed them.

3

u/strongdoctor Nov 10 '21

Ah, good to know 🙂

0

u/[deleted] Nov 10 '21

[deleted]

16

u/robin-m Nov 10 '21

It was fixed for rust.

0

u/[deleted] Nov 10 '21

[deleted]

17

u/usr_bin_nya Nov 10 '21

The lint is a part of the compiler itself, not a tool like clippy; and it is deny by default, so code with directionality overrides will not compile unless the lint explicitly disabled with #![allow(text_direction_codepoint_in_literal)] and/or #![allow(text_direction_codepoint_in_comment)]. Here are the lints' implementations in the compiler.

→ More replies (3)

0

u/[deleted] Nov 11 '21

[deleted]

2

u/[deleted] Nov 11 '21

Or at least in some other color.

1

u/Shautieh Nov 13 '21

It would be such a pain for anyone not coding in English.

0

u/[deleted] Nov 10 '21

[deleted]

5

u/ShinyHappyREM Nov 10 '21

I just tried it in Firefox on several distros - it's invisible in the code blocks (even when selecting the text), but it appears in the "A destructuring assignment is used to" and the "Similarly, when the checkCommands array is constructed" paragraphs.

0

u/[deleted] Nov 10 '21

[deleted]

1

u/MountainAlps582 Nov 10 '21

Try grabbing the noto font package. I forget which one has emoji. I use arch btw.