r/programming • • Feb 25 '21

An Exploration of JSON Interoperability Vulnerabilities

https://labs.bishopfox.com/tech-blog/an-exploration-of-json-interoperability-vulnerabilities
128 Upvotes

39 comments sorted by

33

u/headhunglow Feb 25 '21

Finally some real programming content on /r/programming! Great writeup! However, I'm wondering why any JSON parser would support comments?

26

u/evaned Feb 25 '21

Lots of things use JSON for configuration files and other things intended to be hand-written, as I'm sure you've noticed. Tools have been "updated" to support that use, which hugely benefits from comments, to the point of it turning a terrible language for that purpose into one that's usually merely mediocre.

As for why JSON is used there instead of INI or TOML something... well that I think mostly falls into "everything is terrible" category.

13

u/[deleted] Feb 25 '21

I think JSON is better than TOML for configuration purposes except for the fact that it often doesn't support comments and trailing commas (cough package.json) and it isn't clear when it does and when it doesn't. JSON5 solves that problem but isn't as widely supported.

TOML is ok, but it still has more weird quirks and is less obvious than JSON (but way better than YAML).

In particular the [[table]] syntax is very unintuitive, to the point that I still don't understand it. JSON arrays are blindingly obvious.

Another weirdness is that you can put a table in braces, but only if it is all on one line! Another thing JSON got right.

For all its limitations JSON is probably the best thing to come out of JavaScript. It got a lot right.

9

u/evaned Feb 25 '21

I think JSON is better than TOML for configuration purposes except for the fact that it often doesn't support comments and trailing commas (cough package.json) and it isn't clear when it does and when it doesn't. JSON5 solves that problem but isn't as widely supported.

I'd add a couple more: no multi-line literals, it is silent on the representation of numbers and so there's no distinction for example between ints and floats, no hex ints (I'm probably biased here; this is probably much more of a problem when you want program-generated but human-readable interchange). Compared to TOML, TOML also allows you to write numbers like 1_234_567 for 1234567, and that kind of thing always super nice.

From a configuration file perspective, I also think JSON is very syntax-heavy -- the need for commas between items, the quotes you need everywhere, you will basically always need a top-level {} and so your editor probably indents everything in the file which should already tell you that the format is stupid for that use case...

And it also has the advantage of being pretty completely specified, unlike JSON.

In particular the [[table]] syntax is very unintuitive, to the point that I still don't understand it. JSON arrays are blindingly obvious.

I'll grant you that -- but I think this is getting into "most conf files don't need this" territory.

6

u/[deleted] Feb 26 '21

Valid point about top-level {} but I disagree about commas and quoting. They help make it really really unambiguous, unlike YAML for example, and TOML's tables.

Like, you don't really have to learn JSON. If you've done any programming at all it's obvious from a tiny example how it works. That definitely isn't true for YAML or for TOML's table syntax.

1

u/AttackOfTheThumbs Feb 26 '21

Json is the answer. I don't think we need comments, not trailing commas. They are nice, but not must have.

We certainly don't need multiline. It's annoying to scroll, but we can turn on line breaks.

Json is easy. Anyone can learn it in minutes.

Json is not intended to be configuration, but it can work. Git stash to try different configs (or create multiple files). Write real documentation for the config, instead of comments. It kind of forces a good practise.

4

u/[deleted] Feb 26 '21

We definitely do need comments for some of the stuff JSON is used for. Most annoyingly package.json - you can't add comments like "this dependency pinned at V1.5.5 because of a bug in 1.5.6 - see http://...".

That's probably the only JSON config I commonly hand-edit that doesn't support comments.

Most tools support an alternative config format to JSON precisely because it doesn't support comments.

I agree multi-line and trailing commas are just nice to have in most applications, though multi-line support is basically the only reason anyone uses YAML - it's popular in CI because it makes embedding multi-line scripts in config files easy.

1

u/AttackOfTheThumbs Feb 26 '21

I don't think script files should even be in config. That's how we got into this fucking mess.

1

u/nigeltao Feb 26 '21

I think JSON is better than TOML for configuration purposes except for the fact that it often doesn't support comments and trailing commas

Funnily enough, I just posted this a few days ago: https://nigeltao.github.io/blog/2021/json-with-commas-comments.html

0

u/[deleted] Feb 26 '21

Isn't that exactly the same as VSCode's JSONC? It also doesn't solve the ambiguity problem unless he is advocating using a .jwcc extension. I wish the VSCode guys had used a .jsonc extension.

(And I wish done other guy hadn't also made a dumb format called JSONC.)

1

u/nigeltao Feb 26 '21

I have updated my blog post. No, it's not quite the same as Visual Studio's JSONC, because commas. Yes, I am advocating a .jwcc extension.

1

u/[deleted] Feb 26 '21

What about commas? They both support trailing commas.

1

u/nigeltao Feb 27 '21

Ah, I didn't know that. https://code.visualstudio.com/docs/languages/json#_json-with-comments doesn't mention commas at all. It only talks about comments.

In that case, I'm guessing that Visual Studio's JSONC is the same thing. I'd still prefer a .jwcc extension instead of .json, since it's not JSON.

1

u/[deleted] Feb 27 '21

Yeah bit of omission but it does. And yeah I just wish they used .jsonc.

-1

u/ForeverAlot Feb 26 '21

I think JSON is better than TOML for configuration purposes except for the fact that it often doesn't support comments and trailing commas (cough package.json) and it isn't clear when it does and when it doesn't.

It is perfectly clear when JSON supports comments: never.

4

u/[deleted] Feb 26 '21

Ok nitpicker, it isn't clear when a file is JSON and when it is JSONC.

10

u/leberkrieger Feb 25 '21

I've seen more than one use of JSON where comments were forbidden because they weren't portable across JSON parsers. They had to forbid comments specifically because it was so common for people to want to use them.

And why wouldn't they? Comments are used in JSON for some of the same reasons they are used in code:

  • to explain obscure values whose rationale is not apparent

  • to temporarily remove a value, but leave it in the text for future use (like when troubleshooting)

  • to make a template/sample payload used to teach how to create similar payloads, without having to supply separate documentation

3

u/masklinn Feb 26 '21 edited Feb 26 '21

I've seen more than one use of JSON where comments were forbidden because they weren't portable across JSON parsers.

No. Comments would be completely portable as-is if specified.

Crockford did not allow comments in JSON because he knew people would start using them to embed parser-specific meta-directives (that is done parsers would start treating comments as more than comments) and did not want any meta-concerns in json. In fact the original version had comments. Conveniently, removing comments also simplified implementations a lot, which made porting parsers easier, and thus contributed to the popularity of json.

Also crockford intended json as an exchange format.

6

u/KernowRoger Feb 25 '21

Well JSON is designed to be relatively human readable so I can see it being handy to comment documents.

3

u/djavaman Feb 25 '21

First you allow comments.

Then, its annotations.

And then, its an embedded scripting language.

-3

u/caleeky Feb 25 '21 edited Feb 26 '21

Depends what you mean by JSON. That's the crux of the issue. It's a format that's a side effect of junior devs trying to use interpreted code as a data format and then trying to clean up the mess and pretend it was all a plan.

edit: thanks /u/masklinn for offering the explanation :)

6

u/SwiftOneSpeaks Feb 26 '21

Do you...know the history of JSON at all? Because this is not it.

3

u/binford2k Feb 26 '21

I mean, they’re not entirely wrong. The first JSON parser was eval()….

2

u/SwiftOneSpeaks Feb 26 '21

My memories may fail me, but this feels wrong. Crockford defined the spec, and it specifically didn't match JS. SOME parsers may have used eval(), heck Crockford's first library may have used eval(), but that's not the same as BEING eval().

3

u/masklinn Feb 26 '21 edited Feb 26 '21

it specifically didn't match JS.

That was not intentional and the incompatibility is in an edge case: two “line terminator” code points were treated as thus in JS tokenisation (thus invalid in literal form), while they were not in JSON.

In fact Crockford claims that he discovered it, not invented it. And that he was not even the first discoverer of the idea. And specifically calls it a subset of JS in multiple talks. That was always the intention.

This has since been fixed, by not treating these two code points as line terminator in JS either. In recent browsers JavaScript should be a strict superset of json, which was the intent all along. That you could eval() a json string was rather the point before bat I’ve support appeared, even more so as this long predates advanced JS JITs so json parsers in JavaScript were really slow even compared to all the unnecessary work of eval.

14

u/yawkat Feb 25 '21

Parsing differential vulns (be they http request smuggling or json parsing or iOS plists) are so unnecessary. It's in their nature that they'll keep popping up, and trying to make tens of parsers work exactly the same way all the time is an exercise in futility.

The only solution is fully parsing the data and then re-emitting it. That way you can be sure only the degrees of freedom your model defines will appear in the output.

But it seems like we still need a few hundred WAF bypass exploits until this fact filters through...

6

u/Eluvatar_the_second Feb 25 '21

This, it seems really weird to have such thin layers that you can just pass the direct request on to the next layer.

1

u/sievebrain Feb 26 '21

Did you read the article? It points out that re-encoding doesn't solve the problem with some JSON engines.

9

u/Thaxll Feb 25 '21

Related and still very good: http://seriot.ch/parsing_json.php

5

u/Y_Less Feb 26 '21

From a developer's perspective, why would you waste computation by re-serializing a JSON object that you just parsed and validated when the string input is readily available? This assumption should be sound.

That assumption is absolutely NOT sound. Why are you using request data like that at all? Why is the original text version of data being passed around your system? Read it in, pass around the parsed internal format form between your components (as you will need to anyway, since not every one of those should be re-parsing the data), and them emit new when you make the request. This also allows for adding and removing and data in the middle-end, since the output is an entirely new form. JSON comes in, in-memory data gets processed, JSON goes out. This is suggesting also passing JSON around the edge.

2

u/cameron314 Feb 25 '21

I like how that one ruby parser returns 3e14159... (pi!) as the result of parsing 1e4096 :D

1

u/[deleted] Feb 25 '21

How is it compared to XML? Better/worse/similar?

3

u/mpyne Feb 26 '21

XML is probably worse overall (XML-specific flaws are the #4 threat on the OWASP Top 10 Application Security Vulnerability list even though the list was compiled after JSON was much more predominantly in use).

The float/int problem isn't present in XML, even if that is because XML doesn't natively care about typing numeric values, but the other four major categories are all represented in some way.

4

u/evaned Feb 26 '21

but the other four major categories are all represented in some way.

I think this claim will require some support.

The first vulnerability listed is JSON parsers silently omitting part of the tree because of duplicate values. I would be shocked if an XML parser would do this. At last AFAIK, the closest analogue I know of would be an error in program logic rather than the parser -- your program logic expects that only one instance of a tag is present, but there are multiple, and then the program logic does the first/last precedence inconsistently. The main argument I'd make here is that's the place for a schema -- make a schema that requires that said element is present only once. If you use a schema, which IMO you should anyway (and should in JSON), then anything that validates won't have this problem. By contrast, a typcial JSON Schema validator will do nothing to help you prevent this vulnerability in JSON parsers, because they typically do validation after parsing, after the parts of the tree have been omitted. For example, I did a search for an online JSON Schema validator and picked the first link -- it has no problem validating an "invalid" input with duplicate keys. (You could argue that such input is not invalid JSON, but then I would argue that virtually every JSON parser is horrendously buggy.)

The stuff about string truncation because of weird characters I can totally see affecting XML parsers, so I'll give you that one. However, the "using comment truncation" is something I do not see affecting XML parsers, because parsers don't usually support grammar-level extensions like that AFAIK, and even if they do they're far less frequently used.

1

u/mpyne Feb 26 '21

The first vulnerability listed is JSON parsers silently omitting part of the tree because of duplicate values. I would be shocked if an XML parser would do this.

The error is differences in handling of duplicate values by different parsers, not mere omission. If every JSON parser always omitted the same duplicates there would be no issue.

XML suffers from the same issue, where different XML parsers can lead to different XML document trees based on how they handle complex XML document trees. XML directly supports duplication of tags but the general issue of different parsers coming to different parse trees remains. This is a common source of bugs with security-minded XML applications like SAML.

You are right that these need application involvement to get to the confusion, but since an application ultimately produces and consumes XML, and since XML invites so many ways to get to "confusion" here (including duplicate tags confusing applications in the same ways duplicate keys in JSON confuse applications) I can't fully give it a pass.

By contrast, a typcial JSON Schema validator will do nothing to help you prevent this vulnerability in JSON parsers, because they typically do validation after parsing, after the parts of the tree have been omitted.

Doing this with a JSON schema is admittedly a hard luck case even if you look beyond the first Google hit :). But a JSON linter is a possibility and does the right thing and those come up on Google as well.

However, the "using comment truncation" is something I do not see affecting XML parsers, because parsers don't usually support grammar-level extensions like that AFAIK, and even if they do they're far less frequently used.

So my reference to that was to real bugs affecting SAML libraries, which can be confused into minting tokens for the wrong user by the crafty placement of XML comments in otherwise legitimate SAML XML documents.

And of course, many applications have been caught out trying to parse XML comments incorrectly (since it's difficult to do correctly) and while, yes, "they should have just used a parsing library", parsing libraries are complicated, XML seems simple enough if you use it for simple tasks, and soon you're writing your own parser that will let through hyphens that mangles a real parser.

2

u/argv_minus_one Feb 26 '21 edited Feb 26 '21

real bugs affecting SAML libraries

The bug in the linked article appears to be the result of some extremely sloppy programming in the vulnerable XML parsers. In those parsers, there is a “get this element's text content” accessor which only returns the content of the first text node! When you're getting the text content of an element, you basically never want only the first text node because there might be a comment/tag/PI in the middle of the text! The DOM textContent property concatenates all text nodes (including ones in descendant elements), which is the right thing to do and is not vulnerable to this attack.

XML seems simple enough

I invite anyone who thinks this to actually read the productions in the XML spec. Simple, it ain't.

1

u/mpyne Feb 26 '21

I invite anyone who thinks this to actually read the productions in the XML spec. Simple, it ain't.

Exactly! All the stuff you point out about how it's theoretically possible to use XML safely for security applications misses the bigger question of why you would do this in XML in the first place as it is far too complicated to make this a good idea.

Even namespaces can gum things up, so to be security conscious you even have to ensure that all your XPath expressions fully write out the namespace for the tags they will use each time!

For example, an XPath expression like /soape:Envelope/soape:Body

would need to be converted to instead read as:

/*[local­name()="Envelope" and namespace­uri()="http://schemas.xmlsoap.org/soap/envelope/"][1]/*[local­name()="Body" and namespace­uri()="http://schemas.xmlsoap.org/soap/envelope/"][1]

each and every single time.

1

u/masklinn Feb 26 '21

Definitely better, no contest.

To starts with, the xml spec has built-in security issues, things like DTD, and entity definitions, and encoding concerns, …

Second, XML is a way bigger and more complex format, even if you strip out the stuff from (1), so more opportunities to fuck up.

Third, because XML is so much more complex it’s more likely implementors will need to optimise it, which provides further opportunities for vulnerabilities.

And that’s before you account for the ancillary-but-really-required stuff like namespaces.

0

u/[deleted] Feb 25 '21

[deleted]

1

u/procrastornado Feb 25 '21

They see me trolling. They hatin'.