I ran a small benchmark to see how reliably AI agents generate Angular state-management code when they’re given library docs as context.
The setup:
Five Angular state libraries.
Eight prompts.
Three priming modes: no docs, llms.txt, and llms.txt plus a second guidance doc.
Six models.
720 scored outputs total.
Temperature 0.
Same prompts for each library.
The part I cared about was not “which library wins.” I maintain one of the libraries in the test, so the absolute ranking is not the interesting bit anyway. The interesting bit was what happened between runs.
In the first run, adding a 14KB llms.txt file improved generated code a lot for my library. It went from 49% correct to 91%. Good result. Or so I thought. Then I added a second doc as extra context and accuracy dropped to 87%. One model fell off especially hard. Another model seemed stuck at 80%, even with docs.
My first explanation was pretty lazy: “Context dilution.”
“Maybe that model just can’t do this library well.”
“Probably a model architecture thing.” Then I looked at the failures. The agents were not hallucinating randomly. They were copying bad information from my own docs.
A few examples:
byId() was documented as returning a Signal, but it actually returns EntityNode | undefined. updateMany was documented with the wrong shape.
One documented subpath import did not exist.
Deprecated method names were still being taught as canonical in multiple files.
A tagline had been updated in one doc but not in the others.
There were about 70 doc issues across four files. Some had been wrong for multiple releases. The API had changed, but the docs had not been re-grep’d carefully afterward. So I fixed the docs and ran the same benchmark again. Same models.
Same prompts.
Same scoring.
Cleaner docs. The score moved from 49% to 98%. Five of the six agents hit 100/100. The “context dilution” result also flipped. Adding the second doc now improved accuracy instead of hurting it.
So the problem was not that the context was too long. The problem was that the context contradicted itself. The model that looked like it had an 80% ceiling also reached 100/100 after the docs were fixed.
That was the useful lesson for me: for this kind of codegen, docs are not just user-facing documentation anymore. They are also training-time-ish context for tools people are already using. If the docs are stale, vague, or internally inconsistent, the generated code fails in ways that look like model weakness. But sometimes the model is doing exactly what you told it to do. You just told it the wrong thing. The benchmark made that visible faster than normal user reports would have.
The thing I’d defend here is not the absolute score. I wrote the library, the prompts, the priming docs, and the rubric, so the exact numbers are easy to argue with. The relative movement is harder to ignore.
Bad docs:
49% cold, 91% with one doc, 87% with two conflicting docs.
Cleaned-up docs: 98%, with most agents perfect.
For anyone maintaining a library that people use with Cursor, Claude Code, Copilot, or similar tools, it may be worth treating docs like executable surface area:
• remove stale examples
• grep for old method names
• test documented imports
• keep llms.txt consistent with normal docs
• avoid documenting “almost right” API shapes
• make deprecated patterns visibly deprecated everywhere
The boring work mattered more than I expected.
