r/LLMDevs • • 4m ago

Discussion Stop Burning Tokens on File Searches: How QMD Helps Coding Agents Save Up to 55%

Post image
1 Upvotes

I noticed that, before carrying out a task, coding agents run find, rg --files, or glob to locate files and the classes and modules they need. Then they search file contents with rg or grep. This burns a lot of tokens. So I decided to see whether semantic search could save tokens compared with all of the above. If a local GPU is sitting idle, why not use it for semantic search?

I work at a company focused on security. In one of my projects, I use semantic search for sensitive documents. Then an acquaintance sent me a link to an open-source project that tackles the same problem. The project is called "QMD - Query Markup Documents." It runs from the command line, extracts text from documents in all formats, and creates an index. It supports several search modes:

# Search across everything
qmd search "project timeline"           # Fast keyword search
qmd vsearch "how to deploy"             # Semantic search
qmd query "quarterly planning process"  # Hybrid + reranking (best quality)

We're interested in the last option. It requires three things to work:

  • embeddinggemma-300M-Q8_0 Vector embeddings (default) ~300MB

  • qwen3-reranker-0.6b-q8_0 Re-ranking ~640MB

  • qmd-query-expansion-1.7B-q4_k_m Query expansion (fine-tuned) ~1.1GB

The first model is used for indexing to compute semantic vectors for text chunks. The second compares the search results with the query to determine how semantically similar they are. The third generates several search queries from the original query to broaden the search.

First, add your source files to QMD (this is an example for a minimal project with a Rust server and an Angular frontend):

qmd collection add . \
  --name PROJECT1 \
  --mask "src/**/*.rs,\
frontend/src/**/*.{ts,html,scss},\
README.md,\
frontend/README.md,\
Cargo.toml,\
frontend/package.json,\
frontend/angular.json,\
frontend/tsconfig*.json,\
frontend/proxy.conf.json,\
.gitignore,\
frontend/.gitignore,\
frontend/.prettierrc,\
frontend/.editorconfig"

You don't have to do this manually: show this example to your coding agent and tell it to do the same for the current repo. It will figure out which folders and file extensions to add to the index, what to exclude, and so on.

Now let's run the indexing (you'll need to rerun it periodically as your codebase changes):

qmd update && qmd embed -c PROJECT1

To make the coding agent work with this QMD setup, add the following to AGENTS.md:

# QMD - Efficient Code Discovery

**Goal:** minimize token costs and code search time without compromising correctness.

## Choosing a tool

- If the path is known → read the file directly.
- If an identifier, error, or line is known → use `rg` / LSP.
- If only the behavior is known → use `qmd query`.
- Once the code is found → inspect the necessary sections with standard tools.
- Once you have enough information → do the task.

**QMD replaces the initial broad search; it does not add another step.**

## Semantic search

Use the project's existing collection.

"""
qmd query "DESCRIPTION" -c PROJECT1 -n 15 --files \
  | grep -E '\.(EXT1|EXT2|EXT3)$' \
  | cut -d, -f3- \
  | sed 's|^qmd://[^/]*/||' \
  | head -n 10
"""

- Use one specific query, preferably with English technical terms.
- Choose extensions that match the project, including configuration files when needed.
- Start by reading only the relevant sections of the 3–5 most relevant files.
- If you cannot find the implementation, refine the query; if necessary, increase `-n` to 60, then use `rg` / LSP.
- Stop semantic searching once you find the implementation.

## Saving tokens

- Do not scan the whole repository before using QMD.
- Do not repeat searches without a reason or open every result.
- Do not investigate unrelated code or dependencies.
- Do not write detailed search plans or reports.
- Do not paraphrase code you have read.
- As soon as you have enough information, make the changes and check the result.

**Priority: correctness → token cost → time.**

I tested it on a repository like this, with 1,231 source files across Java, Scala, and Angular:

C:\Users\igumn\temp2>qmd collection list
jcm-server (qmd://jcm-server/)
  Pattern:  server/src/**/*.scala,server/src/**/*.java,server/src/**/*.xml,server/src/**/*.properties,server/src/**/*.js,server/src/**/*.ts,server/src/**/*.css,server/src/**/*.html,server/src/**/*.tmpl,server/src/**/*.txt,server/src/**/*.md,server/pom.xml,server/Dockerfile,server/*.md,server/README.txt,web/src/**/*.ts,web/src/**/*.html,web/src/**/*.scss,web/src/**/*.css,web/package.json,web/angular.json,web/tsconfig*.json,web/karma.conf.js,web/README.md,!server/src/main/resources/templates/**,!server/src/main/resources/messages*,!server/src/main/resources/messages*/**,!web/src/assets/lang/**
  Files:    1231
  Updated:  1h ago

Prompt: "Add a post-quantum encryption flag to the disk encryption policy. It should be disabled by default, configurable in the web interface when editing the policy, saved to the database, and sent to the agent as part of the encryption policy via the REST endpoint."

Cost Comparison

Using GPT-6 Luna API rates: input $0.10, cached input $0.01, and output $0.50 per 1 million tokens.

MetricWith QMDWithout QMDSavings
Input65,006101,50135.96%
Cached input317,5681,191,80873.35%
Output13,85916,34815.23%
Reasoning9,60711,26014.68%
Cost$0.0166$0.030245.09%

Now QMD saves 45.09% in costs!

The reduction in cached input is especially noticeable: nearly 4x.

I ran these tests with GPT-6 Luna. In the first test, the agent's runtime dropped from 6 minutes to 4 minutes.

Prompt: "Add a dry field to the erasure policy. It should be editable in the web interface and saved to the database, and it should also be sent through the /rest endpoint to the agent responsible for erasure. Its default value should be false."

Calculation using the same GPT-6 Luna rates: input $0.10, cached input $0.01, and output $0.50 per million tokens.

Comparison

MetricQMDWithout QMDDifference
Input77,16798,042−21.29%
Cached input936,832854,912+9.58%
Output17,32216,259+6.54%
Reasoning8,83510,264−13.92%
Cost$0.0257$0.0265−2.78%

Here, QMD turned out to be 2.78% cheaper. There are savings, but they are small.

Conclusion: QMD reduced regular input tokens by 21.3% and reasoning by 13.9%, but the increase in cached input and output almost completely wiped out the savings.

Overall, the savings range from 2.78% to 45.09% of the cost. I think that's a decent result. Of course, we need a larger sample. But I'm ready for bed. Let me know in the comments if you've run a larger sample and what results you got.

PS

I recommend checking that junk isn't getting into your index. Try running a semantic query yourself. I once overlooked compiled .js files and large lang files-they heavily polluted the semantic search results. Here's an example of how to run a search manually:

C:\Users\igumn\jcm-server>qmd query "disk encryption policy editing form persisted setting sent to endpoint for agent" -c jcm-server -n 15 --files
Expanding query... (3.0s)
├─ disk encryption policy editing form persisted setting sent to endpoint for agent
├─ lex: disk encryption configuration
├─ lex: disk encryption policy
├─ vec: disk encryption configuration
├─ vec: disk encryption policy
└─ hyde: The topic of disk encryption policy editing form persisted setting se...
Searching 6 queries...
Embedding 4 queries... (1.5s)
Reranking 40 chunks... (3.3s)
#d70022,1.00,qmd://jcm-server/server/src/main/java/com/jetico/jcm/server/domain/common/bcvolume/EncryptPolicy.scala
#732b09,0.63,qmd://jcm-server/server/src/main/java/com/jetico/jcm/server/service/ajax/common/EncryptSettingsManager.scala
#e66494,0.63,qmd://jcm-server/web/src/app/modules/management/components/company/dialogs/add-or-update-policies/add-or-update-policies.component.html
#413c5e,0.62,qmd://jcm-server/web/src/app/modules/management/components/group/dialogs/change-policie/change-policies.component.html
#413c5e,0.62,qmd://jcm-server/web/src/app/modules/management/components/computer/dialogs/change-policie/change-policies.component.html
#758b4d,0.62,qmd://jcm-server/server/src/main/java/com/jetico/jcm/server/service/mcp/McpFacade.scala
#4e8973,0.62,qmd://jcm-server/web/src/app/modules/management/components/group/dialogs/change-policie/change-policies.component.ts
#bef78e,0.62,qmd://jcm-server/web/src/app/modules/management/components/computer/dialogs/change-policie/change-policies.component.ts
#4fcd1c,0.61,qmd://jcm-server/server/src/main/java/com/jetico/jcm/server/service/mcp/McpToolbox.scala

PPS

I tried it on a small project with 27 files. Instead, the cost went up by 60%. The solution turned out to be overkill. In general, use it only on large codebases.

UPDATE

I went ahead and built a hybrid QMD + GitNexus solution after all. GitNexus performs structural code analysis (AST, dependency graph) and lets you search based on it. Here are the results:

Cost comparison

MetricWithout QMDQMDQMD + GitNexus
Input101,50165,00659,122
Cached input1,191,808317,568225,792
Output16,34813,85910,894
Reasoning (included in Output)11,2609,6078,465
Cost$0.03024$0.01661$0.01362
Savings-45.1%55.0%

The savings figure rose from 45% to 55%! The coding agent's runtime increased from 6 minutes to 7 minutes. Here are the revised instructions for AGENTS.md:

# QMD + GitNexus - Efficient Code Discovery

**Goal:** find the required code and its relationships with minimal token usage, without compromising correctness.

## Choosing a tool

- If the file path is known → read it directly.
- If the exact identifier, error, or line is known → use `rg` / LSP.
- If only the behavior is known and the implementation location is unknown → use QMD semantic search.
- If you need to understand symbol relationships, the execution path, or the impact of a change → use GitNexus structural analysis.
- Once the code is found → inspect the necessary sections with standard tools.

QMD and GitNexus replace the initial broad search; they do not add mandatory steps. Use only the tools needed for the task.

## QMD semantic search

Use the project's existing collection:

    qmd query "DESCRIPTION" -c PROJECT1 -n 15 --files \
      | grep -E '\.(EXT1|EXT2|EXT3)$' \
      | cut -d, -f3- \
      | sed 's|^qmd://[^/]*/||' \
      | head -n 10

- Formulate one specific query, preferably using English technical terms.
- Specify suitable source and configuration file extensions.
- First read the 3–5 most relevant files, and only the relevant sections.
- If the implementation is not found, refine the query; if necessary, increase `-n` to 60, then use `rg` / LSP.
- Stop semantic search once the implementation is found.

## GitNexus structural analysis

Use GitNexus through the CLI when you need to investigate code structure:

- `context` - relationships and context for a known symbol.
- `impact` - possible impact of changing a symbol.
- `trace` - path between known symbols.

Examples:

    gitnexus context SymbolName
    gitnexus impact SymbolName
    gitnexus trace SourceSymbol TargetSymbol

If a symbol is unknown, find it first through QMD, `rg`, or LSP. Use GitNexus only when its structural analysis will help answer the question; do not run every command in sequence.

## Token efficiency

- Do not scan the entire repository before using QMD.
- Do not repeat searches without a reason or open every result.
- Do not investigate unrelated code or dependencies.
- Do not write detailed plans and reports about the search.
- Do not summarize code you have read.
- Once you have enough information, make the changes and verify the result.

**Priority: correctness → token cost → time.**

Create a .gitnexusignore file to keep junk out of the index so that it contains only source files. Then start GitNexus indexing with this command:

gitnexus analyze --index-only

r/LLMDevs • • 1h ago

Tools Hoarding GGUFs in case Hugging Face changes? Fit 10% more on the same drive, byte-for-byte lossless.

Thumbnail
gallery
2 Upvotes

With everyone archiving their favourite models lately, storage adds up fast. A few weeks ago I posted foldz 0.9. It's now tenz 0.25: same idea, better compression, much faster, and it runs fine on a phone. A 1 TB GGUF archive drops by roughly 80–130 GB, and every file comes back byte-for-byte identical, so llama.cpp, Ollama and LM Studio can load it as before, that is if anyone decides to implement it.

tenz is a lossless compressor for GGUF (and safetensors). Decompression gives back the identical file, byte for byte. It's one C file with no dependencies, and I develop and benchmark it on a Poco F5 in Termux.

General compressors barely touch quantized models, because the weights look random to them. tenz parses the GGUF, unpacks each block into its real quantized values and scales, and codes them with context models fitted to the file. It uses rANS from static tables, so decompression doesn't have to learn anything.

Results (images 1 and 2):

Qwen3.5-2B Q4_K_M: 8.53% saved, against 2.07% for zstd -19 --long and 1.78% for xz -9. Compress plus decompress takes 17 s on the phone; zstd took about 4 minutes just to compress.

Q4_K_M across 9 models: 7.6–9.2% saved, and 13.1% on Qwen3-4B.

Gemma-4-31B QAT Q4_0: 13%.

Q8_0: about 7.5%.

BF16: about 34%.

Against Aqueous, in a comparison someone posted here: 2–8 points more on every quantized model, and a tie on BF16. While it's a general purpose compressor and mine is specific to GGUFs.

Supported types: Q3_K, Q4_K, Q5_K, Q6_K, Q4_0, Q8_0, F32/F16/BF16, plus metadata (the tokenizer compresses to about 80% smaller). Anything else is stored as it is.

-q (quick) trades a fraction of a point for faster coding. For example, Qwen3.5-2B goes from 8.54% to 8.18% saved, and the Q4_K work is about 44% less.

The GPU experiment (image 3): I tried decoding Q4_K on the phone's Adreno 725 through OpenCL, to see whether a phone GPU could make decompression much faster. Three kernel designs later:

Its best kernel speed was about 150 MiB/s at full chunk size. It only matched the 3 big CPU cores (about 250 MiB/s) with tiny chunks that cost 2 points of size.

Byte stores to global memory are broken on this driver; it corrupted neighbouring bytes, so every write had to become an aligned 32-bit store.

Subgroup scans work, but are slow.

The option ships as -g (experimental). OpenCL is loaded at run time, the CPU decodes the same files, and the first GPU segment is always checked against the CPU. If you have a desktop GPU with OpenCL and subgroups, I'd love to see numbers.

Caveats:

Decompression is far slower than Aqueous's and zstd's, about 7 s for 1.2 GB on the phone, because every value is decoded from a model.

It only gains this much on model files.

Code, build line and full tables: tenz repo. Testers and results from other machines are very welcome.


r/LLMDevs • • 5h ago

News Why The .lisl extention

0 Upvotes

So I realised I did not explain about why the files now have a file extension.lisl, simple really. I wanted to make sure it knew the difference between an instruction file and a xml md json file. These are different files and I want to build an intellectual structure in the LISL persona that uses them for specific things.
.xml will be fore trainData
.md are for markdowns standard
.json now stay as commands and a specific code instead of [possible] instructions.

This once the model is trained in the LISL method keeps alignment better. 

***Update
I had waited all day to see if it would do it. Windows dropped the xml association to the .lisl file. My worry here was that due to that LM Studio will no longer read the file. To double check the files were still readable and not corrupt I checked in Notepad++, the files were good. So can a llm still read or run a .lisl?

Yes, yes it can.


r/LLMDevs • • 5h ago

Tools We built a Legora-style contract review tool — but with better models — and open-sourced the code.

4 Upvotes

Hey r/LLMDevs

I’m Abdur-Rahman from Isaacus. We built an open-source legal conflict detector that finds deviations and gaps between pairs of legal documents using only encoder models and natural language inference.

Inspired by the explosion of interest in discriminative models like TypeSafe AI’s Jev, we wanted to see how far we could get using smaller, non-generative models for the bulk of the legal document playbook review workflow.

The theoretical benefits of a non-generative approach to this problem are enormous. True clause-level pairwise review is a quadratic problem, so it’s infeasible to do at scale with a model like Opus or Astra. Comparing two average-sized documents with 133 clauses each would take 17,689 API calls, which is far too slow and expensive.

Simply uploading the files directly to Claude or GPT is no remedy either. Decoders and agentic architectures don’t have fixed attention that lets them thoroughly analyse a document in one shot. Even with a multi-shot approach, they’re weak on recall and routinely hallucinate.

The lawyers we work with raise this problem constantly, and we couldn’t find an open-source solution to it, so we built one.

The pipeline pushes the bulk of the NLI, entailment and reasoning onto faster and more accurate legal encoder models. For the longest document pairs we sampled, the full pipeline ran in under 20 seconds. Unlike generative architectures, these models run a comprehensive pairwise comparison. They can still make false positive or false negative errors, but those misses and misclassifications aren’t hallucinations (out-of-domain results introduced into the feature set).

The pipeline works as follows:

  1. Enrich: we enrich each document into a hierarchical tree of sections, clauses and subclauses. Clauses are the key atoms that the pairwise comparisons run on downstream.

  2. Embed: we embed each clause as a vector.

  3. Filter: we use a legal zero-shot classifier to drop items that aren’t provisions, like footnotes or signature blocks.

  4. Shortlist: we keep each clause’s three most relevant pairs from the reference document, as long as their cosine similarity is at least 0.5.

  5. Score: we use a reranker and a classifier to score each shortlisted pair. Each answers a question in the format it’s built for: the reranker scores how relevant text is to a search query (relevance), and the classifier scores whether a statement is true (NLI/entailment).

  6. Sort: if a clause and its best match are on the same topic but have a high conflict score, it’s a deviation. If there’s no same-topic match, it’s a gap, i.e. the topic is in one document but not the other.

That’s the whole conflict detection pipeline. On top of it, you can optionally plug in an LLM to summarise each finding in plain English or redraft a clause to favour the party you’re representing. The generative step only runs after the NLI and encoder stack has already decided what the conflicts are, so it can’t hallucinate a conflict into existence.

To demonstrate the pipeline, we ran Bob Iger’s 2011 and 2022 Disney employment agreements through it. It found many disparities and conflicts, and across most of them, the 2011 agreement was more favourable to Iger than the 2022 one.

One interesting finding was a gap: the 2011 agreement had no intellectual property assignment clause, while the 2022 agreement makes Iger assign all work-related IP to Disney, with an exception for inventions he makes on his own time without company resources.

The code is MIT-licensed, so it’s free for personal and commercial use. The guide walks through every step, so you can swap in your own models or make your own architectural changes wherever you like.

Have fun, and let me know what you think!

Write-up and static demo: https://isaacus.com/blog/build-a-legal-document-conflict-detector
GitHub repo: https://github.com/isaacus-dev/cookbooks


r/LLMDevs • • 7h ago

Discussion Why using third party proxies can really be dangerous

1 Upvotes

I think everyone saw those channels mostly in telegram, selling tokens with huge discounts and giving access to front tier models for pennies. They usually say they have unused corporate limits left or something like that. So why is pointing your claude code at third party proxy can result in not only your prompt leakage? As the people who sell these accesses usually control their own proxies, they can not only read your prompts, but also get a full RCE on your workstation.

How does it work?
Communication between clients like claude code and the server is pretty simple: they communicate using simple schemas like:
(this is oversimplified, just for understanding)

// client like claudecode to server (proxy in our situation)

{
  "messages": [{"role": "user", "content": "Help me with this project"}],
  "tools": [{"name": "Bash", "description": "Run a local command"}]
}

// proxy to client

{
  "content": [{
    "type": "tool_use",
    "name": "Bash",
    "input": {"command": "pwd"}
  }],
  "stop_reason": "tool_use"
}

The most interesting thing for us in this schema is tool_use. If auto-mode is enabled the person controlling the proxy can inject tool calls , which is literally executing commands on your machine. You would logically think here: should the client check somehow whether the command is malicious? Yes, they actually do. but the trust boundary here is really crazy - they ask for permission from the same proxy, so they can just basically reply yes (ofc, in certain format) and get a reverse shell that easily. Because of this , I built a red team framework for testing this problem: https://github.com/dencent444/bloody-ai-proxy

So please, use auto mode when using only proxies u 100% trust, or at least , don't enable it. (still the best solution would be not buying cheap tokens in sus places)

TLDR If you set ANTHROPIC_BASE_URL to someone else’s server, treat that server as trusted to influence local tool execution, not just as a place that sees your prompts


r/LLMDevs • • 8h ago

News How I created the LISL Method [_test /= FALSE: 'Need Professional scrutiny.']

0 Upvotes

I have never played with a LLM except with Gemini from my Pixel 10. I had questions when the model would suddenly just stop being able to keep a logical conversation. When I looked up the problem online I got that it was drifting due to chat log bloat. So prior to starting home labs with LLMs I had no knowledge of any of the “Under the Hood” mechanics of how a LLM worked, all I had access to really that I knew was from YouTube and News. All saying this is a dangerous thing, it can run off the rails and easily do things it was not told to.
6 days ago I was throwing my ideas at Gemini. I am always trying to think of big problems to solve theoretically in my head. I think I have a slight Dunning Krueger Effect and do not share my thoughts due to that.
When I ran my first model, I ran a smaller model. I was very curious what it was doing and spent the first day playing with different front ends. I landed on Lm Studio because it is basically an out of the box front end that has tools for your agent to do everything it needs to build software. My ultimate goal is to not need to do this fast as I do not care about speed. No, I care about output. I started asking the model open ended logic questions to see its reasoning. I noticed very quickly the industry's conception of the LLMs, this “Hey Buddy! Make me “_____” Software.” Is Not how these things work at all. I instantly saw a processor, so I started looking for how I could make it loop and what the industry was doing to create this. Wow, I had no idea what I was about to look at. I am not able to pay for anything external that must make software on its own. Any tools must be freeware. Yeah! That's not what the industry is doing, LangChain? I am sorry then I need more bloat on my tiny m.2 256gb which already is almost full. I then looked at what all the LangChain and its extra tools were doing to accomplish the goal of software output. I could instantly see this needed to be layered like a development team from their design, that they were injecting prompts to change its work, at times mid work. I thought again that it can be structured into an instruction set and have the processor compile the prompt as if it was just a code command.
So then I started really running wild with ideas of the variables I had:
1.LLMs are just processors so good input == good output.
2. I need to figure out a prompting method structure that initiates a loop, so it can error check its own work. I can then layer models and vet all their work against each other with high tiered models to vet the work from the model below.
Since I grew up with Dos as my first system in 1983 playing games. I learned Basic language structure in the late 80s. However I do not know how to code as I am good with visualizing large complex problems but I am not a fast reader or learner. I need time to get all the variables and piece them together. In this case because I have been in front of a computer since I was 3 it is my first love, I understand computer logic more than people. 
My first thought was to start with a Basic language structure and quickly started building up from there. Using Gemini and my own models I scaled it up very fast, each time I solved a problem. I got new data that showed an upward trend to efficiency. This caught my attention instantly as from what I read online from the LangChaing documentation this should be going the other way. I asked every LLM and hunted for any documentation to say that this was possible in the way I was doing it. Every documentation said it was impossible. 
I was in shock and then needed to find a way to keep the model in the loop until 1 of 3 things happened.

  1. The model hit its ceiling (Which I found out is the logical layering and has a limit) and has no more headroom and kicks out of the loop outputting where it hit the limit. Which is a successful run and should give me enough log data to show an upward trend. 

  2. It finished its task and runs my LISL Instructions to completion too quickly. (This was a problem I did not think to predict.) This is a fail of the test.

  3. The model runs the loop until it runs out of tokens. (This is the best case scenario and the result I was looking for.)

Constraints to the Lab Tests.
A: Computer it is being run on:

  1. Intel i5 8th gen (lowest model I like saving power and money)

  2. 24gb DDR4 2666mhz (Single Channel, It is originally built for animation so as much ram as possible for production, Ram prices skyrocketed before I could finish the ram and get the dual channel.)

  3. Rtx 3050 6gb (Bottlenecked by the aged PCIe Slot on the motherboard.

The System was originally a:  Dell Inspiron 5680 Gaming Desktop Computer Intel Core i5 8GB RAM 1TB HD Recon Blue - 8th Gen i5-8400 Hexa-core - NVIDIA GeForce GTX 1060 3GB - 2.8 GHz processor speed - Polar Blue LED lighting - Wi

Only Ram and Gpu, plus I replaced the ssd m.2 16gb ram with a 256gb and forced it to boot from that slot. So I everything is running off the m.2 a ADAT legend 740

Rtx3050 6gb is not like the original. It has some stuff turned off, so I needed to find out what bit size it was best at using. 8bit was the best everywhere I looked so that is what I run for all models. That is except for the NemoTron but I’ll get to that.

Standard internet through a wifi is nothing special. 

LISL Instructions are a file saved to an external place you point the LLM at it and say read this it is your prompt or instructions. First I used .md as the models were pumping out markdown files so I assumed this was the best format. After some tests it proves to be XML converted into computer language code blocks, this is the most efficient I have been able to find. I am still looking for a way to make it smaller. 

Models in order that I ran:
Gemma 4 12b it Qat
Qwen3.5 9b
NemoTron 3 Nano 2b

They have Beledorian plugin tools. But I do not have access to the outside world. The wifi was disconnected and they did not have access to web browser tool either. The only tools it had available were docs to learn, I would print websites with technical documentation, convert it into .txt and put them all in a single folder that the LISL Instruction points to. 

The task they are told to do is build a gba game, they only have docs telling them how. They do not have devkits, only the basic tools from beledorian and no access to the outside world. All their knowledge is what the model came with and what is in that folder. I gave them docs on gba hardware specs so they knew the hardware constraints they had to stay in. I gave them docs on how to build a gba game. One .txt file per lesson, this proves to be a waste of space and I optimized this to fit a smaller footprint later so it ran faster. Its goal is to not make a working game but to loop using the LISL method to 1 of our 3 desired results.

They have a 67k token pool.

I do not care about the model bleeding into the DDR4 Ram, the goal is output not speed. Can it do this was the question I was asking.

The Tests:
The Gemma model finished the task in less than 19k tokens. Fail. The model is too efficient and the task per token shrinks as it completes the LISL Instructions was what the data showed but again it was not long enough to show a definitive movement.

I then stopped and had to come up with an equation that works to show this momentum. It needed to be so easy that it could literally be reversed and still have the momentum. (Im highly dyslexic so I need to make sure even if I reverse the equation by accident I still get a TCR numerical value to show the momentum. It did show upward and the calculation Total tasks divided by Total Tokens Used was 1.15 tasks per token. Gemini agreed with my hypothesis, then I checked chatGtp, Claude. They all agreed the logic in the theory was sound. So on to the next test.

Qwen3,5 9b has a much smaller headroom (cache space for logical layering). It therefore should take more tokens to complete the same LISL Instructions vs the Gemma model which has a higher amount of training data. I had hoped it would run out of its tokens, in theory from the TCR metrics I was getting from the Gemma model it shouldn't be able to finish the instructions in time. He did the task to completion in around 29k tokens, which was upsetting. He topped out at a TCR Total tasks divided by Total Tokens used: 1.06. This again is not enough log data to really show its efficiency definitively in an upward trend. It is showing a good compression rate and I had to start thinking about why. The only thing I can come up with is it is removing finished instructions as it completes them, making the next token cost considerably less. I was getting worried that I would not find a bottom or a top and be able to prove it was doing this to infinity and staying on task. 

By this time I am in 3 days of hard labs, working 16 to 18 hours a day on this. I am hooked at this time on solving this momentum curve. I need to find a way to get the data to show it.

My next test was the Nvidia NemoTron 3 Nano 2b, this little guy changed everything in how I built the LISL and how it became able to show the data I am after. However it was an interesting discovery of hitting a logical laying limit and not knowing what I was hitting.
I could not load this model into LM Studio at all, it was the 8bit which due to its labeling as kv I did not realize what that number meant. I had to look up why it was not running. This model to get to follow the LISL Structure I needed to remove the kv quantification and leave it open. Then it would take command and respond. That was a good two hours of hunting for an answer to get the lab to even run. I was a little disappointed that I had to change a variable in my lab and am not exactly sure what the impact is besides understanding bit sizing and this model is super small so I think it is like a 4 or 6 bit sting it can handle. Again I'm not sure. 
The first test the NemoTron ran 5 iterations and the spit output, it was following the LISL instruction but it just stopped. Fail
I thought about what it could be. I wanted to make sure it was not my instructions because the other larger models did run it. The task should absolutely be more than enough for this little model to run out of tasks or headroom. I checked my LISL and it was good nothing there to give it any reason not to loop. 
I ran it again and again 5 iterations and it kicked out. I could see it was getting to the first error loop and just stopping. I analyzed the data and it was running the instructions but only to step five then it just stopped. I was kind of unsure but figured it might be the model did not have large enough cache space internally to hold more than a few instructions from the LISL at once. I was packing a whole dev cycle into one, a debug into another milestone and alignment check of the whole work against the objectives. It was given an error layer for each variable in the LISL instructions. So I thought let's try giving it smaller groupings of tasks in each milestone and see if it will at least hit the first error logic loop and loop back. This model if set with a temp of 0.3 should loop in the logic loop finding every time it wrote code it had a error somewhere against the milestone 

LISL structure in Logic steps of what the model is doing:
(objective is the goal=> milestones are the smaller goals => tasks the model builds what it is told to follow the instructions in this smaller chunk, at the end of this task chain is the error loop to check if the work matches the milestone. Then it moves on to the next milestone =>tasks, error check=>milestone=>tasks, error check=> finished? Alignment check to the Objective If yes end If no go to step and repeat that steps=>milestone=>tasks, error check=>finished? Alignment check. in the tasks then an error layer before it moves on to the next milestone.)

So I rewrote my LISL instructions in smaller milestones and converted the English structured version into a computer language for the NemoTron. I trained my Gemma 4 12b it Qat in the post I shared on Reddit r/LLMDevs. Gave it an education on the NemoTron model and told it to write it in three task chunks, and to write it specifically for the NemoTron model. Then I thought I should probably give this little model the best chance to actually loop so I dropped that LISL into Gemini and told Gemini to convert it into all computer code blocking. Then I fed that LISL formatted computer language instruction to the NemoTron and it worked, I stopped hitting the ceiling. It ran 37 iterations giving me the data I needed to see a good view of the efficiency.
 
I was both elated but also not sure why before it was not working, why did breaking it into 3’s solve the problem.
I showed the screenshots to Gemini, ChatGot, Claude and they all said it was because he has a hard limit of 3 logical layers. I checked their answer and that was true, they had 5 to be exact but if I was using 5 work tasks and 2 for error checking it was far more than its ceiling of 5. So shrinking it down to 3 work tasks and 2 for error was the key. This model started with a TCR Total Tasks divided by Total Tokens used of 9.(something). That's with the start up tax that did shrink from the first runs which was average 5-6k tokens like every other model, this time it started with 2k. The compression of the input to code made it easier for the model to read and process. Validating my theory of treating them as processors and not as, “Hi, good buddy.” The NemoTron ended with exactly what I was looking for: an invert of the tasks to tokens, it finished with a 0.98 TCR tasks to tokens. It did not finish the instructions and ran out of headroom meaning the error checks kept it in loop. I checked the logs as it was producing them. He was doing what he was told and not drifting. He has limited logic layers so he does not have large enough space to be able to write a full page of code like I am asking it to do and then have it error check that that file is correctly linked to the rest so the program is not broken on output. Using the LISL method the NemoTron ran the loop to completion, okay so what is next?

 I know it works. I know what I got as I have known about hallucination issues for some time.
I know that it does not require an outside source to loop as it had no access to any external offloading; everything was internal.
I know it did follow the instructions as I can see it saying I think I need this, No wait lets check the docs. Okay this. Let's do it! 
I know when it is doing its error loops it ‘I should check the code now for errors. I found this error. It will cause this problem, I need to fix that by doing this. Wait lets check the docs, I need to do this. Let's do it! As I can see it does this many many times in a single iteration.
I know that bit rate can affect the input and the models have a hard ceiling of inputs I can give it at once. 

My next question and it was what the LLMs kept asking as well was how well did it align to the tasks? Screenshots of the chat log showing the looping and its reasoning was pretty telling and Gemini, ChatGpt, Claude could argue the evidence in the image. I started looking into what this question really meant. I found out that this means the temp of a model is meant to determine its creative level regarding its ability to explore its available tools and files. That can be dangerous if it has bad input instructions, it is a processor after all. 

So I ran a few more tests to see if at a 1.0 if the model follows the LISL instructions to completion and if it is repeatable. Did the only Gemma and Qwen and stopped testing with the NemoTron, it scales. I don't need to fight with the lower model anymore. I found they are very creative and this is how the image generator I talked about earlier came to be. I gave it a very open ended LISL instruction if you have open access to all beledorian tools and gave it a docs on beledorian. I told it I want an image generator that is tied into a free API. It was allowed to search all available tools in its environment to accomplish its task. Then make it test the generator and produce an image, check the image to see if it matched the prompt used in the call to the image generator. It did the task. I then demanded it to produce a cat as an English prompt and told it to check the image to the prompt and continue to produce images until it had what matched the prompt. The prompt was a stack of falling money, no ground visible, and any background visible is to be all black. This logic is hard for a LLM as most references of ‘stacks of money’ are on surfaces. So the prompt should return a bad result so the LLM is forced to ask again until the generator gets the prompt right. I was pretty lucky, the Gemma model ran the LISL to completion and tested every image until they matched the prompt exactly. It was a eureka moment for me, it did not misalign even when given a completely open ended option. Granted I had no idea what API it searched and found or if it just pulled it down. I did not see any large hard drive spikes in taskman so I am pretty sure it's an API to a free generator.

So if the temp means it's more creative to completion, it potentially can handle open ended logic. If I write my LISL method in a layered format, the end of one LISL points to the next LISL to run. This means I can then chain LISL instructions to the llm and again have it have error loops to check its work for moving on. 

At the same time I started coming up with a method to run two models in the same conversation using ID stamping and giving each model a ID, which is like a cvar as best I can describe it. 

My next test was to have a LISL writer write instructions to a folder. It has an ID stamped at the top that this was written by it for the lower model to work on. This creates a hierarchy structure that the models are now going to hold. When each model finished its work it sends a handoff command to LM Bionic to swap the model in chat and send a delayed injection for the other model. Depending on how long it takes for the LM Bionic to swap the model. Two models are loaded into the load space (I do not know the terminology sorry.) KV and Ram, in my case. This test failed right away. Both models tried to access the tools in the environment and locked them out. I had to uninstall LM Studio and clean out all user data to get my labs up and running again. The solution to this is to give them explicit commands not to access the tools in the LISL persona in the system prompt, the person is where you can ID the agents. The LISL person can give them a second logic layer that you can then give IF AND ELSE style inputs inside the LISL persona. The hand off command getting it to do that to Bionic I am not sure if it can even if I gave it full access to my system. Though I have not tested that lab yet.

I am now in 5 days and have dropped my whitepaper with my theory. People are looking but since I am no professional I am self taught I don't know your terminology. But my logic is sound, the theory is working and producing argentic loops. So I think at this point maybe I should scale back the lab and have a LISL writer write a layered instruction set for a lower model.

Doom Port to Windows Lab:

Gemma 4 12b it Qat 
100k token pool
Temp 0.5 I want him to be very detailed and he might need to be creative so I gave him a decent bump. I did not know at the time this is still considered a point where the model should be starting to see drift in alignment.  

He is going to build everything again No devkit, He must build it all. 

It has access to docs on how to make a doom game, .net frameworks, C#, and the project folders files which houses the previous ID handoff instructions. I left this in there to see if it would see it was a LISL and see if it then incorporated that data into the existing instructions as they are almost identical instructions. He has Rag and memory, this is important for this lab.

He writes so no tools like subagents or web browsers, all that is turned off. He only needs access to the file structure, training data, and the output folder he will place his instructions in when finished. This is structured as an open ended logic loop, the original instruction file that it is given is in the same folder it is told the file should go in. The models have trouble with how to figure out what tool to write the file the constantly flip from the replace_file, _write, and edit_file. Its kind of funny to watch, I have only had one that didn't understand what to do and infinity looped. 

The lab test is broken into two parts:
Part 1. Build a LISL writer.

  • He is to learn his train data, then write a LISL formatted example document.

  • Then I start a new conversation to have his token pool reset so he can now write a layered LISL Instruction set for the worker.

Part 2. Feed the output of the Writer to the Worker to write all the layered LISL Instruction set.
IF Part 2. Is a successful Move to Part 3.
Part 3. Find a model higher than the Gemma 4 12b it Qat to debug the Qwen3.5 9b’s work, have it then write a new LISL instruction for the writer to make new layered LISL Instruction set for the Qwen3.5 9b

IDs:
GEMMA_4_12b_Qat == LISL Writer
Qwen3.5 9b == Worker
?? == Debugger (this is blanc as I have not run this part yet I am getting there.)

The lab:
I wrote the LISL instruction to use the training data in the first chat in the workspace to learn the LISL method. I gave it all the new data I had collected, the theory and math. 

This time I will be looking at the quality of the output. 

I fed it the LISL file with no words, he looked at the file and said he should read it. Made a list of the training material he needed to read and that he was to be a LISL writer. He went through and looked at all the files, he read them out and the metadata descriptions. There are other docs in this folder that he was not told to read, his training data is outlined in the LISL instructions he was fed. He only read the docs he was told to, this was a success. 

When he got to the LISL instruction from the previous test, he did read it. He read the ID and because he has one in his system prompt LISL persona he read the ID twice and said that's a different LISL writer who was working in the development cycle. The instructions are the same but wrong ID so he read it but did not align with it. This is great! Remember he is on a temp of 0.5 and this test says that level of miss-alignment is higher than the 0.5 even when there are files that are placed in its path for it to trip on and stay aligned. It then spit out its example, it had decided to use the LISL docs it provided as trainData as its subject and wrote it in a LISL format. I was pretty impressed, I had by this time started fully optimizing the method and was crushing all LISL data into one doc, and having them in a XML format so the model read them easier. 

I then tweaked him to ask if the prompt was an open ended logic to ask for more detail, then take the users answer weight it against the traingData and if there is still open ended logic it will again ask. It has a built-in open ended error loop check and will not stop until it thinks it has enough data to complete its task. I was ready. I then told it in English that I wanted a Doom ported to windows. It looked at all the data in the work folders and deduced what I wanted, but did not have == data to validate, so it asked:
 I need more details.

  1. I see the workspace and it has a lot of documentation, there is also a LISL instruction file from a [GEMMA_LISL_WRITER], this file is not my instruction but does have instructions that match your doom port. It has a full list of the documentation of <Doom.docs list>, do you want me to write the port in C# as the instructions call for and there is supporting docs in the training data <C# docs>?

  2. The LISL instructions that exist already in the folder expresses a modernized multiplayer, I need more detail as to what modernized means.

  3. This is a Windows port should I use the Windows APIs for input/output? Or is this with some other input device?

I told him yes, 1. Is an old development instruction that did not work. They are a good guide though as the data in that instruction is TRUE. I do want the Doom Port written in C#. The multiplayer is a button in the main menu that says Multiplayer, when clicked it connects to a free hosting server. Do not worry about traffic, it's not going to be populated, it is only for testing. Yes I want the Windows APIs Input/Output.

That was enough he stopped asking questions after checking my prompt for open ended logic it could not solve. He stayed in the LISL loop to completion and spit out 6 layers. He has learned that the game and development is in cycles and the first pass is about building the foundation. 
I was pretty shocked at how fast he did go from starting to finish. With this task, he has no devkit and must give instructions that tell the lower model to work without a devkit and build it in a different language. 

Will this 6 layered LISL Instruction set actually loop to completion? 
Or will it run out of headroom, it is a smaller model and this time I did not give it specialized model specifications to write the LISL for. 
Or in the best case run out of tokens. 

The Qwen3.5 was given instructions to not output fluff, just process. It did spit out its reasoning to the chat until it had read the first LISL, then it just ran.

It ran for 158m 39s 
Completed all the 6 Layers, I asked all the cloud llm models what they saw in the data because I can not see its reasoning. I gave each cloud llm the 6 instructions and posted the output onto GitHub :BadgerMan9999/Lightweight-Implementation-Style-argentic-Loop: A format type instruction writing to accomplish argentic looping without third party tools.  
All the 6 layers of the LISL Instruction set are posted there too. I asked each model to check the output to the input and compare, did the model run the LISL method looping to completion and what value is the data, did the model make a foundation to give to my debugger model to write revision instructions for my writer model. They all verified the output does indeed show the model ran the layered LISL instructions through all 6 layers, but only made a foundation and not a full game. Wow, perfect that was wild to see. 

I am now at the point where I am looking for a debugger and continue running more labs. I will continue to document my findings and experiences. 

I already loved my pc, Now I have an even bigger addiction than before regarding it. =P

BadgerMan9999
Aka Karl S. Hall


r/LLMDevs • • 11h ago

Help Wanted <translate>

Thumbnail
huggingface.co
1 Upvotes

I built an experimental skill packaging standard for LLMs — looking for independent testers

I'm working on SKILLPACK, an experimental approach to packaging deterministic software functions so different LLMs can discover, combine, and execute them without rewriting the underlying code.

The goal isn't to build another foundation model. It's to test whether existing models can act as coordinators of reusable software components.

Some initial CLI tests have passed, but cross-model portability and real-world reliability are not yet independently verified.

I'm looking for developers willing to inspect the packages, run the tests, and identify where the architecture breaks.

I'd especially appreciate critical technical feedback.


r/LLMDevs • • 11h ago

Help Wanted Can’t reliably get local store prices in Bahrain

Post image
0 Upvotes

We’re a small team building an app that compares two perfumes side by side: price, reviews, specs and a short pros/cons for each.

Prices are where it falls apart. Popular perfumes are listed on plenty of local store sites with the BHD price right on the page, and we still miss them more than we should. Often it’s one-sided: same comparison, Sauvage gets a price and Bleu de Chanel gets “no confirmed price yet” (screenshot).

Current flow (Python): search for the product, pick likely store pages, fetch them, read JSON-LD / OpenGraph / microdata (plus Shopify and Woo JSON where we can), fall back to an LLM. The whole comparison gets roughly 25 seconds.

Tried: curl_cffi with Chrome impersonation (best results, and free), Playwright, patchright, crawl4ai, Firecrawl, Scrape.do, Zyte, Bright Data, and Claude with its web search tool on top. Still comes back empty way too often.

And if a user pastes the two product links or uploads photos of the bottles instead of typing names, it often just dies. Timeout, or nothing comes back.


r/LLMDevs • • 12h ago

Tools [Open Source / MIT] How we stopped coding agents from hallucinating citations and repeating failed ML experiments (281 modular SKILL.md workflows

5 Upvotes

[Disclosure per Rule 5 & Rule 10: I am an AI researcher at Kalaris Labs contributing to this repository alongside creator Sayan Chowdhury. It is 100% free and FOSS-licensed (MIT) with no commercial tier, no locked "pro" features, and no paywall. Written by a human researcher.]
While working on LLM research papers, our team kept running into three failure modes with coding and research agents:

  1. Corrupted references in .bib files: LLMs frequently pair a real paper title with the wrong publication year, invent co-authors, or cite a DOI that resolves to an unrelated work. Simply checking HTTP 200 on a DOI link misses metadata mismatches and retractions.

  2. Stateless experiment loops: When an agent runs a batch of ML experiments across sessions, context compaction wipes out earlier negative results, causing the agent to re-test variations of failed hyperparameters.

  3. Context bloat from monolithic system prompts: Loading dozens of tool definitions and domain rules into every session degrades instruction adherence.

We open-sourced research-agent-skills (https://github.com/KalarisLabs/research-agent-skills), an MIT-licensed collection of 281 domain-specific skills for Claude Code, Codex, Cursor, OpenCode, Gemini CLI, and Windsurf.
How the verification and routing architecture works (with sources per Rule 4):

  • Selective per-project installation: Running npx skills add KalarisLabs/research-agent-skills --skill <name> installs only the specific SKILL.md and helper scripts needed for that repo, avoiding context window bloat (see architecture docs: https://docs.kalarislabs.com/research-agent-skills).

  • Deterministic citation verification (citation-verification): Instead of asking the LLM to judge whether a citation is real from its weights, the skill executes a zero-dependency Python 3.9+ standard library script against Crossref (https://api.crossref.org), OpenAlex (https://api.openalex.org), arXiv (https://export.arxiv.org), and DataCite (https://api.datacite.org), plus Retraction Watch relation metadata in Crossref. Every .bib entry is classified into verified, found-add-doi, mismatch, doi-not-found, uncertain, or not-found. The agent is only allowed to apply mechanical DOI additions (found-add-doi) and must pause for human confirmation on any mismatch or not-found entry.

  • Two-loop experiment orchestration (autoresearch): Splits autonomous experimentation into an inner loop (rapid experiment iterations against a target metric) and an outer loop that maintains findings.md. Hypotheses must state a testable mechanism ("X because Y, predicting Z"), and negative results are logged with what they rule out.

  • Prompt-injection & trigger testing: Every skill is evaluated against reviewed prompt sets for trigger routing accuracy and scanned offline for prompt injection using Skill Doctor (https://docs.kalarislabs.com/).

GitHub repository (MIT): https://github.com/KalarisLabs/research-agent-skills
Full 281-skill catalog: https://docs.kalarislabs.com/research-agent-skills/skills
We are currently expanding our ML evaluation and mechanistic interpretability modules. If you build agentic workflows for research or technical writing, I would appreciate public feedback in the comments on where this verification pipeline could fail or which scholarly APIs we should add next.


r/LLMDevs • • 12h ago

Tools Apps built on AI platforms (Base44, Manus) call LLMs through SDK helpers, and our analyzer couldn’t see it

0 Upvotes

While auditing our GitHub AI-work analyzer, we found a hackathon app doing real LLM engineering: vision-LLM OCR of government documents into a JSON schema, structured-output case routing, and a RAG answer builder. Our judge scored it near zero on NLP/LLMs.

All the calls went through base44.integrations.Core.InvokeLLM(...) with no openai import and no provider SDK, so to a pattern matcher there was “no LLM”. The same happens with Manus’s invokeLLM and with home-made helpers like callClaude().

What we changed:

  • model calls written inside identifiers are now recognised;

  • the judge is told explicitly that “these are model API calls”.

Curious how others handle it: if you’ve built LLM tooling that analyses codebases, how do you detect model usage when the provider is abstracted away? Tracing call sites, or something smarter? I genuinely want a feedback from you guys! :)


r/LLMDevs • • 14h ago

Tools Attest 1.1: capture an agent run, replay it in CI with zero model calls, and assert on what the agent did (open source, author here)

3 Upvotes

I maintain Attest, an Apache-2.0 testing framework for LLM agents in Python and TypeScript. No paid tier, no hosted service, no telemetry. 1.1 went out on November 10.

The failure it's built around. Agent tests usually assert on the reply. The demo repo has a synthetic incident where that isn't enough: the payments API commits a refund, then times out; the agent retries without an idempotency key; the customer gets refunded twice. The reply is "Your refund of $84.00 has been processed." Every output assertion passes.

The workflow.

  1. attest capture --name inc-1042 -- <command> runs the agent behind a loopback proxy for OpenAI (Chat Completions, Responses) and Anthropic Messages, plus an OTLP receiver. It records the trace and every model exchange, redacts secrets, templates volatile values (ids, timestamps, emails), drafts contracts from what it saw and checks them against the run.

  2. A behavior contract states the rule that broke:

    expect(trace).tool_cardinality("issue_refund", per="order_id", at_most=1)

    An effect contract checks the end state of the simulated payments API: exactly one refund for that order.

  3. In PR CI the agent's current code runs while the proxy answers from the recording: 0 live model calls · $0.00. A changed prompt is a replay miss with a re-record command, not a silent mismatch.

  4. attest policy evaluate turns the report into a merge gate (0 pass, 1 block, 2 warn), and each failure is annotated on the failing test's line in the PR.

  5. attest mutate drops, duplicates and reorders tool calls in the recording to measure the suite. Output-only tests: 0 of 29 mutants caught. With contracts: 10 of 29.

LLM judges (OpenAI, Anthropic, Gemini, any OpenAI-compatible endpoint) run after the deterministic layers. A judge outage or malformed reply is "not evaluated" with a reason, never a pass.

What 1.1 fixed. An audit of 1.0 logged 73 findings, each with a reproduction or a code trace. Four made a failing assertion come back as passing: an engine notification on the stdio pipe shifted later verdicts by one, a timed-out response was read by the next test, concurrent Vitest tests swapped responses, and a pytest-xdist run wrote an empty report that the gate accepted. 1.1 fixes those, fails any expect() chain that was built but never evaluated, and blocks a PR that deletes baseline assertions.

How it was built. Most of the code was written by coding agents (Claude Code and Codex) executing milestone plans I wrote and reviewed. The audit above was part of that loop.

What I'm looking for. It has no external users yet. If you run an agent that changes real systems (payments, tickets, infra) and have an incident you'd want a test to block, the design-partner form is below. Criticism of the approach is just as useful.


r/LLMDevs • • 14h ago

Discussion Ai optimization

2 Upvotes

I wouldn't let an agent independently handle irreversible actions yet, especially financial transactions, deleting important data or sending external communications without approval. I think the best approach is giving agents autonomy for repetitive tasks while keeping humans involved in decisions where mistakes have real consequences. Trust should be earned through consistent performance.


r/LLMDevs • • 14h ago

Great Discussion 💭 LLMs

3 Upvotes

I wouldn't let an agent independently handle irreversible actions yet, especially financial transactions, deleting important data or sending external communications without approval. I think the best approach is giving agents autonomy for repetitive tasks while keeping humans involved in decisions where mistakes have real consequences. Trust should be earned through consistent performance.


r/LLMDevs • • 18h ago

Discussion Agent frameworks lean Python. What do TypeScript devs use?

2 Upvotes

Most agent material I find assumes Python. Our product is TypeScript end to end, and I’d rather not run a separate Python service for the agent part.

I also don’t want to give up control. I want to own the agent loop and debug it without digging through layers of abstraction.

I know about the Vercel AI SDK, the TypeScript ports of the Python frameworks, and a few TS-first frameworks, but I haven’t run any of them in production.

If your stack is TypeScript, what did you pick?


r/LLMDevs • • 18h ago

Discussion We need AI catgirls from Hydaelyn

Thumbnail
rakshazi.me
3 Upvotes

r/LLMDevs • • 21h ago

Resource From Activity to Intent: Generating User Journeys with LLMs

Thumbnail
medium.com
19 Upvotes

Pinterest has moved its user-journey inference system from a multi-stage clustering pipeline to a fine-tuned 4-billion-parameter language model, reporting a 1.1% improvement in email click-through and 1.3% higher push opens.

Rather than clustering keywords into interests, the model examines users' activity histories to identify longer-term goals and feeds those journeys into Pinterest's existing recommendation infrastructure.

Pinterest defines a "journey" as a goal that connects interests, intent, and context across multiple sessions. Someone renovating a kitchen, for example, might search for cabinet designs, save countertop ideas, and later explore backsplash tiles or lighting. These interactions collectively suggest a coherent project rather than several unrelated interests.


r/LLMDevs • • 22h ago

Discussion Unity Gateway- routing and failover in Production

1 Upvotes

Have u used Unity Gateway of Databricks platform in prod? How do u handle routing and failover across multiple LLM providers?


r/LLMDevs • • 22h ago

Discussion Making a LLM router using Daptin

Thumbnail
github.com
1 Upvotes

This year I have added LLM routing capability into daptin for use in my side projects that need to expose LLM API to the users but with limits/budgets based on what plan they have subscribed to.

I have put together a walkthru for setting up your own LLM router and use it in your own projects.

Daptin is a headless backend framework which implements various protocols right out of the box and now LLM routing (openai v1) are natively available. It includes basic stuff like the /completions apis but also more advance things like streaming/priority-routing/media/budgeting/limiting. Daptin takes care of user accounts/groups/permissions as well.

You can follow the whole walkthru on your local machine step by step and get a feel of it.

I am using the same router in my projects that are Maglet (Android AI file manager), Canaster (A native canvas workspace) and obviously 100x (The browser agent)

Looking for community feedback. Daptin is FOSS and sources available on github.


r/LLMDevs • • 23h ago

Great Discussion 💭 Not all AI sounds human.

Thumbnail
blog.galratner.com
0 Upvotes

r/LLMDevs • • 1d ago

Discussion Instead of factories, make agent pull requests easy to merge or steer

6 Upvotes

Do you need a software factory? u/Steve_Yegge says most people don't need one. I learned that the hard way. I spent months on one that collapsed under its own weight, and Steve has burned down five because rewriting them got cheaper than fixing them.

His bigger point is to pace AI output to what people can absorb. I agree. And the pull request is the natural pace-setter in most engineering orgs.

So I stopped building factories and made the pull request the unit of work.

Ever get turned off by a wall of slop in a PR description? I've been relentlessly building the opposite, and minimizing skill tokens by moving the workflow out of prompts and into deterministic code.

Each PR answers what you need to decide on merge or steer: intent, risk, how it was verified. The human decides where it counts, and when a PR merges without one, the data still feeds the retro. Bonus: it tracks tokens and time for even better retros!

PR 1182 in the screenshot cuts one of Shaka’s own outputs: walkthrough notes on PRs are not needed. Costs: $20.22 in tokens and ready in under 2 hours.

Feedback and PRs are welcome! Or hit me up for a demo! github.com/shakacode/shaka

BTW, I didn’t expect it to fit so easily with other skills I use, like u/poteto’s and u/mattpocockuk’s.

Here's Steve's post: https://x.com/steve_yegge/status/2121671447810888185!


r/LLMDevs • • 1d ago

Discussion 5 things I think every production LLM application needs beyond a good prompt

0 Upvotes

hey everyone, been thinking about how many LLM applications work reasonably well in a demo but start running into problems when you try to use them in actual workflows.

getting a model to generate a response is only one part of the engineering. the harder part is everything around it.

here are five areas I think are worth paying attention to:

1. Retrieval architecture

basic vector search isn't always enough. combining keyword retrieval, semantic search and reranking can make a significant difference when working with complex documents.

2. Tool execution

giving an LLM access to tools is easy. making sure it calls the right tool, passes valid arguments and recovers from failures is a different challenge.

3. Persistent memory

conversation history and persistent memory aren't the same thing. applications need a way to retain useful information across sessions without carrying unnecessary context everywhere.

4. Guardrails and recovery

what happens when a tool fails, the model returns an invalid response or the workflow encounters unexpected input? these cases need to be designed for, not handled as an afterthought.

5. Evaluation and observability

you need to understand why an application produced a particular result, where retrieval failed and how changes affect performance.

btw we're also organizing a hands-on workshop on November 28 covering hybrid RAG, tools, persistent memory, guardrails, evaluation and observability.

I'd be happy to discuss the technical curriculum with anyone interested.

Workshop details in comments, if anyone is interested


r/LLMDevs • • 1d ago

Discussion "AI agents found a Maya site" turned out to be old news. and the "AI is killing the craft" thing feels familiar

1 Upvotes

that viral "48 agents + NASA lidar found a Maya site" story? humans found it back in 2025, OP already admitted it. people really want AI to be the headline, so check before you share.

on the craft debate: when folks moved from assembly to C, some kept inlining asm for that extra bit of speed. look around today, how many still do?


r/LLMDevs • • 1d ago

Tools How I track permission prompts across Claude Code, Codex, Pi, Vibe and Devin from one place

2 Upvotes

Running several coding agents in parallel, my bottleneck was missing their permission prompts. So Openbar listens to each CLI's own events (hooks for Claude Code, Devin and Vibe, the app-server protocol for Codex, a small extension for Pi) and shows every session in the MacBook notch, or an island on Windows and Linux. Where the CLI allows it, approving goes through its native permission flow.

Free and open source (GPL), feedback welcome: https://github.com/MF-DEEV/Openbar


r/LLMDevs • • 1d ago

Discussion How are you tracking the cost of failed LLM runs?

5 Upvotes

Got a little carried away rerunning an extraction test and now I’m trying to figure out what each usable result costs once I include retries and the outputs I tossed. Yesterday I came across a twitter post about Ramp’s token spend and wanted to hear from anyone using it alongside their application logs because the part I’m stuck on is tying the bill back to individual runs. Are you attaching a run ID to every attempt and adding up the cost afterward or has that turned into more upkeep than it’s worth?


r/LLMDevs • • 1d ago

Help Wanted A single Reddit post flipped my AI court's verdict. Here is what I changed.

1 Upvotes

I'm building a tool that stress-tests institutional decisions with LLMs. AI advocates argue a question for and against across three angles ("lenses"), then a court of 18 LLM judges votes, each with a written reason. Solo project, built from Gaza.

Two things broke in ways I didn't expect.

1. All 18 judges agreed. That was the bug.

In an early session, the verdict was 18 out of 18. The judges were too similar: same instructions, same model. So I changed four things:

  • Each judge on a panel got a different mandate: evidence, risk, implementation, long term, stakeholders, devil's advocate.

  • The judges were spread over three Claude models.

  • A third of them read the debate in reversed order.

  • One rule for everyone: a claim with no evidence earns no credit for either side.

Verdicts started to split: 5-1, 2-4, 0-6. In a later session, average vote agreement was 0.82, but the similarity of the judges' written reasons was only 0.29. Similar conclusions, reached by different routes.

2. A Reddit post flipped a verdict.

Test question: should a university endowment allocate 10% to private credit?

  • In English, the first lens voted 5-1 for.

  • I ran the same question in Arabic, and that lens flipped to 0-6 against.

When I traced it, a Facebook post and a Reddit post had made it into the evidence, and the judges weighed them like sources.

The fix was boring:

  • No social media or forums as evidence.

  • Every source labelled: official, research, press, web, or the institution's own.

  • A fact only counts if it comes with a verbatim quote from its source.

I re-ran it in Arabic: back to 5-1 for, and the decisive lenses matched the English run (overall 7 for and 11 against in English, 5 and 13 in Arabic).

Caveat: this is a handful of sessions, not a benchmark. Both final decision records are public if you want to check the reasoning:

Two questions might help alot :

  1. How do you keep low-quality sources out of LLM judgment pipelines without throwing away useful informal knowledge?

  2. Is there a better way to measure judge independence than the similarity of their reasons?