r/PiCodingAgent • • 3d ago

Question Agent drift

Is the agent drifting from original plan a result of the model or harness? Im using QFN gyro-M and pi-blackhole but with the observational memory component is turned off due to the workers going every turn bogging the process down because im on a potato. I Told the agent to start step 5, walked away and at some point it had drifted from the orginal plan.md... it was a mess. Ever since then after every compaction im afraid it will drift. So, now every so often I'll have it do a "drift check". Is there an extension or skill I can add to the harness to help prevent this?

0 Upvotes

10 comments sorted by

6

u/makingthematrix 3d ago

Drift is inevitable. You are there to control it. Write specifications and unit tests, make a TODO list with well-defined steps divided into sub-steps, review the changes after each small, and refactor the codebase after each big step.

4

u/Aubrey_D_Graham 3d ago

Drift is a model thing, but the harness enables it. Harnesses give the agent tools, but the LLM is still responsible for the decisions that lead to drift.

A tool that prevents drift would be a million dollar idea, but the best safeguard will always be a human in the loop.

1

u/gandazgul 3d ago

It's partly both. LLMs want to get to the objective as quick as possible so if skipping parts of the plan still achieves this it will do it. In my experience you need plans written with goals in mind rather than imperative language (instead of: do this, use: at the end X should behave this way) but also you need strong system prompts not just regular prompts or skills to keep the LLM aligned to the plan.

You have both here plus adversarial reviews of the plan and the code. https://runwield.dev/

Built in Pi all local works with any sub or API key Pi supports as well as using Antigravity via custom agents and MCP.

1

u/electric-dragon-80kv 3d ago

This is not 100% solvable but you can approximate it by having adversarial steps with subagents. Split your task into deliverable and verifiable steps, and after each step apply a set of adversarial steps focusing on different aspects. Code review, spec loyalty, proof of work could be separate steps. You can go even further and define adversarial steps for the adversarial steps themselves.

1

u/Hubba_Bubba_Lova 11h ago

This is a good idea. Do you automate the adversarial steps? Do you grill-me type of defining them? Or do you have a review plugin that automatically does this? Do you use separate clean sessions for the adversarial work?

1

u/electric-dragon-80kv 9h ago

There is an orchestrator which is in charge of executing these loops and coordinating. Each task requires a different approach. Some tasks will require a more rigorous, hands on approach while others can be mostly left to the agents to figure out. I wrote my own extension for this: https://github.com/electric-dragon-80000v/pi-cowboy

1

u/ResearcherFantastic7 3d ago edited 3d ago

Create yourself hooks that do a count every X turn and prompt a cheaper model to evaluate,you can build a score system if it lower come back with lower than it should ask your better model to fix the drift or what ever criteria you set

Things that I score

  • goal intention drift

  • files to be updated as sideefect

  • design doc alignment, gap, ambiguity

  • taking a step back and evualte holistically on your design principles

These are different hooks, also I use bespoke nimble to make the decision if it needs to run through the eval or not

1

u/RelationshipLong9092 2d ago

well, you're running a relatively tiny model compared to the frontier models. I'm not surprised that its being a bit dumb. Especially if you let its context window fill over 70% or so.

1

u/LORD_CMDR_INTERNET 1d ago edited 1d ago

QFN drifts like a motherfucker, all models do to an extent. The best way I've found to manage is subagents - one agent does the heavy lifting, the other is the "manager" that pokes it to keep going and checks that what is being done is according to the plan. The manager will eventually drift too, but at an exponentially slower rate. The other way I've managed drift is to have it work only to X, then prepare handoff documentation for Y, clear context completely, have it pick up and do Y. Repeat. Basically a manual "compaction" process.

Edit: also keep it's context window small, that will help enormously. 100k max.

0

u/randomgenericbot 3d ago

Since you're on a "potato", you might give my memory extension a try. I can exactly run one model with one slot, and fresh context means 3+ minutes wait (this is what happens with pi-observational-memory: it kills your kv-cache if you do not have a second model running)

It does not store permanent memory, but keeps the session coherent while not hogging down too much on summaries. I am using it for months now, and still tweaking it.
My sessions last far over 20 compactions no problem, without the model losing track.

My reddit post from the past: https://www.reddit.com/r/PiCodingAgent/comments/1xb7k4n/i_solved_compaction_for_singlesession_on/