r/PiCodingAgent • u/startassets • 19d ago
Plugin pi-agent-computer-use: give your access to computer (browser+desktop)
Hi! Currently computer-use is a must for my agents, so I decided to build an extension with a custom CUA Driver fork and Playwright. It’s purposefully designed to be run in YOLO mode (although Safe is included) within the environment where you trust your agents. It works in hybrid mode: supports both raw image followed by input (agent sees a screenshot and then clicks on something) and structural way (playwright for browser and UIA for normal desktop applications). Agents can use their own browser profiles, or your own, work in sandboxes and remote desktops, work together (to some extent, there are limitations on the OS side).
I would say it was definitely worth working on that, my agents can now work in 3D, browse some stuff I need and can test my applications, acting like real users.
I’ve mostly benched it via GIMP drawing, which doesn’t support UIA so all the interactions are purely from picture based mode. But also Terra 6.1 High has gone to 42th level of https://neal.fun/not-a-robot/ until it got banned by Cloudflare, and we couldn’t pass further. It’s windows only (windows from WSL as well), but if anybody can contribute into linux/macos branch of it, it would be great, I’m just lacking such devices and can’t really test them to the full extent as I can with Windows.
https://github.com/alexshpunt/pi-agent-computer-use
https://pi.dev/packages/pi-agent-computer-use
To install: `pi install npm:pi-agent-computer-use`
Demo of deepseek-flash, haiku, opus, terra and sol drawing the same image with that: https://youtu.be/69doTus760U
7
u/Megatron_McLargeHuge 19d ago
It looks like you gave it a skill for using the GIMP tools. Did you try letting it figure out the interface itself? I tried using GIMP recently for the first time in a decade and the interface was as obtuse as I remembered (at least the layers part). I'd be really impressed if an llm got it working from the docs alone.
6
u/startassets 19d ago
The skill was specifically created so we have parity between the models and we avoid spending time on "learning". But otherwise they always figure out how to use any UI from my experience, tested with Unity, software I develop at work, GIMP, Blender, nothing really holds them far back. Yes there are always trials and errors, but that's why you create skills, right? So they don't repeat the same stupid mistakes all the time. Otherwise I'm pretty much with you regarding the UX of GIMP and Blender, I've learned 3ds max and Photoshop a long time ago and it seems like they actually nailed the UX very well, so it's intuitive.
1
u/berot3 16d ago
What about using MCP to control gimp?
1
u/startassets 15d ago
Which mcp? For gimp? The idea is that with the extension it doesn't matter what to control and how, any application.
4
u/yawk-ra 19d ago
Hi,
I built an agentic Painting Tool, Toska, that's a fork of Krita if you are interested. This tool just uses Krita's Python SDK to do paintstrokes . It's currently Mac only for now. Toska is on Github and I will be shortly releasing the Windows and Linux version as well.
Feel free to check out a Video of Toska painting here.
4
u/startassets 19d ago
Hi! Looks very interesting, I'll keep an eye on that one, how efficient is that in real life usage? How long does it take to paint something complex? Can you use it do do AI, concept-arting etc.? Comparing to gen-art, how different is that? What are the benefits?
2
u/yawk-ra 18d ago
I like it because Toska basically controls a paint editor. It can do underpainting, layers, and effects, so you can fully act as an artist versus just one-shotting and hoping the model gets it right. Toska also has all of Krita's regular Toolbox of Editing Tools like Masks, Selection Tools, Layer Management.
For very complex paintings with heavy realist elements it can take a 2-3 hours still. New Moscow is the most complex painting that I've made with Krita. New Moscow took me about 5-7 hours just because while I referenced AI images, I still refined and made the agent do several edits to get according to my artist vision.
I like to think of it similar to Agentic Coding. It can save hours of time but it still requires tinkering and speaking to the Agent about your design choices.
3
2
u/chelvis2 19d ago
Impressive! Do you have any test with a Qwen model?
3
u/startassets 19d ago
Tbh no, I think I have one with Go, but I don't have enough quota for that atm. Maybe I should try some plan from Alibaba to test this.
2
u/Extra_Loquat_7667 12d ago
Hello, I was quite interested by your mention in the article about beating level 42 of not-a-robot.
Since I've recently been researching agents specialized in browser operation, I've also been testing this website to observe how agents interact with the interface.
I'm using the `terra high` model as well, but I've been stuck at Level 10.
The reason is that this stage requires the user to have high-speed reactions to track an object moving across the screen.
In my tests, the `terra high` model simply doesn't have a high enough reaction speed; by the time the agent receives the response, the timing has already passed.
It feels like unless you're using some kind of ultra-high-speed model (or relying on extreme luck...), passing this level is nearly impossible.
I was wondering if you could share how your framework managed to clear it?
2
u/startassets 11d ago
Hi! Yes that's one is quite tricky, but the solution was relative simple. There is no llm right now with such reaction speed, so that's why you have to have something that can react instant. So your computer use tool should support wait and react conditional blocks and ideally a codemode to connect everything together. The most naive and a bit cheaty way is to simply for the model to figure out that there are two different panels in html appearing on the page: mole and grass. Once the waiter detects the mole it should instantly emit click even over this block. Another a bit more fair way is to use classifiers, either a local or cloud ones (jev, decision api, clef). Then the algorithm is the same, cut the region into grid and for each grid pass the check until it detects the mole.
So it's more like think long during the observation (the agent should be able to capture the page for a while to see the dynamics of the scene) and then react super fast (with classifiers or html checking).
Although I've passed the test with classifiers as well, the actual run when Terra passed to 42th was done without them simply by observing the state of the page, so it was slightly cheating.
I hope that it answers your question, but feel free to ask more, it's an exciting topic to me and I've spent quite a time trying to figure out how to make it efficient and fast.
2
u/Extra_Loquat_7667 11d ago
Got it, thanks for the response
So during testing, do you give different instructions for different test cases?My approach is actually a bit more brute-force: I simply feed the Agent the URL and ask it to do its best to clear the challenge, without giving any hints along the way. Then, I observe how far it can go autonomously. (In quite a few levels, the methods the Agent used to pass actually felt like cheating...)
I’m using this approach to study whether the tools and context provided to the Agent are sufficient.
2
u/startassets 11d ago
No, during testing I tried to allow to figure out how to deal with the game on it's own. But after that we do some debrief to figure out what was difficult to do, was that the task itself or the tooling ergonomics, bugs or maybe some errors etc. And then brainstorm a bit how that be improved. With the next iteration I do it from clean context. So the agent doesn't know how to pass each individual test.
2
u/Extra_Loquat_7667 10d ago
Thanks for the reply.
Later on, I found that I only needed to enable the web JS tool for the agent.
Due to security concerns, JS tools were previously blocked by default.
Once enabled, the agent can monitor changes and dispatch events on its own, making it easy to break through levels all the way to Lv42 (even I don't quite understand what Lv42 is testing lol...).
It seems the biggest challenge in automation is actually figuring out how to make everything programmable.2
u/startassets 10d ago
What happens after 42? Because for me my agent just got blocked by cloudflare and we couldn't pass further.
1
u/Extra_Loquat_7667 10d ago
LV42 is a chat interface.
You must keep the conversation going to prove that you are human.
I learned from a YouTube video how this level is supposed to function.
Right now, the chat just keeps outputting error messages.
1
1
u/smugself 15d ago
I'll have to check it out. I've been doing a fair bit of computer use, recipe training for repeatable skills
37
u/Remote_Book 19d ago
cute benchmark