r/reinforcementlearning • • Sep 14 '26 Regret

Google copied our open-source code, removed our engineers’ names, and gave us zero credit one year after we beat them on their own benchmark

Post image
7.7k Upvotes

Google copied our open-source code, removed our engineers’ names, and gave us zero credit.

For context, let me share the story of mobile-use, the origin of my company (I'm a founder).

We wanted to build the best agent in the world to control any mobile interface with natural language.

We built it, became #1 on Google Deepmind's AndroidWorld benchmark and open-sourced it. That was the beginning of my company.

Since then, we got almost 3,000 stars on the repository, became the first to ever achieve 100% on AndroidWorld, grew the team, spent thousands of calls with customers, and now we're building the best regression testing product the world has ever seen.

Last week, I found Artemis.

Artemis is Google's new open-source tool for mobile automation. There are quite a few of these on GitHub now, so I thought: "Oh, a new YAPA (Yet Another Phone Agent)."

Curious, I looked at the repository more in depth.

First, I was disappointed to see that the benchmark chart in the README didn’t include us, topping AndroidWorld in January.

Then I looked at the code, and to my biggest surprise, I recognized the lines we had written ourselves.

One of my engineer named an agent randomly “hopper”, and it was there, with its prompt, word for word. Out of the 229 files, 228 were identical.

Even crazier: I found our engineers' and my co-founder’s names in the “authors” section of an old commit.

Pierre-Louis Favreau, Jean-Pierre Lo, Nicolas Dehandschoewercker.

All three names had been replaced with another author’s name in a commit. The other 228 files from mobile-use were unchanged.

From a personal point of view, I am deeply disappointed.

Google was the company I wanted to join when I started my studies, before changing my mind and deciding to build a unicorn.

I’ve always associated Google with great engineering and contributions to open source. I would have been proud to see them building on our work and acknowledging the team behind it.

Instead, I found our code, then our names, then the commit that deliberately removed them.

What about the other engineers and the community contributors who spent their time improving mobile-use?

There are people behind those files. I’ve seen the work they put in, the problems they worked through, and how much they care about what they build.

They deserve to have their names attached to that work.

We made our code open source because we wanted people to use it and build on it. Of course that includes Google.


r/reinforcementlearning • • Aug 18 '25

We beat Google Deepmind but got killed by a chinese lab

797 Upvotes

Two months ago, some friends from AI research and I asked ourselves: what if an AI could actually use a phone like a human?

So we built an agentic framework that taps, swipes, types… and somehow it’s beating Google DeepMind and Microsoft Research on the AndroidWorld benchmark.

We were super happy about our results until we saw a chinese lab (Zhipu AI) releasing their results this week: they took the number 1 spot.
They’re a bit ahead, but they have an army of 50 phds and I don't see how a team like us can compete with them...

... however, they're closed source.

We decided to open-source it, as that’s the way we can make our work stand out.

Currently, we’re building our own custom mobile RL gyms, training environments made to push this agent further and get closer to 100% on the benchmark. Even as a small team, we want to contribute and make this framework available to anyone who wants to experiment.

Do you have any tips on how we can compete with bigger than us?

Repo’s here if you want to check it out or contribute: github.com/minitap-ai/mobile-use


r/reinforcementlearning • • Oct 03 '26

Live footage of successful paper review rebuttal

604 Upvotes

r/reinforcementlearning • • Sep 17 '26

DL I literally build the jev architecture one year back and made it open-sourced

562 Upvotes

Everyone now talks about the architecture that's not auto regressive and does lightning fast probability prediction with a json schema. I worked on this literally one year back in March 2025, published an arxiv paper, pushed the model to huggingface along with the pypi package and training dataset. And then one year later, a

frontier lab came, proposing the same idea like literal breakthrough without technical papers, open weights and no open dataset.

Paper: https://arxiv.org/abs/2503.23303

Model: https://huggingface.co/DeepMostInnovations/sales-conversion-model-reinf-learning

Dataset: https://huggingface.co/datasets/DeepMostInnovations/saas-sales-conversations

Also the second work published in September 2025 was exactly the same one jev proposed now

Paper: https://arxiv.org/abs/2510.01237

My model uses PPO over sequence embeddings to output turn-by-turn conversion trajectories (probabilities from 0.0 to 1.0).

Jev uses parallel sampling (trained via RLCD) to output confidence distributions and schema choices.

It's incredibly frustrating that the thing that you made with months of hard work, sweat and sleepless night is architecturally similar with the vertical use case and don't get the support you deserve because frontier lab build something horizontal. The open-source story in general 🙂


r/reinforcementlearning • • Dec 09 '25

Trained a PPO agent to beat Lace in Hollow Knight: Silksong

558 Upvotes

This is my first RL project. Trained an agent to defeat Lace in Hollow Knight: Silksong demo.

Setup
- RecurrentPPO (sb3-contrib)
- 109-dim observation (player/boss state + 32-direction raycast)
- Boss patterns extracted from game FSM (24 states)
- Unity modding (BepInEx) + shared memory IPC
- ~8M steps, 4x game speed

I had to disable the clawline skill because my reward is binary (+0.8 per hit).
Clawline deals low damage but hits multiple times, so the agent learned to spam it exclusively. Would switching to damage-proportional rewards fix this?


r/reinforcementlearning • • Apr 07 '25 Helpful

IT'S LEARNING!

Post image
554 Upvotes

Just wanted to share cause I'm happy!

Weeks ago I recreated a variant of Konane as it is found in Mount & Blade II: Bannerlord, in Python. (only a couple different rules like starting player and first turn)

Tried QLearning at first, and self-play, in the end went with PPO with the AI playing as the black pieces VS white pieces doing random moves. Self-play had me worried (I changed the POV by switching white and black pieces on every move)

Konane is friendly to both sparse reward (win only) and training against random moves because every move is a capture. On a 6x6 grid this means every game is always between 8 and 18 moves long. A capture shouldn't be given a smaller reward as it would be like rewarding any move in Chess, also a double capture isn't necessarily better than a single capture, as the game's objective is to position the board so that your opponent runs out of moves before you do. I considered a smaller reward for reduction of opponent player's moves, but decided against it and removed it for this one, as I'd prefer it'd learn the long game, and again, end positioning is what matters most for a win, not getting your opponent to 1 or 2 possible moves in the mid-game.

Will probably have it train against a static copy of an older version of itself later, but for now really happy to see all graphs moving in the right way, and wanted to share with y'all!


r/reinforcementlearning • • Oct 10 '26

Reinforcement learning Racing drone learns to dodge obstacles using an 8*8 time-of-flight sensor

395 Upvotes

Greetings. I'm sharing what I'm doing for my doctoral thesis. If you're interested in more information, please contact me. I'm willing to collaborate.

I am using stable baseline 3 and pybullet


r/reinforcementlearning • • Aug 15 '25

Robot PPO Ping Pong

360 Upvotes

One of the easiest environments that I've created. The script is available on GitHub. The agent is rewarded based on the height of the ball from some target height, and penalized based on the distance of the bat from the initial position and the torque of the motors. It works fine with only the ball height reward term, but the two penalty terms make the motion and pose a little more natural. The action space consists of only the target positions for the robot's axes.

It doesn't take very long to train. The trained model bounces the ball for about 38 minutes before failing. You can run the simulation in your browser (Safari not supported). The robot is a ufactory xarm6 and the CAD is available on Onshape.


r/reinforcementlearning • • Jan 29 '25

DL, M, I Why is RL fine-tuning on LLMs so easy and stable, compared to the RL we're all doing?

357 Upvotes

I've been watching various people try to reproduce the Deepseek training recipe, and I've been struck by how stable this seems compared to the RL I'm used to.

They reliably hit 50% accuracy on their math problem after about 50 training steps. They try a few different RL algorithms and report they all work approximately equally well, without any hyperparameter tuning.

I'd consider myself lucky if I could get 50% success at balancing a cartpole in only 50 training steps. And I'd probably have to tune hyperparameters for each task.

(My theory: It's easy because of the unsupervised pretraining. The model has already learned good representations and background knowledge - even though it cannot complete the task prior to RL - that makes the problem much easier. Maybe we should be doing more of this in RL.)


r/reinforcementlearning • • Mar 05 '25

N, MF Andrew G. Barto and Richard S. Sutton named as recipients of the 2024 ACM A.M. Turing Award

Thumbnail
acm.org
343 Upvotes

r/reinforcementlearning • • Apr 17 '25

Can RL redefine AI vision? My experiments with partial observation & Loss as a Reward

324 Upvotes

A few days ago, someone asked if reinforcement learning (RL) has a future. As someone obsessed with RL’s potential to mimic how humans actually learn, I shared a comment about an experiment called Loss as a Reward. The discussion resonated, so I wanted to share two projects that challenge how we approach AI vision: Eyes RL and Loss as a Reward.

The core idea

Modern AI vision systems process entire images at once. But humans don’t do this, we glance around, focus on fragments, and piece things together over time. Our brains aren’t fed full images; they actively reduce uncertainty by deciding where to look next.

My projects explore RL agents that learn similarly:

  • Partial observation: The agent uses a tiny "window" (like a 4x4 patch) to navigate and reconstruct understanding.

  • Learning by reducing loss: Instead of hand-crafted rewards, the agent’s reward is the inverse of its prediction error. Less uncertainty = more reward.

Eyes RL: Learning to "see" like humans

My first project, Eyes RL, trained an agent to classify MNIST digits using only a 4x4 window. Think of it like teaching a robot to squint at a number and shuffle its gaze until it figures out what’s there.

It used an LSTM to track where the agent had looked, with one output head predicting the digit and the other deciding where to move next. No CNNs, instead of sweeping filters across the whole image, the agent learned to strategically zoom and pan.

The result? 69% accuracy on MNIST with just a 4x4 window. Not groundbreaking, but it proved agents can learn where to look without brute-force pixel processing. The catch? I had to hard-code rewards (e.g., reward correct guesses, penalize touching the border). It felt clunky, like micromanaging curiosity.

Loss as a Reward: Letting the agent drive

This led me to ask: What if the agent’s reward was tied directly to how well it understands the image? Enter Loss as a Reward.

The agent starts with a blurry, zoomed-out view of an MNIST digit. Each "glimpse" lets it pan or zoom, refining its prediction. The reward? Just 1: classification loss. No more reward engineering, just curiosity driven by reducing uncertainty.

By the 3rd glimpse, it often guessed correctly. With 10 glimpses, it hit 86.6% accuracy, rivaling full-image CNNs. The agent learned to "focus" on critical regions autonomously, like a human narrowing their gaze. You can see the attention window moving in the video.

Why this matters

Current RL struggles with reward design and scalability. But these experiments hint at a path forward: letting agents derive rewards from their own learning progress (e.g., loss reduction). Humans don’t process all data at once, why should AI? Partial observation + strategic attention could make RL viable for real-world tasks like robotics, medical imaging or even video recognition.

Collaboration & code

If you’re interested in trying the code, tell me in the comments. I’d also love to collaborate with researchers to formalize these ideas into a paper, especially if you work on RL, intrinsic motivation, or neuroscience-inspired AI.


r/reinforcementlearning • • Jan 26 '25

Built a custom robotic arm environment and trained an AI agent to control it

319 Upvotes

r/reinforcementlearning • • Sep 06 '25

Robot Looking to improve Sim2Real

304 Upvotes

Hey all! I am building this rotary inverted pendulum (from scratch) for myself to learn reinforcement learning applies to physical hardware.

First I deployed a PID controller to verify it could balance and that worked perfectly fine pretty much right away.

Then I went on to modelling the URDF and defining the simulation environment in Isaaclab, measured physical Hz (250) to match sim etc.

However, the issue now is that I’m not sure how to accurately model my motor in the sim so the real world will match my sim. The motor I’m using is a GBM 2804 100T bldc with voltage based torque control through simplefoc.

Any help for improvement (specifically how to set the variables of DCMotorCfg) would be greatly appreciated! It’s already looking promising but I’m stuck to now have confidence the real world will match sim.


r/reinforcementlearning • • Jul 27 '21

N, P OpenAI Gym is now actively maintained again (by me)! Here's my plan

292 Upvotes

So OpenAI made me a maintainer of Gym. This means that all the installation issues will be fixed, the now 5 year backlog of PRs will be resolved, and in general Gym will now be reasonably maintained. I posted my manifesto for future maintenance here: https://github.com/openai/gym/issues/2259

Edit: People in the comments and elsewhere repeatedly asked about open source donations, so I created links:

https://liberapay.com/jkterry

https://www.buymeacoffee.com/jkterry


r/reinforcementlearning • • Oct 18 '21

N, P DeepMind buys & open-sources MuJoCo

Thumbnail
deepmind.com
278 Upvotes

r/reinforcementlearning • • Mar 30 '25

Showcase Implemented 18 RL Algorithms in a Simpler Way

264 Upvotes

What My Project Does

I was learning RL from a long time so I decided to create a comprehensive learning project in a Jupyter Notebook to implement RL Algorithms such as PPO, SAC, A3C and more.

Target audience

This project is designed for students and researchers who want to gain a clear understanding of RL algorithms in a simplified manner.

Comparison

My repo has (Theory + Code). When I started learning RL, I found it very difficult to understand what was happening backstage. So this repo does exactly that showing how each algorithm works behind the scenes. This way, we can actually see what is happening. In some repos, I did use the OpenAI Gym library, but most of them have a custom-created grid environment.

GitHub

Code, documentation, and example can all be found on GitHub:

https://github.com/FareedKhan-dev/all-rl-algorithms


r/reinforcementlearning • • 28d ago

Robot I’m building a browser game where players design robots that learn to walk through reinforcement learning

249 Upvotes

I'm building Bitbots, a browser game where players assemble robots from parts and train them to move using reinforcement learning.

This video shows the game's “brain view”: the policy's inputs, selected hidden-layer activations, and motor outputs, synchronized with the robot's movement. These are recorded values from the actual policy, not a decorative animation.

The idea is to let players see more than just whether their robot walks or falls over. They can watch how the network's activity changes alongside its movement. It isn't an explanation of what individual neurons “mean,” but I wanted to give players a way to look inside the controller.

The current simulation uses MuJoCo, with GPU training through MJX's Warp backend. Players choose the robot's body and joint configuration; the policy learns how to control it. During races, the robot acts autonomously.

For people working on locomotion across different morphologies: how would you support creative, unconventional robot designs under a limited training budget? I’d like players to experiment with weird bodies and joint configurations, rather than feel pushed toward a few designs known to work. How would you approach this without making the reward so prescriptive that every robot ends up moving the same way?

I'm the developer. Bitbots is currently in closed alpha, and free invites are available by comment or DM if you'd like to experiment with your own robot designs.

Project page: https://bitbots.endresgames.com

EDIT: EDIT: Wow, thank you all for the interest, it means a lot! Bitbots is a passion project, and I'd love your ideas and feedback. Join the Discord for updates and alpha invites: https://discord.gg/zgxpuhgB8U 💡


r/reinforcementlearning • • Feb 01 '26 Hooray

Diablo 1 Agent Trained to Kill The Butcher Using Maskable PPO

Post image
244 Upvotes

TL;DR

I trained a Maskable PPO agent to navigate Tristram and the first two levels of the cathedral and kill The Butcher in Diablo 1. You can grab the repo with a dedicated DevilutionX fork to train or evaluate the agent yourself (given you have an original valid copy of Diablo)!

Long(er) Version

So I've been working on this project on and off for the past several months and decided that while it's still messy, it's ready to be shared publicly.

The goal was basically to learn. Since AI got very popular, as a day-to-day developer I didn't want to fall behind and wanted to learn the very basics of RL.

A very big inspiration and sort of a "push" was Peter Whidden's video about his Pokemon Red experiments.

Given the inspiration, I needed a game and a goal. I have chosen Diablo since it is my favourite game franchise and more importantly because of the fantastic DevilutionX project basically making Diablo 1 open source.

The goal was set to be something fairly easy to keep the learning process small. I decided that the goal of killing The Butcher should suffice.

And so, over the course of several adjustments separated by training processes and evaluation, I was able to produce acceptable results.

From last training after ~~14 days 14 clients have killed butcher ~~13.5k times

Last Training Results

As mentioned the code is definetly rough around the edges but for RL approach I hope it's good enough!


r/reinforcementlearning • • Jan 31 '20

Multi mods don’t be mad

Post image
234 Upvotes

r/reinforcementlearning • • Jun 28 '26

Spot walking procedural terrain: Isaac base policy + transfer learning, all driven from Python on my own Vulkan renderer

233 Upvotes

Short clip of Spot crossing procedural terrain. The base policy came from Isaac Lab, everything else is mine.

Policy: chain of PPO transfers, not zero-shot. Isaac Lab flat walker → fine-tuned for rough terrain → fine-tuned again for discrete stairs.

Keeping the Isaac gait while learning new terrain: I keep a frozen copy of the original Isaac policy as a teacher and add a scan-gated imitation reward. On flat ground the policy is penalized for drifting from the teacher's actions, so it only deviates where the terrain actually demands it. Plus a per-env adaptive curriculum (1024 parallel envs, each promotes/demotes its own step height).

Obs/action: 94-d obs (48-d Isaac proprio + base height + a 45-cell forward height-scan), 12-d joint-target actions. Training is my own PPO loop over batched PhysX on GPU.

The stack (all mine, driven from Python): C++ engine with Python bindings (threepp). A Vulkan deferred renderer with procedural terrain, PhysX physics, and the live SLAM map. No Isaac Sim / Omniverse at runtime; the base policy is the only imported piece.

The current policy largely ignores the depth sensor data. Working on it, but has very good stability nonetheless.

UPDATE: Due to license issues, I have now generated a new similar gait that is not warm-started from a Isaac policy.


r/reinforcementlearning • • Dec 13 '24

D RL is the third most popular area by number of papers at NeurIPS 2024

Post image
233 Upvotes

r/reinforcementlearning • • Sep 18 '26

DL Made the horizontal open-source model for Jev with RLCD, and it surpasses all the Jev benchmarks. HF space, benchmark, model, repo

Post image
228 Upvotes

Thanks for the exceptional support (https://www.reddit.com/r/reinforcementlearning/comments/1wihdun/i_literally_build_the_jev_architecture_one_year/) and for the dozens of requests to make a generic model, run benchmarks, and create an HF space so anyone can test it. So here you go, guys. I trained an improved model on a large data corpus, its now called Laya. It is trained on a single RTX 6000 Pro (96 GB VRAM); the model architecture is a 421M-parameter non-autoregressive decision model pairing a bidirectional ModernBERT-large encoder with a scratch Transformer head that scores [MASK] option markers to resolve typed schemas in a single ~35 ms forward pass. The dataset is a 100% human-annotated corpus of over 25,000 real-world examples across intent routing, fact-checking, moderation consensus, prompt guardrails, rubric scoring, and multi-turn conversation trajectories, without synthetic data shortcuts. The RLCD(unofficial, btw) I did is a policy-gradient reinforcement learning approach that kinda optimizes decision models against strictly proper scoring rules, ensuring maximum reward is achieved only when outputting true, mathematically calibrated probabilities.

HF space to try: https://huggingface.co/spaces/convaiinnovations/laya-demo

GitHub Repo: https://github.com/NandhaKishorM/laya

HF Repo: https://huggingface.co/convaiinnovations/laya

Thank you to everyone who supported me, shared the story, gave personal DM. It will need more refinement, of course.

If anyone wishes to buy me a coffee, here is the link: https://github.com/NandhaKishorM


r/reinforcementlearning • • Jan 13 '20

I feel for them....

Post image
221 Upvotes

r/reinforcementlearning • • Aug 18 '21

DL, MF, Multi, D MARL top conference papers are ridiculous

218 Upvotes

In recent years, 80%+ of MARL top conference papers have been suspected of academic dishonesty. A lot of papers are published through unfair experiments tricks or experimental cheating. Here are some of the papers,

update 2021.11,

University of Oxford: FACMAC: Factored Multi-Agent Centralised Policy Gradients, cheating by TD lambda on SMAC.

==================

Tsinghua University: ROMA (compare with qmix_beta.yaml), DOP (cheating by td_lambda, env numbers), NDQ (cheating, reported by GitHub and a people), QPLEX (tricks, cheating)

University of Sydney: LICA (tricks, large network, td lambda, adam, unfair experiments)

University of Virginia: VMIX (tricks, td_lambda, compare with qmix_beta.yaml)

University of Oxford: WQMIX(No cheating, but very poor performance in SMAC, far below QMIX),

Tesseract (add a lot of tricks, n-step , value clip ..., compare QMIX without tricks).

Monash University: UPDeT (reported by a netizen, I didn't confirm it.)

and there are many more papers that cannot be reproduced...

2023 Update:

The QMIX-related MARL experimental analysis has been accepted by ICLR BLOGPOST 2023

https://iclr-blogposts.github.io/2023/blog/2023/riit/

full version

https://arxiv.org/abs/2102.03479