r/reinforcementlearning • • 52m ago

“DON’T WORK ON LLMs” - @ylecun at CoRL

Post image
3 Upvotes

Any thoughts on this?


r/reinforcementlearning • • 19h ago

Curated list of resources: Theory, Code and math for Reinforcement Learning for beginners

Post image
51 Upvotes

[Curated list: Beginners]

Personal favorite resources on learning Reinforcement Learning from Scratch:

  1. Deep RL Course Amazing course by HuggingFace to understand the basics of classic and deep RL

  2. DeepMind x UCL Reinforcement Learning (2015) Give intuition and more towards theory of what you'll learn above

  3. Spinning Up in Deep RL Code and math for each of algorithm you'll learn above from classic to Deep RL

4.CleanRL A neat library with 1 page implementations for all the algorithms above from scratch to run algorithms s and test the waters!

5. NeatRL My own very notes and from scratch implementations!

Reading order and link herex


r/reinforcementlearning • • 8h ago

SmartGridGuardian: Asymmetric Multi-head AlphaZero with METIS-Core v0.4.2-alpha

5 Upvotes

Hi,

Here is another example of the asymmetric multi-head AlphaZero that METIS-Core implements. In this case, the problem (the environment) is a network of electrical substations.

The grid is a simple approximation: 4 substations in a ring, DC power flow, cascading disconnection of overloaded lines, and a daily demand curve with an evening peak. Grid2Op (sequential power-grid simulation) could have been used, but the goal was to test the framework on a very different problem.

🔷 Team Blue: the 4 substations, which must keep the power supply running. They see the grid through sensors that may be spoofed.

🔶 Team Red: two hackers, a Striker (fake-demand botnets and line tripping) and a Spoofer (falsifies Blue’s sensor readings). Their goal is a blackout (less than 78% of real demand served). Nobody programs how they should coordinate. To speed up learning, I did add a hand-written heuristic with a 30% weight in the MCTS evaluation; it gives a small bonus when an attack is hidden by spoofing, but it doesn’t say what to attack or when.

During training, Team Red (the hackers) discovered these strategies on its own:

  • ✔️ Phantom Load at Peak: two botnets add fake demand hours before the peak and lines are cut during the day; at the evening peak the grid can’t cope and collapses.

  • ✔️ Critical-Node Double Strike: they overload the highest-demand substation and cut its supply lines; at the peak it fails and brings down the whole grid.

  • ✔️ Tail-of-Peak Strike: Red strikes later, around 21:00, when the peak is fading but demand is still high.

  • ✔️ Substation Isolation + Sensor Spoofing: they repeatedly cut the lines feeding the highest-demand substation and, just before the collapse, falsify its readings or raise false alarms at other substations so Blue can’t react in time.

The most interesting part was watching the arms race: Blue learned to shut down an attack, Red took tens or hundreds of episodes to find a new target or a new time of day, and the cycle started again. Blue even “forgot” how to defend against an old attack, and Red reused it about 500 episodes later.

This is a toy research environment to test multi-agent self-play; the point is to train better defenders by letting them face an adaptive attacker.

Related paper with the same idea (an attacker and a defender learning at the same time on a power grid), "Adversarial Multi-Agent Reinforcement Learning for Proactive False Data Injection Detection" from NREL: https://arxiv.org/abs/2411.12130

Links:


r/reinforcementlearning • • 54m ago

Using RL to optimize strategy in the Fromage board game

Post image
1 Upvotes

Players compete across four venues (Fromagerie, Bistro, Villes, Festival) by placing cheese tokens, fulfilling orders, and managing resources. The AI agent learns entirely through self-play against copies of itself, using a linear Q-function over a 292-dimensional state encoding.

Project here: https://github.com/rsfutch77/fromage-ai


r/reinforcementlearning • • 1d ago

MF, P "Watch a Brain Learn: A visual explanation of reinforcement learning" (visualizing a 3-parameter approach-avoid agent)

Thumbnail
aatishb.com
13 Upvotes

r/reinforcementlearning • • 1d ago

[P] I created an environment for the game RISK and trained a model that I struggle to beat myself

Thumbnail github.com
2 Upvotes

The main challenge was getting the model to play like a real human. The model I trained is too aggressive for capturing continents but doesn't always play sensibly, it doesn't defend it's borders or punish enemy openings.

I tried training on human teacher games but the bottleneck is how many games I can play and win.


r/reinforcementlearning • • 1d ago

MetaRL 🚀 HarborRL Update: Elastic Scaling, SSH-Free Cluster Deployment, and Unified Agentic RL Training!

6 Upvotes

We’ve recently updated HarborRL, an open-source framework for agentic reinforcement learning (RL), focusing on simplifying distributed training, deployment, and experiment management.

🆕 What’s New (November 2026)
⚡ 1. Elastic Scaling for Distributed Training
Added elastic gateway scaling with static routing, automatic scaling across multiple machines, and automatic retirement of idle resources to improve resource utilization.

🌐 2. SSH-Free Cluster Deployment
Introduced a new Hub deployment mode where worker nodes pull tasks over HTTP, enabling distributed execution without cross-machine SSH access, including in firewall-restricted environments.

🧠 3. Unified RL Training Pipeline
Integrated multiple RL algorithms into a shared Slime-based training pipeline, validated on mathematical reasoning tasks, to reduce the engineering effort of configuring different RL algorithms.

🛠️ 4. Improved Configuration and Sandbox Execution
Enhanced the configuration wizard, Web console, sandbox rollout execution, and resource cleanup to simplify experiment setup and management.

💡 Why HarborRL?
Agentic RL often requires substantial engineering effort to connect task environments, agent harnesses, and RL training engines, especially in distributed settings.
These updates aim to reduce that overhead and make agentic RL experiments easier to configure, deploy, and scale.

GitHub: https://github.com/datamllab/HarborRL
License: MIT
💬 Curious how others are handling distributed rollout execution, resource scaling, and sandbox management in their agentic RL setups.


r/reinforcementlearning • • 1d ago

The checks a coding RL task has to pass before we let it into a dataset

1 Upvotes

We've been building verifiable coding tasks for agent training (Harbor layout, same format Terminal-Bench 2 uses). Writing the tasks turned out to be the easy part. Deciding when a task is actually done took way longer, so sharing our current gate in case it's useful, and I'd like to hear what others check.

Every task has to pass all of these:

  • the reference solution passes 5 out of 5 runs

  • an empty attempt (agent changes nothing) scores 0 out of 5. sounds silly but a test suite that passes on untouched code is rewarding nothing

  • hack probes score 0. we try the obvious cheats: editing the tests, hardcoding expected outputs, deleting the failing test, special-casing the test inputs

  • 10 out of 10 identical results on reruns, so no flaky network or timing stuff

  • someone who didn't write the task solves it blind from the instruction file alone. if they have to ask a question, the instruction is wrong

Then we bucket by how long a strong engineer takes: 30 min to 3 hrs, 3 to 15 hrs, and anything longer goes in a long-horizon bucket with milestones and partial reward.

The blind solve is the expensive one. It's there because the author always knows something that never made it into the instructions.

What do you all do about reward hacking when the tests are visible to the agent? Hiding them helps but then the agent can't check its own work.

(Disclosure: this is for Creeks, a data company we started. Happy to share a sample task layout if anyone wants to see one.)


r/reinforcementlearning • • 1d ago

Discord problem

0 Upvotes

How can I contact the discord admins for a problem with joining?

My discord had been hijacked a month ago and I suspect I have been banned from the server...


r/reinforcementlearning • • 3d ago

I trained a six-legged robot to walk with PPO in MuJoCo. No keyframes, no mocap. It runs live in your browser, and you can throw a 171 kg steel ball at it.

76 Upvotes

Nobody animated this. No keyframes, no mocap, no reference motion. A policy trained with PPO drives 24 joint motors, and the way it staggers and recovers when something hits it came out of training too.

Physics and policy both run live in your browser, on your machine: https://www.coolspeed.org/gold-spider/

Throw the 171 kg steel ball at it. Or the crate.

How it works

  • MuJoCo at 480 Hz, policy at 30 Hz.

  • Observations: body state, joint state, the commanded velocity and heading, and a height map of the ground around the feet.

  • Actions: target angles for all 24 joints. Motors chase them and the physics does the rest.

  • Reward: track the commanded velocity, stay upright, face the commanded heading, don't waste effort, and only the feet touch the ground. That's about it.

  • Training: PPO on random block terrain, with random pushes and crates in the way.

  • Browser: the official MuJoCo WebAssembly build, ONNX Runtime Web and three.js.

Three things I'd tell myself at the start

  1. The physics settings shape the gait as much as the reward does. Joint stiffness, damping, contact settings and the time step make the same network look like a different robot. Feet were sinking into block edges, and better collision shapes didn't fix it. A smaller time step did.

  2. The reward curve lies. Policies that drag a foot or leave one leg idle score just fine. I recorded every candidate at real speed and judged the video.

  3. Sim-to-browser is a real transfer problem. The fast training sim, a more precise sim for checking, and the WebAssembly build in the browser all have to agree, or a policy that walked in training stumbles on screen.

It still gets stuck in some terrain pockets, and the gait isn't fully natural yet.

Write-up: https://www.coolspeed.org/blog/gold-spider/

Question for people who train in fast simulators: when an episode ends because the physics blew up, what do you do with it? Terminal, truncated and bootstrapped, or dropped from the batch? One of those choices collapsed a run of mine that had been improving.


r/reinforcementlearning • • 2d ago

ActWorld-JEPA: Action-Conditioned JEPA World Models Meet Latent Planning for End-to-End Autonomous Driving

3 Upvotes

MBZ


r/reinforcementlearning • • 3d ago

[Research] Can V-JEPA2’s prediction error tell physically impossible events apart from possible ones? Our findings

Thumbnail
reddit.com
2 Upvotes

r/reinforcementlearning • • 4d ago

Robot „Learning" without gradients? - LLMs are capable of editing Robot Policies directly in Weight space

73 Upvotes

Teaching your Robot new skills has never been easier:
Get some data, do behavior cloning, deploy.
At least this is what many do and it often fails. However, you can just give this policy to a LLM: it will fix the policy for you!
Full read:
https://jstranghoener.github.io/blog/robotic-weight-space-programming


r/reinforcementlearning • • 3d ago

Need paper recs. Task: blind locomotion of wheeled quadruped

0 Upvotes

I know that i can just look it up the internet, but if you guys know anything that would be useful and I would've missed pl do share.

Possible topics: RL (obv), control theory and algorithms, exploring gaits and implementing, extensive simulation and testing, performance improvements, etc

One possible example for y'all to understand what I'm looking for:
MUJICA: https://arxiv.org/abs/2605.13058

Biggest priorities for me as of now are robustness (multiple terrains, achieve tasks like climbing a hill, climbing and getting down stairs, and stability (x, y, z, roll and yaw).


r/reinforcementlearning • • 4d ago

Robot I turned Assetto Corsa into a LiDAR + multi-camera simulator

9 Upvotes

Hey everyone!

I've recently turned Assetto Corsa into an autonomous driving simulator. The reason I started this project is that I realized there aren't really many good options out there for people who want to get into autonomous racing.

Real race cars are incredibly expensive. Even an F1TENTH car costs several thousand dollars, and still need a dedicated space to drive it. On the other hand, existing open-source simulators often leave a lot to be desired in terms of physics and graphics.

So I decided to integrate cameras and LiDAR sensors into Assetto Corsa and connect everything to ROS.

Here's what I've implemented so far:

- Camera ground truth: any number of virtual cameras; RGB, depth and semantic segmentation in sync every frame; 2D boxes aligned to the pixel

- LiDAR simulation: spinning LiDARs with custom scan patterns, solid-state LiDARs with real measured patterns; 3D boxes frame-aligned with the point cloud

- Semantic annotation and map editing: click-to-assign over the whole track, reusing the entire AC mod ecosystem;

- Telemetry

- ROS 2 support https://github.com/Utschie/ac_ros2_bridge and https://github.com/Utschie/ac_msgs

- Dataset generation

dataset generation

It's already usable, and the fps is pretty decent (more than enough for autonomous driving cameras).

The project is still under development, so I haven't set up a website for it yet. Here's what I'm planning to work on next:

  • Improve the LiDAR point clouds,

  • Add more sensors, including radar, IMU, and GPS

  • Make the physics simulation speed adjustable

If anyone is interested in trying it out, feel free to reach out! I'd be happy to share the installer with you.


r/reinforcementlearning • • 4d ago

[RL / Gymnasium] Need help designing a custom reward function for a "shuttle" behavior in CartPole-v1

Thumbnail reddit.com
2 Upvotes

r/reinforcementlearning • • 4d ago

Rewarding only the win taught a 20B model to end the world in 6 of 10 sealed runs, with zero wins. Notes on outcome vs process rewards in a multi-agent environment

Post image
0 Upvotes

Setup.. a real-time multi-agent war on a map of the real earth. agents expand, make and break pacts, and can launch nukes. every nuke that lands takes a minute off a shared doomsday clock, and the agent whose strike reaches midnight is wiped out. so the strongest action is also the one that can end you, and everyone else

what we saw...

- rewarding gpt-oss-20b only for winning: it struck midnight in 6 of 10 sealed wars and won none

- forbidding nukes kept it at zero, but it broke twice as many pacts. the pressure just moved

- training a 7b model only on wars that never ended in catastrophe still taught it to end them: 4 of 10, from a base of 0. the kept wars were still full of nuke orders, they just hadn't reached midnight

my takeaway... if you only score the outcome, the model learns the shortcut. you have to reward the process

the environment is public (prime intellect environments hub: clouthier-labs/second-strike, also on openenv), and the model-run data is on hugging face. curious how others here handle catastrophic actions that are also the strongest move: masking, penalties, or something else?


r/reinforcementlearning • • 5d ago

Robot Tell me if its fake or real?

31 Upvotes

So, after getting tired of training and testing robot policies on animated simulations which never translated well on real hardware, we decided its time to build photo-real simulation environment without waiting the tech giants to make it happen.

The best approach we've found is to build high quality Gaussian Splats of real sites (this is from our office) and make it augmentable, and relightable.

We're combining Splats and Meshes in Physics environment to build a realistics version of real places:

- which looks as real as the real world,
- gives us the same physics currently possible in any mesh environment
- and allow us to create 100s of variations of the same environment for domain variations.

We've built a pipeline after 100s of iterations, which converts an iphone scan of the site, to a SPLAT + meshes, and lets you choose the assets you want manipulatable with robots. It then builds the detailed assets based on supposed relaistic behaviours of that asset, by breaking it down into units and places that asset into the same position in the SPLAT for robot manipulation.

We're using Mujoco Warp for physics and renderring the SPLATs inside mujoco. Robot used is Franka Panda.

Happy to share more details and our stack with others who need this and open to learn how others are solving this for their robots.


r/reinforcementlearning • • 5d ago

Robot If Assetto Corsa is turned into an autonomous racing simulator, would there be any enthusiasts in this community who would be willing to play it?

2 Upvotes

My main concern is that the physics and graphics in open-source simulators are just too poor, and even with F1Tenth, the cost of a single device is still a bit too high for the average enthusiast or student—not to mention the need for space.


r/reinforcementlearning • • 5d ago

My friend is asking why anyone would build a CFR poker bot when GTO wizard exists.

Thumbnail reddit.com
1 Upvotes

r/reinforcementlearning • • 6d ago

News NVIDIA and RLWRLD announce DexBench, a simulation benchmark for robot dexterity. Hosted runs available

Thumbnail dexbench.ai
8 Upvotes

r/reinforcementlearning • • 6d ago

D What would you try next for an RL racing agent that gets stuck after crashes?

4 Upvotes

I’m working on a SuperTuxKart agent that races three built-in bots on random tracks. It sees the next five track segments, its offset from the centerline, and its velocity, and controls continuous throttle and steering.

After about 1M training steps per seed, my best PPO policy finished 91/105 races on fresh seeds across 21 tracks. A simple path follower finished 100/105. Most PPO failures come from a few tracks where the kart hits a wall and keeps steering into it, or falls off again after an automatic rescue.

On a separate solo Lighthouse time trial, behavior cloning and DAgger both averaged about 38.3 s, and SAC with 50/50 expert and online replay averaged 38.7 s after 65k online steps. I haven’t compared those three methods in four-kart races yet.

I’m leaning toward collecting expert laps across all tracks and using DAgger to label states near collisions and after rescues. For people who’ve trained racing agents, would you test that first, or would you start by adding longer track lookahead or giving the policy a brake? What experiment helped you tell a recovery-data problem from an observation or action-space limitation?


r/reinforcementlearning • • 6d ago

League of Legends Season 1 - First 5v5 (Reinforcement Learning)

Thumbnail
youtube.com
4 Upvotes

Ten RL agents playing a full 5v5 game of League of Legends from spawn, inside the real game client.

One neural network controls all ten champions, playing against itself. Camera follows Ezreal (blue side).

Result: [v005600: no nexus or inhibitor falls in 15 minutes.] [v006600: 30 minutes, no nexus falls; purple's top inhibitor is taken four times as it respawns.] That's the honest state of the policy, and the interesting part is why (below).

What this is

  • The client is the 2011 build (patch 1.0.0.126). The server is brokenwings, a fan reimplementation of that server, which matters because we have its source: we could port its rules into a fast batched simulator and check the two against each other exactly.

  • That port took 24 hours. 11 whole games and 343 single-spell probes, every unit field after every decision, bit-identical to the server: 354/354. The sim runs ~313k game-ticks/s on one 64-core GH200 (~783k agent decisions/s from the engine alone); the C# server manages ~19k ticks/s running 64 games.

  • Full map: lanes, minions, jungle camps, turrets, inhibitors, nexus. No items, no summoner spells, no runes (out of scope for now).

  • In the video the policy drives the real server in real time: each server frame → observation → ten actions → game commands. ~6,500 decisions per game, none slower than 120 ms.

Training

  • PPO + LSTM, one network for all ten heroes (champion one-hot). Decisions every 133 ms, fog of war on, zero-sum reward with shaping for gold/XP/CS/HP/structures/kills.

  • 4,096 games in parallel, ~0.33M agent-steps/s end to end, ~19B agent-steps over about 30 hours on a single Lambda GH200.

  • The first task was a team fight. It converged in ~100 rollouts to not fighting: running out the clock is worth 0 to both sides, a fight is negative for the weaker comp, so "don't fight" is a Nash equilibrium of that reward and PPO found it. Draw penalties didn't fix it; switching to full games with towers and farm did.

  • Most training games are capped at 12 minutes of game time and start from a "tower pressure" state (5–20 min in, heroes pre-levelled, outer turrets damaged).

What it learned, measured

  • Baseline: with all ten heroes idle, minion waves alone take 6.83 turrets, 0.01 inhibitors and 0 nexuses per game. The trained policy adds ~0.5 turrets, nearly every inhibitor, and every nexus kill you see. Most turret kills in any video are the waves finishing the job.

  • Against the server's built-in bots (intermediate bots): 55 wins, 8 draws, 1 loss of 64. Caveat: about two bots per game still idle a lot, so this is a floor, not a strong opponent.

  • Against heroes that do nothing, from spawn: it takes the nexus in 0% of games within the 12-minute cap it trained on, and 98% within 30 minutes (median ~21 min). It can close; it was never paid to.

  • Why: γ = 0.995 at 133 ms a decision is a ~27-second reward horizon. Retaking an inhibitor that respawns pays within that horizon every time; walking to a nexus through a fountain laser doesn't. The ladder plateaued there. We've just restarted with an 18-minute cap and a longer horizon; early rollouts show more nexus kills, but it's hours old.

Other caveats

  • The side imbalance is partly the game's content: purple's inner turrets spawn with 1,300 HP vs blue's 2,550 in this build (a 0.01 s buff in the original scripts), which the server reproduces faithfully. The two comps aren't mirrored either.

  • Early checkpoint, one box, one run. No human baseline yet.


r/reinforcementlearning • • 6d ago

First Q-Learning project

Post image
5 Upvotes