r/reinforcementlearning • u/felixr314 • 5h ago
SmartGridGuardian: Asymmetric Multi-head AlphaZero with METIS-Core v0.4.2-alpha
Hi,
Here is another example of the asymmetric multi-head AlphaZero that METIS-Core implements. In this case, the problem (the environment) is a network of electrical substations.
The grid is a simple approximation: 4 substations in a ring, DC power flow, cascading disconnection of overloaded lines, and a daily demand curve with an evening peak. Grid2Op (sequential power-grid simulation) could have been used, but the goal was to test the framework on a very different problem.
đˇ Team Blue: the 4 substations, which must keep the power supply running. They see the grid through sensors that may be spoofed.
đś Team Red: two hackers, a Striker (fake-demand botnets and line tripping) and a Spoofer (falsifies Blueâs sensor readings). Their goal is a blackout (less than 78% of real demand served). Nobody programs how they should coordinate. To speed up learning, I did add a hand-written heuristic with a 30% weight in the MCTS evaluation; it gives a small bonus when an attack is hidden by spoofing, but it doesnât say what to attack or when.
During training, Team Red (the hackers) discovered these strategies on its own:
âď¸ Phantom Load at Peak: two botnets add fake demand hours before the peak and lines are cut during the day; at the evening peak the grid canât cope and collapses.
âď¸ Critical-Node Double Strike: they overload the highest-demand substation and cut its supply lines; at the peak it fails and brings down the whole grid.
âď¸ Tail-of-Peak Strike: Red strikes later, around 21:00, when the peak is fading but demand is still high.
âď¸ Substation Isolation + Sensor Spoofing: they repeatedly cut the lines feeding the highest-demand substation and, just before the collapse, falsify its readings or raise false alarms at other substations so Blue canât react in time.
The most interesting part was watching the arms race: Blue learned to shut down an attack, Red took tens or hundreds of episodes to find a new target or a new time of day, and the cycle started again. Blue even âforgotâ how to defend against an old attack, and Red reused it about 500 episodes later.
This is a toy research environment to test multi-agent self-play; the point is to train better defenders by letting them face an adaptive attacker.
Related paper with the same idea (an attacker and a defender learning at the same time on a power grid), "Adversarial Multi-Agent Reinforcement Learning for Proactive False Data Injection Detection" from NREL: https://arxiv.org/abs/2411.12130
Links:
đ METIS-Core on GitHub: https://github.com/felixromo314/METIS-Core
đ SmartGridGuardian tutorial: https://github.com/felixromo314/METIS-Core/blob/main/tutorial/Tutorial_AlphaGoZero_SmartGridGuardian.md
đ Source Code: https://github.com/felixromo314/METIS-Core/tree/main/examples/SmartGridGuardian