Hey everyone,
If you’ve ever left a fine-tuning run, an RL policy update (PPO/GRPO), or a mixed-precision (FP16/FP8) job running overnight only to wake up to a silent NaN explosion, reward hacking, or a flatlined loss curve, you know how painful wasted GPU hours can be.
Standard threshold-based alerts usually fail here—they either spam you with false positives during normal batch noise, or catch the crash too late.
To solve this, I built limen_pulse (v0.6.0)—an adaptive, real-time training stability monitor and crash auditor.
How it works under the hood:
Zero-Math Lightweight Client SDK: You drop a clean, isolated Python client into your training loop. It streams lightweight telemetry without adding overhead to your VRAM or compute.
Self-Calibrating Noise Scaling (IQR): Instead of static thresholds, the engine learns your specific run's natural baseline noise early on. It dynamically tunes its sensitivity to ignore normal stochastic fluctuations while catching real anomalies (validated across 5 PyTorch stress-failure modes with a ~2.5%–3.4% false-positive rate).
Regime-Aware Monitoring: It adapts its detection strategy depending on what you are training:
Supervised Learning: Distinguishes normal mini-batch variance from genuine plateaus or LR-induced divergence.
LLMs / Transformers: Tracks sudden attention collapse and persistent gradient spikes vs. transient noise.
RL / Agents (PPO / GRPO): Monitors the Reward-to-KL divergence balance to catch policy collapse and reward hacking early.
Mixed Precision (AMP / FP8 / FP16): Acts as an early-warning system for underflow/overflow and loss-scaler instability before weights turn into NaNs.
Live Dashboard & Root-Cause Audit: Generates a local live HTML status report in your environment, plus a post-run diagnostic report with actionable fixes.
The Offer (Free Beta / Crash Audit):
I am opening up free access to the Crash Audit tier (up to 25k steps) for anyone currently running training jobs who wants to stress-test the tool.
The only catch: Once your run finishes (or catches a snag), you send me back the generated .md / .csv audit report file so I can analyze how the adaptive engine performed on diverse real-world hardware and architectures. (Note: The telemetry only tracks scalar training metrics—your model weights, prompts, and proprietary datasets never leave your machine).
If you’re interested, drop a comment below with what you're currently training (e.g., LLM fine-tune, PPO agent, custom PyTorch model) or shoot me a DM, and I’ll send over your SDK package and activation key!