r/mlops • • 10m ago

Discussion What if an AI protocol had to prove not only what it can calculate, but also what it cannot yet justify?

2 Upvotes

One thing I am learning while developing the Organic Intelligence Protocol (OIP): a mathematical formula is not evidence, and passing software tests is not scientific validation.

Recently, I completed a focused implementation cycle:

- 21/21 targeted policy tests passed.

- 70/70 existing software regression tests passed.

- Seven identified implementation gaps were addressed in a new candidate version.

- The original candidate remains unchanged.

But there is an important distinction: these results establish software level behavior under the tested conditions..They do not establish empirical effectiveness or scientific validity.

I’m now working on a versioned canonical serialization specification, with cross-platform reproducibility still to be established.

I would rather expose what remains unproven than present an unfinished protocol as a breakthrough.

For researchers and engineers.. What would you independently test before considering this kind of protocol specification reliable?

Critical feedback is welcome.


r/mlops • • 7h ago

Discussion Serving many fine-tunes of one LLM: adapters 55× smaller than LoRA, same accuracy. Looking for feedback from people running this in production

4 Upvotes

Title: Serving many fine-tunes of one LLM: adapters 55× smaller than LoRA, same accuracy. Looking for feedback from people running this in production


Disclosure: I'm the founder of ThousandML and built what's described here. I'm posting because I'd like input from people who run LLM serving infrastructure.

The problem. If you serve one fine-tune per customer or task on a shared base model, the adapters themselves become an ops cost. LoRA r=32 on a 27B model is ~320 MB per adapter. At hundreds of tenants, that's tens of GB to store, ship, cache and keep resident in GPU memory, and it limits how many variants one GPU can hold.

What we built. GCMP is an adapter format that replaces LoRA. On Qwen3.8-27B:

  • Size: 2.9M trainable parameters, ~11 MB per adapter, vs ~159M and ~320 MB for LoRA r=32 on the same projections.

  • Accuracy: same as LoRA r=32 on 9 benchmarks (GSM8K, MMLU, SQuAD, IFEval and others). Every comparison uses the same evaluation code and paired significance tests, on a full-precision BF16 base and on an NF4 base.

Base + adapterSizeGSM8KSQuADMMLUIFEval
BF16 + LoRA r=32~320 MB0.9030.7760.8440.825
BF16 + GCMP~11 MB0.8900.7600.8460.815
NF4 + LoRA r=32~320 MB0.7930.7480.8400.835
NF4 + GCMP~11 MB0.8330.7560.8370.840

None of the differences are significant, except GCMP +4 on GSM8K on NF4 (p=0.04).

Serving (vLLM plugin).

CQ_GCMP_ADAPTERS="tenant-a=/adapters/a,tenant-b=/adapters/b" \
vllm serve <base-model> --quantization cq
# request picks the adapter by model name: {"model": "tenant-a", ...}
  • Loading: all adapters load once at startup, and each request is routed by model name. /v1/models lists them.

  • Mixed batches: requests for different adapters run in the same batch, with one kernel call per layer and no merging or swapping.

  • Resident GPU memory: about 11 MB per adapter. The fixed overhead is a ~1 MB per-token routing buffer and a shared workspace of ≤ ~64 MB. Neither grows with adapter count, and no dense per-adapter weights are ever built.

  • Swaps: adapters can be swapped in place by name without restarting.

What's not ready:

  • Measured scaling: the "1,000 adapters ≈ 11 GB" figure is calculated, not yet measured. Memory and throughput at 10, 100 and 1,000 adapters, compared with vLLM's built-in multi-LoRA, is the next benchmark.

  • Fixed adapter set: the set is fixed at startup. Adding a new adapter needs a restart; swapping an existing one doesn't.

  • Base model: vLLM serving currently needs our CQ 4-bit base. Full-precision bases are next.

  • Parallelism: pipeline parallel works; tensor parallel doesn't yet.

  • Training speed: about 2× LoRA's step time.

Everything runs on your own GPUs. The Python code is open source; the kernels are compiled with a license check, free during early access.

My questions for you:

  1. If you serve multiple fine-tunes today, what limits you first: adapter memory, load or switch latency, throughput with mixed batches, or something else entirely?

  2. What would you need to see in a scaling benchmark before trusting this over vLLM multi-LoRA?


r/mlops • • 21h ago

Discussion [D] What becomes the biggest challenge after a machine learning model is ready for deployment?

0 Upvotes

Getting a machine learning model to perform well during development is one thing. Keeping it reliable once it's running in production is another.

I've noticed that production deployment involves several challenges beyond model accuracy:

  • Monitoring model performance and detecting drift

  • Managing different model versions

  • Automating retraining and deployment workflows

  • Keeping data pipelines consistent

  • Tracking infrastructure and inference costs

  • Making sure models remain reliable as business requirements change

I'm interested in hearing from people who have taken ML models into production.

Which part of the MLOps lifecycle tends to require the most effort in your experience: deployment, monitoring, retraining, or ongoing maintenance?


r/mlops • • 2d ago

Discussion I benchmarked 3 self-hosted LLM gateways. Langfuse's was fastest, but its stack consume 3.5 GB RAM at idle stage

11 Upvotes

Setup. Langfuse AI Gateway 4.77.0 (public beta), LiteLLM proxy 1.108.4 with Postgres and a virtual key, and AcruxCore from its Docker Hub images. All three ran on one Ryzen 9 5900X, called the same OpenAI model with a 5-token cap, and used the same prompt. Latency is the gateway call minus a direct OpenAI call made in the same round. Each run had 150 rounds in random order, and I report the median with a 95% bootstrap interval.

Results

TestLangfuseLiteLLMAcruxCore
Added latency, no streaming−9 to +8 ms (interval includes zero)+61 to +67 ms+15 to +24 ms
1,000 streams on 2 CPUs, as shipped261 ok403 ok1,000 ok
1,000 streams, open-file limit raised1,000 ok, first token 1.7 s1,000 ok, first token 19.2 s1,000 ok, first token 3.8 s
Idle memory, gateway process13 MiB727 MiB200 MiB
Idle memory, whole stack3,584 MiB810 MiB405 MiB

What I took away

  • Langfuse's gateway is the fastest. It is a small Rust process, and with the limit raised it handled 1,000 streams best. But it cannot run alone: it checks every key against the Langfuse web service and records usage into ClickHouse. If you already run Langfuse, it costs you 13 MiB. If you don't, it brings a 3.5 GB stack with it.

  • Docker's default open-file limit of 1,024 decided the concurrency test. Each stream holds two sockets. Node raises its own soft limit at startup, but Rust (Langfuse) and Python (LiteLLM) keep 1,024. One ulimits: nofile line in Compose fixes it. Check /proc/<pid>/limits on your own gateway.

  • LiteLLM's 503s at high concurrency came from virtual-key lookups. When the key is not cached, every call queries Postgres through a pool of 10 connections, and a lookup over 10 s gets a 503. Making one call first, so the key is cached, removed them.

  • The OpenAI address matters more than the gateway. api.openai.com resolved to two Cloudflare addresses for me, and a new connection to the far one added ~250 ms. In my first run this made Langfuse look 43 ms faster than calling OpenAI directly. I pinned every path to the same address after that. If you benchmark gateways yourself, check this first.

Limits of this test: one machine, one network, short prompts, and one version of each. Langfuse can't point at a custom upstream URL, so I couldn't use a local mock model, and differences smaller than the intervals aren't measured.

If you rerun it and get different numbers, please post them. I'll update the write-up.

Disclosure: I build AcruxCore, one of the three gateways in this test. So you have every reason to distrust a benchmark that a vendor publishes on its own website. I would too. That is why everything is open source: the scripts, the Compose overrides and the raw JSON results. You can rerun the whole thing on your own machine and check whether you get the same numbers.


r/mlops • • 2d ago

Discussion Benchmarking "System One" decision APIs vs. in-process tree ensembles + exact TreeSHAP on 64-feature tabular payloads [D]

1 Upvotes

There has been a lot of discussion lately around "System One" decision models: fast, small language model endpoints tuned to return structured routing choices, rubric scores, or risk probabilities inside application code for ~90-210 ms at $100-$345 per 1M calls.

If the input is a messy PDF, support ticket, or audio transcript, calling a small multimodal model makes complete sense.

Where I still see teams get burned is routing structured numeric/categorical feature vectors (say, 20 to 200 features inside a fraud, underwriting, or ranking hot loop) through a language model endpoint so they can get a score plus a natural-language "reason code" in one API call. Aside from the 150 ms network RTT and token cost, even constrained decision endpoints still hit ~1.5% content refusals on edge-case inputs, and generative reason codes are not true feature attributions.

While running an 18-model verification matrix for SMILE 7.2 (smile-serve running both on HotSpot JVM and as a GraalVM native image at ~140-164 MiB idle RSS), I measured where the latency actually goes when you pair classical tree ensembles with exact per-row TreeSHAP in-process. A few numbers stood out:

  1. Scoring itself is almost free. Across 50 to 500 trees and 20 to 200 features (500 rps, concurrency 8, coordinated-omission corrected on an AMD EPYC 7401P), single-row GradientTreeBoost and RandomForest scoring stayed at 0.24-0.65 ms p50 and 0.47-1.57 ms p99 over loopback HTTP.

  2. Exact TreeSHAP is where naive Random Forests blow up. Scoring a tree scales linearly with depth O(T * D), but exact TreeSHAP scales with depth squared, O(T * L * D^2). On an unconstrained 300-tree, 64-feature RandomForest (max_depth=20), single-row scoring took 0.41 ms p50 / 0.92 ms p99, but adding ?explain=true pushed latency to 192.5 ms p50 / 274.4 ms p99 (worse than a cloud language model call).

Here is the depth/leaf ablation on the 300-tree, 64-feature Random Forest (JVM, 100k measured requests after 5k warm-up):

RF Hyperparameters (300 trees, 64 features).sml SizeCold StartScoring p50 / p99Exact TreeSHAP p50 / p99500 rps p99Stream (rows/s)
max_depth=20, max_nodes=0 (unbounded)16.8 MB12.50 s0.41 ms / 0.92 ms192.5 ms / 274.4 ms12.9 ms5,201 /s
max_depth=10, max_nodes=03.40 MB3.91 s0.31 ms / 0.98 ms22.5 ms / 28.9 ms2.43 ms9,195 /s
max_depth=8, max_nodes=2561.56 MB2.49 s0.28 ms / 0.56 ms7.81 ms / 9.09 ms0.90 ms11,039 /s
max_depth=6, max_nodes=128637 KB2.32 s0.26 ms / 0.54 ms2.14 ms / 2.88 ms0.78 ms14,662 /s
GradientTreeBoost (300 trees, 64 features)292 KB1.55 s0.24 ms / 0.47 ms0.69 ms / 0.91 ms0.64 ms19,241 /s

3. One interesting failure mode we hit on GraalVM Native Image: while GradientTreeBoost passed <5 ms p99 TreeSHAP across all 9 matrix cells (1.26 ms to 3.97 ms p99 even at 500 trees x 200 features), the 300-tree RandomForest on Native Image had a bimodal tail. 97.7% of requests finished in under 5 ms (p50 = 4.48 ms, p95 = 4.54 ms), zero requests landed between 9.1 ms and 33.3 ms, and 1.16% clustered at 33.8-34.3 ms. Those spikes came from per-tree path allocations inside TreeSHAP triggering GraalVM Native Image's default Serial GC (~29 ms stop-the-world pause roughly once every 85 requests), whereas HotSpot's G1GC kept p99 at 2.88 ms and p99.9 at 3.97 ms.

The takeaway for hybrid orchestration + ML pipelines is pretty simple: use small language models when you need to extract structured fields out of messy text or documents, and keep closed-set scoring, routing, and exact SHAP attributions in-process under 3 ms.

Code and benchmark harness are in the SMILE repo if anyone wants to reproduce or poke at the TreeSHAP implementation: https://github.com/haifengl/smile

For folks running real-time SHAP or feature attributions in production latency budgets (<10 ms), are you constraining tree depth at training time, using shallow boosted trees over bagged trees, or approximating explanations asynchronously off the hot path?


r/mlops • • 3d ago

Discussion Can you tell good MLOps work from a public repo alone? Here's what I look for. Tell me what I'm missing

5 Upvotes

I'm a 17 year old founder building an app called AI-Atlas, it currently has 70+ users and it basically evaluates people's public AI work across six fields; MLOps is one of them. MLOps is the hardest to judge from code, because so much of it lives outside the repo.

Also: I am passionate in AI Research, so I didn't have time to catch a glimpse over what criteria exactly does a MLOps project need to have to be considered excellent. Likewise, what or how could I identify any lacking areas that deserve to be showcased by the final score on that category, so that's why I decided to write in this sub, I am genuinely asking you guys for advice and feedback!

The signals I use:
- Reproducibility: pinned environments, seeded training, a single command that reproduces the reported numbers.
- Pipelines: training and serving split cleanly, configs instead of hard-coded paths.
- Delivery: CI that actually tests the model code, not just linting; containers and model packaging.
- After deployment: monitoring and drift checks, when they're present.

So what would you add, or weight differently? Someone with solid infra-as-code but no model training: should that count as MLOps at all?

Happy to share my project for anyone genuinely interested :)


r/mlops • • 3d ago

Discussion For those running disaggregated inference, how do you handle failures between prefill and decode?

3 Upvotes

We're looking into disaggregated inference for an agent platform that's starting to handle more traffic. One thing I'm trying to understand is how reliable the handoff between prefill and decode is in production.

Separating these stages across different hardware makes sense since they have different resource requirements. But what happens when prefill finishes and the decode worker fails to pick up the request? Does the system retry the handoff, or does the entire request have to start again? I'd rather understand these failure scenarios before building our serving infrastructure around this architecture.

For anyone running split inference in production, how do you handle retries, cancellations, and failed handoffs? Is this something your inference provider manages, or does your team have to build that logic itself?


r/mlops • • 3d ago

Discussion Fo u use Databricks Genie in prod for MLOps / data pipelines?

1 Upvotes

Does Databricks Genie (Code / ZeroOps) actually save engineering time on pipeline debugging in prod , or is maintaining clean UC metadata more work than it is worth?
What are ur experience on it


r/mlops • • 4d ago

Tales From the Trenches How we built LRU model eviction on top of SGLang to serve 139 open models across 23 agent tasks on one Kubernetes cluster (Apache-2.0)

4 Upvotes

When you run an agent pipeline in production, the main LLM isn't the only thing eating VRAM. A single request often hits a prompt guard (Qwen3Guard-4B), an encoder (Qwen3-Embedding-4B), an entity/intent extractor (GLiNER2.5), a generator (Qwen3.8-27B-FP8), and a verifier (Granite-Guardian-2B).

Deploying those the standard way means running vLLM for guards, N separate TEI containers for embeddings, a custom FastAPI service for GLiNER, and SGLang for generation, each locking down its own GPU memory slice even when idle.

We built the Superlinked Inference Engine (SIE) under Apache-2.0 to collapse that fleet into one Helm chart and a single endpoint:

- The 27B LLM stays pinned on its own GPU pool via SGLang so it never gets evicted.

- Guards share one GPU pool and encoders share another.

- Models load on demand, batch concurrent requests, and use Least Recently Used (LRU) eviction when VRAM gets tight.

- If an individual SGLang process crashes, SIE unloads it and reloads cleanly on the next request.

Moving high-frequency retrieval and extraction off per-token APIs onto shared GPUs reduces the cost of self-hosted embeddings to ~$0.50 per 1B tokens (vs ~$20 on managed APIs) and the cost of reranking to ~$8.50 per 1B tokens (vs ~$87 on Cohere Rerank).


r/mlops • • 4d ago

Discussion Need a secure local AI setup with zero cloud egress for summarising heavy legal contracts and docs. What platforms should I evaluate instead of sending everything to Claude/ChatGPT?

1 Upvotes

Our legal team wants AI to summarise long contracts, and the hard requirement from our insurer is that nothing leaves our servers. No public AI service, and nothing that sends prompts or documents to somebody else’s cloud. I got handed the job of finding something that runs on hardware we control, and so far almost everything I find is built around a cloud dashboard first.

From what I’ve read, the options seem to fall into three groups. There are open source chat front ends you self host and attach a local model to, some self hosted AI platforms that say they run on premises, and legal specific tools that are cloud only once you read the small print. The first group looks doable but we would have to maintain it. The second is hard to judge from marketing pages. The third is out.

Hardware wise there are two servers with a couple of graphics cards and one person who is comfortable with Docker but already has a full plate. What are you all running for this kind of confidential document work, and what did you land on?


r/mlops • • 4d ago

MLOps Questions How much do you lose if your GPU workload gets interrupted?

6 Upvotes

When you run large AI workloads sometimes there is more than just the cost of GPU hours. Sometimes your training will take a long time to run and then fail due to infrastructure issues, unexpected restarts or interruption but i think with checkpoints this may not be too bad but you will lose some computation and have to reload your data, debug and then delay your overall experiment. How do you factor in this cost into your overall training costs?

Do you use lots of checkpoints and auto recovery or make sure your infrastructure is stable? When choosing a cloud provider for your GPU, AWS, CoreWeave, Lambda, Yotta Labs etc how much do you factor in workload reliability?


r/mlops • • 4d ago

Self-promotion How I check slow PyTorch steps while a job is still running

10 Upvotes

I've been working on TraceML for a while. It shows where PyTorch training time goes, especially when one rank in a distributed job starts falling behind the others.

A while ago, an ML platform team asked us if they could pull those timings into their own experiment tracker. So we built it and thought it might be useful to share here.

What it does

TraceML 0.5.1 can write one JSONL record per step, per rank, while the job is still running. If your DDP script is already instrumented with TraceML:

pip install "traceml-ai==0.5.1"
traceml run --exporter jsonl --nproc-per-node=2 train.py

The output goes to logs/<run-name>/aggregator/export/step_time.jsonl.

A small example

I ran a demo with two CPU ranks and added 120 ms to rank 1's batch collation on every other step. These are the CPU wall times (ms):

{"step":1,"rank":0,"cpu_ms":{"input_wait":2.06,"forward":3.98,"backward":25.71,"optimizer_step":4.25,"traced_step":36.36}}
{"step":1,"rank":1,"cpu_ms":{"input_wait":2.05,"forward":3.98,"backward":25.71,"optimizer_step":4.25,"traced_step":36.37}}
{"step":2,"rank":0,"cpu_ms":{"input_wait":0.05,"forward":0.08,"backward":0.49,"optimizer_step":0.18,"traced_step":126.81}}
{"step":2,"rank":1,"cpu_ms":{"input_wait":125.15,"forward":0.07,"backward":0.49,"optimizer_step":0.18,"traced_step":1.66}}
{"step":3,"rank":0,"cpu_ms":{"input_wait":0.07,"forward":0.04,"backward":0.42,"optimizer_step":0.18,"traced_step":0.86}}
{"step":3,"rank":1,"cpu_ms":{"input_wait":0.06,"forward":0.04,"backward":0.45,"optimizer_step":0.17,"traced_step":0.85}}
{"step":4,"rank":0,"cpu_ms":{"input_wait":0.07,"forward":0.05,"backward":127.26,"optimizer_step":0.55,"traced_step":128.23}}
{"step":4,"rank":1,"cpu_ms":{"input_wait":125.35,"forward":0.27,"backward":1.12,"optimizer_step":0.53,"traced_step":2.7}}

A few notes on reading it:

  • traced_step is the timed training block. The DataLoader fetch happens before it, so it shows up separately as input_wait.

  • Every phase has cpu_ms and gpu_ms. gpu_ms comes from CUDA events, so it's null in this CPU run.

Step 4 is the one I find interesting. Rank 1 spends 125 ms waiting for its batch. Rank 0 has no data problem at all, but it shows 127 ms in backward. That's because DDP syncs gradients during backward, so rank 0 just sits there waiting for rank 1 to catch up.

If you only looked at rank 0, you'd deduce that backward was slow. Looking at both ranks side by side, you can see the real problem is rank 1's data loading.

Where this is going

Next, we'd like to support:

  • Weight & Biases

  • MLflow

  • TensorBoard

  • Grafana / Prometheus, for teams that watch training like any other service

  • Whatever else people actually use (Aim, Comet, something in-house?)

How you can help

If you run training jobs and want these timings in your dashboard or tracker, I'm happy to help you write exporters, or build one together that we can ship in TraceML.

Also I am curious how others find the slow rank without something like this. I used to add per-rank timers and diff logs.

GitHub: https://github.com/traceopt-ai/traceml/

Full disclosure: I'm maintainer of TraceML. It's open source. Mainly looking for feedback and people who want to build exporters with us.


r/mlops • • 4d ago

Discussion DevOPS to MLOPS - Worth Switching and the guidance

8 Upvotes

I am DevOPS, with around8-9 years of exp. mostly contracting. I have done some AI deployments and solutions as well. I am thinking to switch to something diff now. MLOPS comes into my mind. Is it worth ? based on current market trends. I can ask this on AI and get some plans but looking answers from contractors in the UK how's the market and job and money into it ?


r/mlops • • 4d ago

Tales From the Trenches Bigger CPU machine, slower inference: PyTorch's idle threads were the cause

8 Upvotes

While benchmarking a YOLO model exported to ONNX on CPU (AWS c7i, ONNX Runtime), I ran into an unexpected slowdown. Processing 1,000 images took 40 s on 4 vCPUs but jumped to 59 s on 8 vCPUs with the exact same code and model.

Splitting the timing by stage showed the ONNX model itself was fine. The bottleneck was pre- and post-processing, which still ran in PyTorch. Preprocessing jumped from under 2 ms per image to 21 ms—taking longer than the model inference.

Why This Happens

  • Thread Overload: By default, PyTorch creates a thread pool sized to the machine (7 threads on an 8-vCPU box). When combined with ONNX Runtime's own threads, you end up with more active threads than physical CPU cores.

  • Aggressive Spinning: PyTorch’s OpenMP threads use "busy-waiting." Instead of sleeping after finishing a task, they spin in a loop to stay ready for the next operation.

  • Gridlock: Every small PyTorch operation is split across 7 threads and can't finish until the slowest thread is done. Meanwhile, the threads that finish early keep spinning and hogging the CPU cores that the lagging threads desperately need to finish.

What I Tried

  • Forcing threads to sleep (OMP_WAIT_POLICY=PASSIVE): This dropped the ONNX benchmark from 58s down to 22s. However, it made a standard, standalone PyTorch model 3.7x slower (from 26s to 94s) because busy-waiting is what normally keeps pure PyTorch fast. It wasn't a safe global fix.

  • Capping PyTorch at 2 threads: Limiting PyTorch to 2 threads when it only handles pre- and post-processing cut the 8-vCPU runtime from 59s down to 27s (while keeping the 4-vCPU speed about the same). This fix was ultimately merged into Ultralytics.

Same cause, bigger effect:
Running 4 plain PyTorch processes on the 8-vCPU box (250 images each) took 1,361 s—whereas 1 process doing all 1,000 images took just 26 s. Setting torch.set_num_threads(1) in each process dropped the 4-process run to 15 s.

Takeaway:
If PyTorch runs alongside another runtime, or you run multiple PyTorch workers on one machine, set the thread count yourself instead of trusting the defaults. Use torch.set_num_threads at runtime (since OMP_NUM_THREADS only works if set before importing torch).

https://medium.com/@thevarunfreelance/why-may-a-model-get-slower-on-a-bigger-machine-6beef626250b


r/mlops • • 5d ago

(Gen)AI / Agents / LLMOps Airlock, offline verification for model files

0 Upvotes

A small Go CLI for moving models into air-gapped networks.

  • Online: checks files against a pinned Hugging Face commit, refuses pickle, signs a manifest with cosign.

  • Offline: verifies the signature and every hash, using only the Go standard library.

  • Zarf: runs as a deploy step, so a swapped model never starts (demo).

No rollback or chat template checks yet. Feedback appreciated.

GitHub · Site


r/mlops • • 5d ago

Discussion What do you have candidates do in an MLOps interview?

8 Upvotes

If you've interviewed someone for an MLOps role recently, what did you ask them to do? How did you assess their troubleshooting? Talking through a past incident gives them a problem they already know the answer to. Giving them something from your own stack could mean spending half the interview explaining the setup. What's worked for your team?
If you've joined a team recently, did the interview resemble the job?


r/mlops • • 6d ago

Self-promotion digline 1.2 adds the tools before the gate: record LLM calls without code changes, get a first suite proposed, know the next step

1 Upvotes

Disclosure, it’s my project (open source, Apache-2.0).

digline is a regression gate for LLM apps and agents: a run you approved is committed in the repo as the reference, every later run is compared to it in CI, exit 1 if something got worse beyond the measured judge noise.

What’s new is the step before the gate. A recorder keeps the calls where the app makes them, without code changes: wrapper for the Python SDK clients, interceptor for Java and Node, local proxy for anything else. Off by default, switched on per tenant, sampling and redaction declared, recordings stay in .digline/<tenant>/ and never leave the machine. An agent turn is kept as one episode with its tool calls.

From the recordings digline suggest proposes a first suite, digline where tells you the next step, and the same is exposed over MCP without a promote tool, so an agent can measure but a person approves the reference.

1.2 is the security pass on the recorder: three advisories found the day after 1.1, fixed, in the changelog.

Curious how people here handle the prod side: do you let recorded production traffic into your eval set at all, and who decides which samples count?


r/mlops • • 7d ago

Discussion Notes from a two night vision training test on an 8x RTX PRO 6000 box: scaling gaps, CPU bottlenecks, and an INT8 collapse

2 Upvotes

I work at B3IQ, we build and host these systems, what are your vision use case bottlenecks? We just spend two nights benching a few vision use-cases on an 8x NVIDIA RTX PRO 6000 Blackwell Server Edition 96GB bare metal node (128 cores, 756GB RAM). Thought it would be interesting to get some feedback what the bottlenecks are, once you iron out the hardware. Trained on COCO 2017 and Food 101.

-> It ran for 19 hours straight (YOLO11x, 600 epochs), no need to restart. Total training time was 19h10 (115 seconds per epoch), average GPU utilization was 86%, average power consumption per GPU was 415W. Highest temperature was 76C on one of the GPUs, no thermal throttling. Best model performance was reached at epoch 412 (13th hour of training, mAP50-95 0.527). After that it was overfitting and at the 19th hour it was at 0.490. Note that this is 2 points lower than the 0.547 Ultralytics publishes, we did not tune the training recipe

-> 1 vs. 8 GPUs: With YOLO11m, speed went from 409 seconds per epoch down to 64 (6.4x). GPU utilization was similar (71% vs. 75%), the rest of the speed-up should come from the fact that power consumption went from 344W down to 220W per GPU. So they were waiting for the CPU to augment and sync the photos.

-> 32 models at the same time: While training 32 MobileNetV3-Small models (1.5M weights), CPU utilization was at 94% (128 cores) due to decoding the photos from JPEG, while GPU utilization was just 10-14%. Also, after converting the model to INT8, accuracy went from 94.6% all the way down to 9-15%. This was due to the first two layers having 6x wider activations (-141 to +143). Leaving the first two layers in float format brought the accuracy back up to 89.2%. Converting a MobileNetV2 to INT8 did not have this issue, and it achieved 96.4%

-> The only thing that “broke” was that we filled up the local NVMe drive with 20GB of checkpoints, and the script that pulled them off the box died. Our dashboard that logged the hardware data worked perfectly for 24 hours, it even logged when we SSH’d into the box. The only issue with it was that it could not log any training events, so we had to stitch the training log with the hardware data manually.

How much do you think these PCIe bound bottlenecks would go away with moving more stuff to the GPU with DALI / nvjpeg?


r/mlops • • 10d ago

Self-promotion Four alerts for LLM pipelines that read long inputs: treating truncation as a data-quality metric

2 Upvotes

Long documents were silently cut off before reaching the model, outputs stayed well formed and error-free, and the problem surfaced weeks later when someone checked a contract by hand.

The point for MLOps: “no errors and valid schema” says nothing about whether the input was complete. Here are four alerts I’d add. Thresholds depend on your data, so I’ve left them as placeholders.

1. Coverage ratio

  • Signal: tokens_sent / tokens_in_source, per document.

  • Condition: any document below 1.0 when your design says everything should be sent, or the share of documents below 1.0 rising above a baseline.

  • Action: fail the run or route to a fallback (splitting, human review) instead of slicing silently.

2. Truncation counter

  • Signal: a counter incremented whenever any preprocessing step shortens the input, with the step name as a label.

  • Condition: counter greater than zero in a path that should never truncate.

  • Action: alert, so a “temporary” limit left in the code doesn’t quietly change production behavior.

3. Input length distribution drift

  • Signal: the distribution of source lengths per day or week.

  • Condition: a shift versus a reference window (for example, more long documents than the test set contained).

  • Action: review whether the test set still resembles production, and rebuild it with the longest cases.

4. Tail match

  • Signal: the model is prompted to quote the final sentence it received, and the pipeline compares it with the end of the source.

  • Condition: mismatch rate above a baseline.

  • Action: investigate. It’s a heuristic, since a model could produce a plausible ending without having seen the real one, so it should be paired with alert 1.

What the postmortem also suggests

  • Record input length and sent length for every run. In the scenario, the team couldn’t tell which past outputs were affected because this wasn’t logged, and finding out took longer than fixing the bug.

  • Include the longest documents, and ones where the decisive clause sits at the end, in the evaluation set.

  • Sampled human audits should look at the end of long inputs, not only the beginning.

Limits: these alerts detect that the input was incomplete, not that the extraction was correct. They also add overhead and some noise, and they depend on a clear definition of “everything should have been sent”.

The full hypothetical postmortem is here, in case it’s useful: https://medium.com/@nagatomopedro05/postmortem-the-contracts-pipeline-that-never-read-the-last-page-cf84251671c8

Which input-quality metrics do you monitor for LLM or document pipelines, and what do you alert on?


r/mlops • • 10d ago

Discussion MLOps folks at big companies: what does the role and interview process look like?

41 Upvotes

Hey everyone, I just started an MLOps job and I’m trying to understand how the field works at bigger companies. It still feels pretty new, so titles and expectations seem to vary a lot. If you work in MLOps at a large company, what’s your role and day-to-day, and what did the interview process look like (number of rounds, coding, system design, ML fundamentals, infra/DevOps questions)? Also curious which tools they asked about and what you’d recommend learning to grow in the role. Any advice is appreciated, even a short answer!


r/mlops • • 10d ago

Discussion I published a research-stage framework for evaluating the financial value of AI systems

4 Upvotes

I have published a new document from my Organic Intelligence Protocol (OIP) research.

The Enterprise Financial Evaluation Framework covers Total Cost of Ownership, ROI, NPV, baseline costs, data quality and operational risk.

It focuses on measuring business value using company-specific data rather than assumed savings.

OIP remains research stage.. This document does not claim empirical or commercial validation.

Report: https://zenodo.org/records/24005418

I welcome constructive technical and financial feedback.


r/mlops • • 10d ago

Self-promotion AI Infra Precheck – A Practical GPU Cluster Health & Troubleshooting Checklist

1 Upvotes

I've been working on an open-source project called AI Infra Precheck, a practical checklist for validating GPU cluster infrastructure and troubleshooting distributed AI training issues.

In large-scale GPU clusters, problems like degraded PCIe links, GPU health issues, RDMA connectivity failures, and NCCL communication bottlenecks can go unnoticed until training performance starts to suffer.

I put together this checklist based on real-world GPU cluster operations, covering areas such as:

  • GPU Health & Topology: GPU availability, P2P, NVLink, GPU health and XID errors.

  • PCIe: Link speed, link width, PCIe topology and ACS configuration.

  • AI Networking: InfiniBand / RoCE, RDMA connectivity, GPUDirect RDMA (GDR) and MTU.

  • Performance Validation: NCCL Tests, RDMA bandwidth testing and communication bottleneck checks.

  • Troubleshooting: Identifying potential hardware issues, slow GPU nodes and infrastructure-related training performance problems.

The goal is to help AI Infra engineers and SREs identify infrastructure issues before they turn into failed NCCL tests, slow AllReduce operations, or degraded distributed training performance.

This is currently a documentation-driven checklist, not an automated diagnostic tool. I'm hoping to expand it with more practical checks and troubleshooting cases from real-world GPU/HPC environments.

GitHub: https://github.com/LiquidFLOPS/aiinfra-precheck

Feedback, suggestions, and contributions are very welcome! I'd especially appreciate insights from people operating large-scale GPU/HPC clusters.


r/mlops • • 11d ago

Self-promotion Would you use Scala/Cats-style IO and fibers in a Python-like language (Mojo)?

1 Upvotes

Disclaimer: I'm the author of fp_mojo (docs), and I'm posting here to get some feedback from the broader data engineering community on the concept of integrating Scala / Cats Effect async and concurrency primitives into a Python-like language.

If you've worked in big data infra, fintech, distributed systems, or any other field where correctness and resiliency matter, you've probably used (or at least crossed paths with) Scala and Cats Effect / ZIO.

Scala's concurrency and FP primitives are great, but the ecosystem as a whole is dealing with an issue that could be characterized as talent squeeze. Because Scala is notoriously tough to onboard beginners onto compared to Python or Go, fewer junior devs are picking it up, and the hiring pool has gotten very senior-heavy and expensive. On the flip side, teams that need pieces of a resilient concurrent system - say, for serving ML models or running streaming pipelines - often don't want to buy into the whole JVM/Scala stack just to get decent concurrency and type safety.

That brings us to fp_mojo. The idea is to take the most appealing parts of Scala/Cats ("best practices") and make them available in a compiled, Python-like language (Mojo). Obviously it's nowhere near as mature or feature-complete as the Scala ecosystem, but the hope is that over time it would be vastly easier to onboard engineers who already know Python and incorporate Scala-like features selectively into broader Python-like projects.

I’m aware that Mojo is a very recent language, so the talent pool at the moment is even smaller. However, I want to talk about the future: engineers who already know Python can pick up Mojo much faster than Scala.

1. Work-stealing pool + deterministic single-thread IO (the Cats Effect model)

For me, the single best part of Scala/Cats Effect is the IO runtime model, so that's where we will begin the exposition.

fp.effects.async_runtime gives you cancelable fibers that can run on either a multi-threaded work-stealing pool (Profile.work_stealing(n)) or a deterministic single-thread FIFO executor (Profile.current_thread()). Collections can be processed concurrently with a concurrency limit via par_traverse_n, where the first failure cancels the rest. Blocking syscalls can be wrapped in IO.blocking(...), or a whole region can be shifted onto a dedicated BlockingPool via .eval_on(), so blocking work doesn't tie up the worker threads, and you can test timeouts deterministically in virtual time with TestControl instead of sleeping in tests:

from fp.algebra import par_traverse_n
from fp.effects.async_runtime import (
    IO, BlockingPool, Duration, IORuntimeConfig, Profile, Runtime, TestControl
)

def parse_rows(var partition: Int) -> IO[Int, Int]:
    if partition == 4:
        print("corrupt partition:", partition)
        return IO[Int, Int].raise_error(partition)
    return IO[Int, Int].pure(partition * 1000)         # rows read

def load_partition(var partition: Int) -> IO[Int, Int]:
    # Simulated fetch: 20 ms of I/O per partition, giving up after 100 ms
    try:
        var fetch = IO[Int, Int].pure(partition).delay_by(Duration.milliseconds(20))
        return fetch^.flat_map(parse_rows).timeout_to(Duration.milliseconds(100), IO[Int, Int].raise_error(partition))
    except:  # Duration constructors raise only on overflow
        return IO[Int, Int].raise_error(partition)

def show(var value: Int) -> NoneType:
    print("value:", value)
    return None

def show_rows(var rows: List[Int]) -> NoneType:
    print("rows:", rows[0], rows[1], rows[2])
    return None

def main() raises:
    # Switch between multi-threaded work-stealing or deterministic single-thread IO
    var config = IORuntimeConfig()
    config.profile = Profile.work_stealing(4)   # or Profile.current_thread()
    var compute = Runtime(config)
    var blocking_io = BlockingPool(max_workers=2)

    # Load partitions concurrently, at most 2 at a time; results keep partition order
    _ = compute.run(par_traverse_n(2, List[Int]([1, 2, 3]), load_partition).map(show_rows))
    # rows: 1000 2000 3000

    # The first failure cancels the partitions still running and waits for their cleanup
    var loaded = compute.run(par_traverse_n(2, List[Int]([1, 4, 5, 6]), load_partition)).value().is_ok()
    print("all loaded:", loaded)
    # corrupt partition: 4
    # all loaded: False

    # Shift work to a blocking pool and return, just like Cats Effect's evalOn
    _ = compute.run(IO[Int, Int].pure(100).eval_on(blocking_io).map(show))
    # value: 100

    # Virtual time testing: a 60s sleep with a 5s timeout finishes instantaneously
    var timed = (
        IO[NoneType, Int].sleep(Duration.seconds(60))
        .map(lambda (u: NoneType) -> Int: 42)
        .timeout_to(Duration.seconds(5), IO[Int, Int].pure(7))
        .map(show)
    )
    var control = TestControl(timed^)
    control.tick_all() # Advances virtual clock through deadlines without wall-clock waiting
    _ = control^.results()
    # value: 7

2. Method chaining & accumulating validation errors (par_map_n)

You get standard Result[T, E] (a primitive like Scala's Either[E, T] but with Rust's naming; personally, I like Result naming more because it's more descriptive) chaining with .map() / .flat_map(), plus pipeline and par_map_n when you want to validate a record and collect all errors instead of short-circuiting on the first bad field:

from fp.data import Result, Ok, Err
from fp.functions import pipeline

comptime Checked = Result[Int, List[String]]

def positive(value: Int) -> Checked:
    if value <= 0:
        return Err(List[String]([String(value) + " must be positive"]))
    return Ok(value)

def total(a: Int, b: Int, c: Int) -> Int:
    return a + b + c

def main():
    # Standard chaining
    var res = positive(21).map(lambda (x: Int) -> Int: x * 2)
    print("chained:", res^.get_or(0)) # chained: 42

    # Parallel validation: reports both negative inputs instead of stopping at -2
    var invalid = pipeline(positive(-2)).par_map_n((positive(3), positive(-4)), total).get()
    if invalid.is_err():
        var errors = invalid^.swap().to_optional()
        for error in errors.value():
            print("validation error:", error)
        # validation error: -2 must be positive
        # validation error: -4 must be positive

3. ADTs, compile-time exhaustive pattern matching & term rewriting

If you like modeling state or expression trees with sum types and writing rewrite rules (similar to how Spark Catalyst works under the hood), fp.adt and fp.matching check exhaustiveness at compile time:

import fp
from fp.adt import Data, Cases, Node, Value
from fp.matching import Next

@fieldwise_init
struct Num(Copyable, Equatable): var value: Int

@fieldwise_init
struct Add[R: Value](Movable):
    var left: Self.R
    var right: Self.R

@fieldwise_init
struct If[C: Value, B: Value](Movable):
    var cond: Self.C
    var then: Self.B
    var other: Self.B

struct Expr(Data):
    comptime Layer[R: Value] = Cases[Num, Add[R], If[R, R]]

comptime E = Node[Expr]

# Missing a branch here is a compile-time error
def evaluate(e: E) -> Int:
    return fp.match(e,
        lambda (n: Num) -> Int: n.value,
        lambda (a: Add[Int]) -> Int: a.left + a.right,
        lambda (c: If[Int, E]) -> Next[E]: Next(c.then if c.cond != 0 else c.other))

# Rule-based term rewriting: 0 + x -> x
def simplify(e: E) -> E:
    return fp.rewrite(e,
        fp.when[lambda (a: Add[E]) -> Bool: a.left == Num(0), lambda (a: Add[E]) -> E: a.right])

Curious to hear from folks here:

  1. Would you be interested in Scala/Cats-like IO and concurrency primitives in a Python-like language?

  2. If you've moved away from Scala, what parts of the language or ecosystem do you actually miss in day-to-day data engineering?

Repo link if you want to poke around: https://github.com/Spellbound-Mojo/fp_mojo


r/mlops • • 12d ago

Self-promotion How a small, decoupled LLM gateway compares to the large ones: only 31 MB of memory

4 Upvotes

I ran our open-source Rust gateway against Bifrost, Portkey and LiteLLM under the same load, then checked how much memory each one was holding afterwards. Ours was at 31 MB, while Bifrost was at 488 MB, LiteLLM at 418 MB and Portkey at 350 MB.

It was also faster, though this matters less than it looks. Ours adds about 0.07 ms per request, compared with 0.16 ms for Bifrost, 0.17 ms for Portkey and 4.34 ms for LiteLLM. These runs used a local mock instead of a real provider, so they only show the gateway’s own overhead, and on a real LLM call you wouldn’t notice most of that gap. Throughput at 50 concurrent requests was 56,734 per second for ours and 18,182 for Bifrost. I wouldn’t take the 56k too literally, because hitting the mock with no gateway in front only gave 46k to 80k per second, which means we’re close to the limit of the test setup.

Part of the memory difference comes from keeping the core small. The core crate has five dependencies and no vendor code, and routing, pricing, credentials and the tool loop are all plugins that the binary adds one by one. The tradeoff is that adding a new plugin means rebuilding the gateway.

I ran everything on an M4 Max. LiteLLM ran as a single Python worker, so more workers would give it more throughput but also use more memory. (actually idk how numbers of workers would be right for comparison)

source: https://github.com/respanai/respan-gateway