r/mlops • • May 04 '25

ML is just software engineering on hard mode.

328 Upvotes

You ever build something so over-engineered it loops back around and becomes justified?

Started with: “Let’s train a model.”

Now I’ve got:

  • A GPU-aware workload scheduler

  • Dynamic Helm deployments through a FastAPI coordinator

  • Kafka-backed event dispatch

  • Per-entity RBAC scoped across isolated projects

  • A secure proxy system that even my own services need permission to talk through

Somewhere along the way, the model became the least complicated part.


r/mlops • • Feb 19 '25

MLOps Education 7 MLOPs Projects for Beginners

174 Upvotes

MLOps (machine learning operations) has become essential for data scientists, machine learning engineers, and software developers who want to streamline machine learning workflows and deploy models effectively. It goes beyond simply integrating tools; it involves managing systems, automating processes tailored to your budget and use case, and ensuring reliability in production. While becoming a professional MLOps engineer requires mastering many concepts, starting with small, simple, and practical projects is a great way to build foundational skills.

In this blog, we will review a beginner-friendly MLOps project that teaches you about machine learning orchestration, CI/CD using GitHub Actions, Docker, Kubernetes, Terraform, cloud services, and building an end-to-end ML pipeline.

Link: https://www.kdnuggets.com/7-mlops-projects-beginners


r/mlops • • May 07 '26

r/mlops has been re-opened

133 Upvotes

r/mlops is open again. Yep, you read that right!

The old mods were inactive and the community entered a restricted mode. There was a huge amount of spam piling up. I'm going to clean it up and see if we can streamline the experience.

For those of you who stumbled upon this place by curiosity:

This community is for practical discussions around ML in production: infrastructure, deployment, serving, evaluation, monitoring, tooling, platforms, reliability, data pipelines, machine learning, orchestration, LLMOps, platform engineering, and real-world operational lessons.

What’s welcome:

  • Technical discussions and architecture deep-dives

  • Open-source tools and projects

    • But do not spam your project or try to get free market research about your project!

  • Case studies and postmortems

  • Research with clear operational relevance

  • Tutorials, benchmarks, and implementation details

What’s not:

  • Low-effort self-promotion

  • Generic AI hype/content farming/AI-generated posts

  • “What AI startup should I build?” posts

  • Hiring posts. Check out some of the communities online for this.

  • Affiliate spam, SEO dumps, or engagement bait

If you’re building, operating, or scaling ML systems, you’re in the right place.

Enjoy, but don't wreck the place!

u/MyBossIsOnReddit


r/mlops • • Mar 08 '26

Tales From the Trenches "MLOps is just DevOps with ML tools" — what I thought before vs what it actually looks like

122 Upvotes

When I started looking at MLOps from a DevOps background, my mental model was completely off. Sharing some assumptions I had vs what the reality turned out to be. Not to scare anyone off, just wish someone had been straight with me earlier.

What I thought: MLOps is basically CI/CD but for models. Learn MLflow, Kubeflow, maybe Airflow. Done.

Reality: The pipeline part is easy. The hard part is understanding why something failed. A CI/CD failure gives you a stack trace. A training pipeline failure gives you a loss curve that just looks off. You need enough ML context to even know what "off" means.

What I thought: Models are like microservices. Deploy, scale, monitor. Same playbook.

Reality: A microservice either works or it doesn't. Returns 200 or 500. A model can return a 200, perfectly formatted response, or a completely wrong answer. Nobody gets paged. Nobody even notices until business metrics drop a week later. That messed with my head because in DevOps, if something breaks, you know.

What I thought: GPU scheduling is just resource management. I do this all day with CPU and memory.

Reality: GPUs don't share the way CPUs do. One pod gets the whole GPU or nothing. And K8s doesn't even know what a GPU is until you install NVIDIA's device plugin and GPU operator. Every scheduling decision matters because each GPU costs 10 to 50x that of a CPU node.

What I thought: My Python is fine. I write automation scripts all the time.

Reality: First time I opened a real training script, it looked nothing like the Python I was writing. Decorators everywhere, generators, async patterns, memory-sensitive code. Scripting and actual programming turned out to be genuinely different things. That one humbled me.

What I thought: I'll learn ML theory later, just let me handle the infra.

Reality: You can actually go pretty far on the inference and serving side without deep ML theory. That part was true. But you still need enough to have a conversation. When a data scientist says "we need to quantise to INT8," you don't need to derive the math, but you need to know what that means for your infra.

What I thought: They just want someone who can manage Kubernetes and set up pipelines.

Reality: They want someone who can sit between infra and ML. Someone who can debug a memory leak inside the inference service, not just restart the pod. Someone who looks at GPU utilisation and knows whether that number means healthy or on fire. The "Ops" in MLOps goes deeper than I expected.

None of this is to discourage anyone. The transition is very doable, especially if you go in with the right expectations. But "just learn the tools" is bad advice. The tools are the surface.

I've been writing about this transition and talking to a bunch of people going through it. If you're in this spot and want to talk through what to focus on, DMs open or grab time here: topmate.io/varun_rajput_1914


r/mlops • • May 10 '22

Difference between MLOps and Data Science?

Post image
121 Upvotes

r/mlops • • Jan 21 '26 Bravo

Coming from DevOps/Infra to MLOps? Here's what I learned after several interviews at product companies

108 Upvotes

I've been interviewing for MLOps and ML Platform Engineer roles over the past few months, and I wanted to share some observations that might help others make a similar transition.

The Interview Gap

Most interviewers I've faced come from research or pure ML engineering backgrounds. They think in terms of model architectures, feature engineering, and training pipelines. If you're coming from a pure infrastructure or DevOps background like me, there's often a disconnect.

You talk about Kubernetes orchestration, GPU cluster management, and cost optimisation. They ask about data drift, model retraining strategies, or how you'd debug a model's performance degradation. The conversation doesn't flow naturally because you're speaking different languages.

What Actually Helped

I realised I needed to invest time in ML fundamentals – not to become a data scientist, but to bridge the communication gap. Understanding basic statistics, how different model types work, and what "overfitting" or "data leakage" actually mean made a huge difference.

When I could frame infrastructure decisions in ML terms ("this architecture reduces model serving latency by X%" vs "this setup has better resource utilisation"), interviews went much more smoothly.

Be Strategic About Target Companies

Not all MLOps roles are the same. If you're targeting companies heavily invested in real-time inferencing (think fraud detection, recommendation engines, autonomous systems), the focus shifts to:

  • Data distribution and streaming pipelines

  • Low-latency prediction infrastructure

  • Real-time monitoring and anomaly detection

  • Data engineering skills

If they're doing batch processing and research-heavy ML, it's more about:

  • Experiment tracking and reproducibility

  • Training infrastructure and GPU optimization

  • Model versioning and registry management

Match your preparation to what they actually care about. Don't spray-and-pray applications.

MLOps Roles Vary Wildly

Here's something that actually helped my perspective: MLOps means different things at different companies.

I've had interviews where the focus was 90% infrastructure (Kubernetes, CI/CD, monitoring). Others were 70% ML-focused (understanding model drift, feature stores, retraining strategies). Some wanted a hybrid who could do both.

This isn't because teams don't know what they want. It's because MLOps is genuinely different depending on:

  • Company maturity (startup vs established)

  • ML use cases (batch vs real-time)

  • Team structure (centralised platform vs embedded engineers)

If an interview feels misaligned, it's often a mismatch in role expectations, not a reflection of your skills. The "MLOps Engineer" title can mean vastly different things across companies.

Practical Tips

  • Learn the basics: bias-variance tradeoff, cross-validation, common model types

  • Understand the ML lifecycle beyond just deployment

  • Be able to discuss model monitoring (not just infra monitoring)

  • Know the tools: MLflow, Kubeflow, Ray, etc. – but more importantly, know why they exist

  • Read ML papers occasionally – even if you don't implement them, you'll understand what your ML colleagues are dealing with

Final Thought

The transition from DevOps to MLOps isn't just about learning new tools. It's about understanding a new domain and the people working in it. Meet them halfway, and you'll find the conversations get a lot easier.

Keep learning, keep iterating.

If anyone's going through a similar transition and wants to chat, feel free to DM or connect here: https://topmate.io/varun_rajput_1914/


r/mlops • • Nov 05 '25

idle gpus are bleeding money, did the math on our h100 cluster and it's worse than I thought

97 Upvotes

Just finished a cost analysis of our gpu infrastructure and the numbers are brutal. We're burning roughly $45k/month on gpus that sit idle 40% of the time.

Our setup: 16x h100 on aws (p5.48xlarge instances). Cost per hour is $98.32, monthly running 24/7 comes to ~$71k, but at 60% utilization we're effectively paying $118/hour per useful hour. That's ~$28k/month wasted doing literally nothing.

For on-prem it's worse because you can't shut them off. Those h100s draw 700w each, at $0.12/kwh that's $1,176/month per gpu just in power. Unused.

Checked our job logs to see why utilization sucks. Jobs queued waiting for specific gpu counts (want 8, only 6 available), researchers holding gpus "just in case" for next experiment, data loading bottlenecks where gpus idle while waiting for data, failed jobs that didn't release resources, weekends and nights with no jobs scheduled.

Tried kubernetes autoscaling... configuration hell and slow scale-up meant jobs waited anyway. Tried stricter quotas but team complained about blocked research. Time-based scheduling (everyone gets X hours/week) created artificial scarcity, people just ran junk jobs to use their allocation.

I ended up switching to dynamic orchestration with transformer lab that utomatically routes jobs to lowest-cost available gpus across on-prem + cloud, if local cluster full it bursts to spot instances automatically. Went from 60% to 85% average utilization, that's $19k/month saved just from better job placement.

Also started auto-killing jobs after 24hr if no checkpoint progress, added monitoring dashboard showing cost per experiment, implemented shared job queue with fair-share scheduling, automatic scale-down of cloud resources.

This isn't just money either. Idle gpus still draw near-full power, we were producing ~15 tons of co2/month from unused compute. Our university has climate goals and this wasn't helping.

Measure first - instrument your cluster. Job placement matters more than autoscaling. Make cost visible to researchers (not to guilt just awareness), remove artificial barriers to resource sharing, use spot instances aggressively for non-critical work.

Anyone else track these metrics? What's your effective utilization?


r/mlops • • Jul 01 '26

Tales From the Trenches Interesting shift in “Platform Engineering / MLOps” interviews — lots of Kubernetes operations, very little ML

93 Upvotes

I’ve been interviewing for several Staff/Principal Platform Engineering and MLOps roles around Silicon Valley recently, and I’ve noticed an interesting pattern. Curious if others are seeing the same thing.

But once the technical interview starts, the discussion quickly narrows into Kubernetes operations.

Typical probing topics include:

Kubernetes
Production support and debugging
little or no time on discussing ML

Instead, many interviews feel like they’re looking for someone with production Kubernetes clusters experince.
One hiring manager described the role as “Platform Engineering,” but nearly every technical question centered around daily Kubernetes operations, CI/CD mechanics, production troubleshooting, and infrastructure automation.

My impression is that many companies are using “Platform,” “AI Platform,” or “MLOps” as umbrella titles for what is fundamentally senior Kubernetes platform operations.

Curious what others are seeing.

Questions for the community:
- Are “Platform Engineering” and “MLOps” titles increasingly becoming Kubernetes operations roles?
- How much architecture discussion do you typically see in Staff/Principal interviews?
- Are companies intentionally broadening titles to attract candidates, or has the definition of platform engineering genuinely shifted toward infrastructure operations?

what percentage of the interview is architecture versus deep operational troubleshooting?


r/mlops • • Apr 29 '22

meme Hi everyone, let's finish April with 4k subs! We're so close! Tell your colleagues about us! You can use this meme as a sure win argument.

Post image
89 Upvotes

r/mlops • • Sep 06 '25

Why is building ML pipelines still so painful in 2025? Looking for feedback on an idea.

86 Upvotes

Every time I try to go from idea → trained model → deployed API, I end up juggling half a dozen tools: MLflow for tracking, DVC for data, Kubeflow or Airflow for orchestration, Hugging Face for models, RunPod for training… it feels like duct tape, not a pipeline.
Kubeflow feels overkill, Flyte is powerful but has a steep curve, and MLflow + DVC don’t feel integrated. Even Prefect/Dagster are more about orchestration than the ML lifecycle.

I’ve been wondering: what if we had a LangFlow-style visual interface for the entire ML lifecycle - data cleaning (even with LLM prompts), training/fine-tuning, versioning, inference, optimization, visualization, and API serving.
Bonus: small stuff on Hugging Face (cheap + community), big jobs on RunPod (scalable infra). Centralized HF Hub for versioning/exposure.

Do you think something like this would actually be useful? Or is this just reinventing MLflow/Kubeflow with prettier UI? Curious if others feel the same pain or if I’m just overcomplicating my stack.

If you had a magic wand for ML pipelines, what would you fix first - data cleaning, orchestration, or deployment?


r/mlops • • Jun 11 '25

MLOps Education Fully automate your LLM training-process tutorial

Thumbnail
towardsdatascience.com
86 Upvotes

I’ve been having fun training large language models and wanted to automate the process. So I picked a few open-source cloud-native tools and built a pipeline.

Cherry on the cake? No need for writing Dockerfiles.

The tutorial shows a really simple example with GPT-2, the article is meant to show the high level concepts.

I how you like it!


r/mlops • • Jul 01 '23

beginner help😓 Where do I start to learn MLOPS?

86 Upvotes

I have basic knowledge of Python & ML, that is, I know scikit- learn but not any deep learning libraries. I don’t have any knowledge of cloud either.

Would learning a cloud platform be the best place to start?

How would you recommend starting off & what do you recommend as a pathway for learning?

Also, are there any resources or courses to learn MLOPS?


r/mlops • • Aug 11 '24

What's your Mlops stack

77 Upvotes

I'm an experienced software engineer but I have only dabbled in mlops.

There are do many tools in this space with a decent amount of overlap. What combination of tools do you use in your company? I'm looking for specific brands here so I can do some research / learning ..


r/mlops • • Mar 19 '25

MLOps Education MLOps tips I gathered recently

76 Upvotes

Hi all,

I've been experimenting with building and deploying ML and LLM projects for a while now, and honestly, it’s been a journey.

Training the models always felt more straightforward, but deploying them smoothly into production turned out to be a whole new beast.

I had a really good conversation with Dean Pleban (CEO @ DAGsHub), who shared some great practical insights based on his own experience helping teams go from experiments to real-world production.

Sharing here what he shared with me, and what I experienced myself -

  1. Data matters way more than I thought. Initially, I focused a lot on model architectures and less on the quality of my data pipelines. Production performance heavily depends on robust data handling—things like proper data versioning, monitoring, and governance can save you a lot of headaches. This becomes way more important when your toy-project becomes a collaborative project with others.

  2. LLMs need their own rules. Working with large language models introduced challenges I wasn't fully prepared for—like hallucinations, biases, and the resource demands. Dean suggested frameworks like RAES (Robustness, Alignment, Efficiency, Safety) to help tackle these issues, and it’s something I’m actively trying out now. He also mentioned "LLM as a judge" which seems to be a concept that is getting a lot of attention recently.

Some practical tips Dean shared with me:

  • Save chain of thought output (the output text in reasoning models) - you never know when you might need it. This sometimes require using the verbos parameter.

  • Log experiments thoroughly (parameters, hyper-parameters, models used, data-versioning...).

  • Start with a Jupyter notebook, but move to production-grade tooling (all tools mentioned in the guide bellow 👇🏻)

To help myself (and hopefully others) visualize and internalize these lessons, I created an interactive guide that breaks down how successful ML/LLM projects are structured. If you're curious, you can explore it here:

https://www.readyforagents.com/resources/llm-projects-structure

I'd genuinely appreciate hearing about your experiences too—what’s your favorite MLOps tools?
I think that up until today dataset versioning and especially versioning LLM experiments (data, model, prompt, parameters..) is still not really fully solved.


r/mlops • • Jun 26 '25

I built a self-hosted Databricks

75 Upvotes

Hey everyone, I'm an ML Engineer who spearheaded the adoption of Databricks at work. I love the agency it affords me because I can own projects end-to-end and do everything in one place.

However, I am sick of the infra overhead and bells and whistles. Now, I am not in a massive org, but there aren't actually that many massive orgs... So many problems can be solved with a simple data pipeline and basic model (e.g. XGBoost.) Not only is there technical overhead, but systems and process overhead; bureaucracy and red-tap significantly slow delivery.

Anyway, I decided to try and address this myself by developing FlintML. Basically, Polars, Delta Lake, unified catalog, Aim experiment tracking, notebook IDE and orchestration (still working on this) fully spun up with Docker Compose.

I'm hoping to get some feedback from this subreddit. I've spent a couple of months developing this and want to know whether I would be wasting time by continuing or if this might actually be useful.

Thanks heaps


r/mlops • • Nov 24 '25

Is anyone else noticing that a lot of companies claiming to “do MLOps” are basically faking it?

73 Upvotes

I keep seeing teams brag about “robust MLOps pipelines,” and then you look inside and it’s literally:
• a notebook rerun weekly
• a cron job
• a bucket of CSVs,
• a random Grafana chart,
• a folder named model_final_FINAL_v3,
• and zero monitoring, versioning, or reproducibility.

Meanwhile actual mlops problems like data drift, feature pipelines breaking, infra issues, scaling, governance, model degradation in prod, etc never get addressed because everyone is too busy pretending things are automated.

It feels like flashy diagrams and LinkedIn posts have replaced real pipelines.

So I’m curious: what percentage of companies do you think actually have mature, reliable MLOps?
5%? 10%? Maybe 20%? And what’s the real blocker? Lack of talent, messy org structure, infra complexity, or just no one wanting to do the unglamorous parts?

Gimme your honest takes


r/mlops • • Feb 19 '26

MLOps Education Friendly advice for infra engineers moving to MLOps: your Python scripting may not enough, here's the gap to close

72 Upvotes

In my last post, I covered ML foundations. This one's about Python, specifically, the gap between "I know Python" and the Python you actually need for MLOps.

If you're from infra/DevOps, your Python probably looks like mine did: boto3 scripts, automation glue, maybe some Ansible helpers. That's scripting. MLOps needs programming, and the difference matters.

What you're probably missing:

  • Decorators & closures — ML frameworks live on these. Airflow's `@tasks`, FastAPI's `@app.get()`. If you can't write a custom decorator, you'll struggle to read any ML codebase.

  • Generators — You can't load 10M records into memory. Generators let you stream data lazily. Every ML pipeline uses this.

  • Context managers — GPU contexts, model loading/unloading, DB connections. The with Pattern is everywhere.

Why memory management suddenly matters:

In infra, your script runs for 5 seconds and exits. In ML, you're loading multi-GB models into servers that run for weeks. You need to understand Python's garbage collector, the difference between a Python list and a NumPy array, and the GPU memory lifecycle.

Async isn't optional:

FastAPI is async-first. Inference backends require you to understand when to use asyncio, multiprocessing, or threading, and why it matters for ML workloads.

Best way to learn all this? Don't read a textbook. Build an inference backend from scratch, load a Hugging Face model, wrap it in FastAPI, add batching, profile memory under load, and make it handle 10K requests. Each step targets the exact Python skills you're missing.

The uncomfortable truth: you can orchestrate everything with K8s and Helm, but the moment something breaks inside the inference service, you're staring at Python you can't debug. That's the gap. Close it.

If anyone interested in detailed version, with an atual scenarios covering WHYs and code snippets please refer: https://medium.com/@thevarunfreelance/friendly-advice-for-infra-engineers-moving-to-mlops-your-python-scripting-isnt-enough-here-s-f2f82439c519

I've also helped a few folks navigate this transition, review their resumes, prepare for interviews, and figure out what to focus on. If you're going through something similar and want to chat, my DMs are open, or you can book some time here: topmate.io/varun_rajput_1914


r/mlops • • Feb 04 '26

MLOps Education The weird mismatch in MLOps hiring that nobody talks about

72 Upvotes

Something I've noticed after being in this space for a while, and mentioned in past weeks' posts as well.

MLOps roles need strong infrastructure skills. Everyone agrees on that. The job descriptions are full of Kubernetes, CI/CD, cloud, distributed systems, monitoring, etc.

But the people interviewing you? Mostly data scientists, ML engineers, and PhD researchers.

So you end up in a strange situation where the job requires you to be good at production engineering, but the interview asks you to speak ML. And these are two very different conversations.

I've seen really solid DevOps engineers, people running massive clusters, handling serious scale, get passed over because they couldn't explain what model drift is or why you'd choose one evaluation metric over another. Not because they couldn't learn it, but because they didn't realise that's what the interview would test.

And on the flip side, I've seen ML folks get hired into MLOps roles and MAY struggle because they've never dealt with real production systems at scale.

The root cause I think is that most companies are still early in their ML maturity. They haven't separated MLOps as its own discipline yet. The ML team owns hiring for it, so naturally, they filter for what they understand: ML knowledge, not infra expertise.

This isn't a complaint, just an observation. And practically speaking, if you're coming from the infra/DevOps side, it means you kinda have to meet them where they are. Learn enough ML to hold the conversation. You don't need to derive backpropagation on a whiteboard, but you should be able to talk about the model lifecycle, failure modes, why monitoring ML systems is different from monitoring regular services, etc.

The good news is the bar isn't that high. A few weeks of genuine study go a long way. And once you bridge that language gap, your infrastructure background becomes a massive advantage, because most ML teams are honestly struggling with production engineering.

Curious if others have experienced this same thing? Either as candidates or on the hiring side?

I've also helped a few folks navigate this transition, review their resumes, prepare for interviews, and figure out what to focus on. If you're going through something similar and want to chat, my DMs are open, or you can book some time here: topmate.io/varun_rajput_1914


r/mlops • • Jan 26 '26

Static model selection did not work (enough) for us

72 Upvotes

We spent a few months now on a solution for dynamic model routing because we tried several things and nothing really solved our problem.

The core issue / our background: we deployed nodes with SLM and RAG to regulated industry teams (the problem is relevant in any setup though). But users couldn't figure out when to use which model (despite ongoing effort to educate). We tried static routing but the classification of queries upfront didn't really work as it was very unpredictable what the users were doing. Also the "guessing" part did not feel right, we iterated really a lot on this. So next we thought hybrid with big models would be the solution but somewhat similar we always had to estimate complexity before we saw output. The estimates missed often enough that we either overspent (like, radically, breaking our unit economics) or quality was bad from routing too aggressively to small models.

We found a Google publication (happy to share) that approaches this very differently, not routing but cascading. Start generating with the small model, validate quality as you go, escalate only if needed.

We developed this and open-sourced our implementation: github.com/lemony-ai/cascadeflow

It plugs into your existing infrastructure, works with LiteLLM, OpenRouter, n8n, LangChain, or direct API calls. From there you can use whatever models you want: OpenAI, Anthropic, Groq, HuggingFace, local models via Ollama, self-hosted via vLLM.

Not replacing your router or orchestration layer, just adding quality validation that decides when the cheap models output is actually good enough.

Seeing 40-90% cost reduction in first production workloads and we are honestly quite excited. Would love feedback and happy to chat with others working on inference layers.


r/mlops • • Feb 28 '24

MLOps project showcase.

71 Upvotes

Hey everyone,

Just wrapped up a project where I built a system to predict rental prices using data from Rightmove. I really dived into Data Engineering, ML Engineering, and MLOps, all thanks to the free Data Talk Clubs courses I took. I am self taught in Data Engineering and ML in general (Finance graduate). I would really appreciate any constructive feedback on this project.

Quick features:

  • Production Web Scraping with monitoring

  • RandomForest Rental Prediction model with feature engineering. Engineered the walk score algorithm (based on what I could find online)

  • MLOps with model, data quality and data drift monitoring.

Tech Stack:

  • Infrastructure: Terraform, Docker Compose, AWS, and GCP.

  • Model serving with FastAPI and visual insights via Streamlit and Grafana.

  • Experiment tracking with MLFlow.

I tried to mesh everything I could from these courses together. I am not sure if I followed industry standards. Feel free to be as harsh and as honest as you like. All I care about is that the feedback is actionable. Thank you.

Github: https://github.com/alexandergirardet/london_rightmove


r/mlops • • Sep 12 '24

LLMOps fundamentals

Post image
70 Upvotes

I've working as a data scientist for 4 years now. In he companies I've worked, we have a engineering and mlops team, so I haven't worked about the deployment of the model.

Having said that, I honestly tried to avoid certain topics to study/work, and those topics are Cloud computing, Deep learning, MLOps and now GenAI/LLMS

Why? Idk, I just feel like those topics evolve so fast that most of the things you learn will be deprecating really soon. So, although it's working with some SOTA tech, for me it's a bit like wasting time

Now, I know some things will never change in the future, and that are the fundamentals

Could you tell me what topics will remain relevant in the future? (E.g. Monitoring, model drift, vector database, things like that)

Thanks in advance


r/mlops • • Nov 11 '21

I am Alessya Visnjic, co-founder and CEO of WhyLabs. I am here to talk about MLOps, AI Observability and our recent product announcements. Ask me anything!

69 Upvotes

Alessya here to talk about how MLOps is taking over the AI noosphere and how WhyLabs is participating in the movement through open source and free to use AI Observability tools.

Prior to my work at WhyLabs, I was CTO-in-Residence at the Allen Institute for AI, where I spent my time talking to ML practitioners and prototyping tools for explainability, fairness, error analysis and observability. Before that, I spent 9 years working at Amazon, deploying some of Amazon's first ML applications to production, carrying a pager to respond to failures, and building tools to make my pager days less painful. My current hobbies include chasing my 10month old baby daughter around the house, cooking dishes of international cuisine, and running (slowly) the trails of the Pacific Northwest.

The team at WhyLabs has been building ML and analytics tools across industry’s leading tech giants like Amazon, Microsoft, Lyft, etc. Last week, we announced the industry’s first SaaS AI Observability Platform – which enables teams to monitor, understand, and improve their AI applications. AI Observatory is also the first observability solution that AI builders can self-onboard and start using for free: https://whylabs.ai/blog/posts/ai-observatory.

AI observability is mission-critical for production ML applications. At WhyLabs we are removing barriers for the adoption of this essential technology. But technology alone is not enough. As a community, we need to help establish culture and process that organizations can adopt in order to implement MLOps and responsible usage of ML/AI. We are thrilled to be partnering with industry experts, like Andrew Ng’s AI Fund and the Allen Institute for AI, to help establish AI Observability as a ubiquitous part of every production ML stack.

I’m here to answer any questions you may have about our announcement, AI Observability or MLOps in general. Proof: https://imgur.com/a/5o3Zv61.


r/mlops • • Jun 25 '24

Tales From the Trenches Reflections on working with 100s of ML Platform teams

65 Upvotes

Having worked with numerous MLOps platform teams—those responsible for centrally standardizing internal ML functions within their companies—I have observed several common patterns in how MLOps adoption typically unfolds over time. Having seen Uber write about the evolution of their ML platform recently, it inspired me to write my thoughts on what I’ve seen out in the wild:

🧱 Throw-it-over-the-wall → Self-serve data science

Usually, teams start with one or two people who are good at the ops part, so they are tasked with deploying models individually. This often involves a lot of direct communication and knowledge transfer. This pattern often forms silos, and over time teams tend to break them and give more power to data scientists to own production. IMO, the earlier this is done, the better. But you’re going to need a central platform to enable this.

Tools you could use: ZenML, AWS Sagemaker, Google Vertex AI

📈 Manual experiments → Centralized tracking

This is perhaps the simplest possible step a data science team can take to 10x their productivity → Add an experiment tracking tool into the mix and you go from non-centralized, manual experiment tracking and logs to a central place where metrics and metadata live.

Tools you could use: MLflow, CometML, Neptune

🚝 Mono-repo → Shared internal library

It’s natural to start with one big repo and throw all data science-related code in it. However, as teams mature, they tend to abstract commonly used patterns into an internal (pip) library that is maintained by a central function and in another repo. Also, a repo per project or model can also be introduced at this point (see shared templates).

Tools you could use: Pip, Poetry

🪣 Manual merges → Automated CI/CD

I’ve often seen a CI pattern emerge quickly, even in smaller startups. However, a proper CI/CD system with integration tests and automated model deployments is still hard to reach for most people. This is usually the end state → However, writing a few GitHub workflows or Gitlab pipelines can get most teams starting very far in the process.

Tools you could use: GitHub, Gitlab, Circle CI

👉 Manually triggered scripts → Automated workflows

Bash scripts that are hastily thrown together to trigger a train.py are probably the starting point for most teams, but very quickly teams can outgrow these. It’s hard to maintain, intransparent, and flaky. A common pattern is to transition to ML pipelines, where steps are combined together to create workflows that are orchestrated locally or on the cloud.

Tools you could use: Airflow, ZenML, Kubeflow

🏠 Non-structured repos → Shared templates

The first repo tends to evolve organically and contains a whole bunch of stuff that will be pruned later. Ultimately, a shared pattern is introduced and a tool like cookie-cutter or copier can be used to distribute a single standard way of doing things. This makes onboarding new team members and projects way easier.

Tools you could use: Cookiecutter, Copier

🖲️ Non-reproducible artifacts → Lineage and provenance

At first, no artifacts are tracked in the ML processes, including the machine learning models. Then the models start getting tracked, along with experiments and metrics. This might be in the form of a model registry. The last step in this is to also track data artifacts alongside model artifacts, to see a complete lineage of how a ML model was developed.

Tools you could use: DVC, LakeFS, ZenML

💻 Unmonitored deployments → Advanced model & data monitoring

Models are notoriously hard to monitor - Whether its watching for spikes in the inputs or seeing deviations in the outputs. Therefore, detecting things like data and concept drift is usually the last puzzle piece to fall as teams mature into full MLOps maturity. If you’re automatically detecting drift and taking action, you are in the top 1% of ML teams.

Tools you could use: Evidently, Great Expectations

Have I missed something? Please share other common patterns, I think its useful to establish a baseline of this journey from various angles.

Disclaimer: This was originally a post on the ZenML blog but I thought it was useful to share here and was not sure whether posting a company affiliated link would break the rules. See original blog here: https://www.zenml.io/blog/reflections-on-working-with-100s-of-ml-platform-teams


r/mlops • • Nov 28 '24

Tools: OSS How we built our MLOps stack for fast, reproducible experiments and smooth deployments of NLP models

64 Upvotes

Hey folks,
I wanted to share a quick rundown of how our team at GitGuardian built an MLOps stack that works for production use cases (link to the full blog post below). As ML engineers, we all know how chaotic it can get juggling datasets, models, and cloud resources. We were facing a few common issues: tracking experiments, managing model versions, and dealing with inefficient cloud setups.
We decided to go open-source all the way. Here’s what we’re using to make everything click:

  • DVC for version control. It’s like Git, but for data and models. Super helpful for reproducibility—no more wondering how to recreate a training run.

  • GTO for model versioning. It’s basically a lightweight version tag manager, so we can easily keep track of the best performing models across different stages.

  • Streamlit is our go-to for experiment visualization. It integrates with DVC, and setting up interactive apps to compare models is a breeze. Saves us from writing a ton of custom dashboards.

  • SkyPilot handles cloud resources for us. No more manual EC2 setups. Just a few commands and we’re spinning up GPUs in the cloud, which saves a ton of time.

  • BentoML to build models in a docker image, to be used in a production Kubernetes cluster. It makes deployment super easy, and integrates well with our versioning system, so we can quickly swap models when needed.

On the production side, we’re using ONNX Runtime for low-latency inference and Kubernetes to scale resources. We’ve got Prometheus and Grafana for monitoring everything in real time.

Link to the article : https://blog.gitguardian.com/open-source-mlops-stack/

And the Medium article

Please let me know what you think, and share what you are doing as well :)