I just found out that someone has been using my locally hosted AI model for over a month, without me knowing.
Apparently, I left the Ollama port open on my router, and someone found it. They’ve been sending it huge chunks of personal information — names, phone numbers, addresses, parcel IDs, job details, even latitude and longitude. All of it was being processed through my setup while I had no clue.
I only noticed today when I was checking some logs and saw a flood of suspicious-looking entries. When I dug into it, I found that it wasn’t just some one-off request — this had been going on for weeks.
The kind of data they were processing is creepy as hell. It looks like they were trying to organize or extract information on people. I’m attaching a screenshot of one snippet — it speaks for itself.
The IP was from Hong Kong and the prompt is at the end in Chinese.
I’ve shut it all down now and locked things up tight. Just posting this as a warning.
I have an RTX 4080 with 16GB memory, and I tried running Gemma 4.31b on it using Ollama, and I'm shocked to see that even a simple 'Hi' message takes 4-6 seconds to respond to, and when I send more context, it takes very much more time and sometimes gets cancelled/killed.
After looking at it... how much are Claude/Gemini/GPT spending on GPUs? Models like Opus are way too crazy, as they are able to read and process ~500 lines of code minimum at any given point.
Hey all. Just set up a workstation with two NVIDIA RTX PRO 6000 Blackwells (96GB VRAM each) for our design studio. Want to use Ollama as our main local inference layer.
What we want to do with it:
Internal copilot for a ~60 person team. research, writing, brief analysis, code assist
Backend for agentic tools we're building (API access is a big reason we picked Ollama)
Run the biggest, best models our hardware can handle
Specific questions:
How well does Ollama handle dual GPU setups out of the box? Any config needed for tensor parallelism across both cards?
What models would you recommend at this VRAM level? Thinking Llama 3.1 70B unquantized, maybe even 405B at Q4?
Anyone serving Ollama to a team via Open WebUI or similar? How's the experience at 10-15 concurrent users?
Any gotchas with large model loading times or memory management I should know about?
First time running Ollama beyond hobby experiments, so any production-ish tips are appreciated. Will report back with what works.
------
UPDATE FOR OTHERS & THANKS FOR THE HELP . THIS SUB WASN'T AS SNARKY AND IN FACT A LOT MORE HELPFUL THAN THE OTHER ONE.
For context: we're a design agency rendering 3D animations, VR/AR walkthroughs, and architectural visualizations. Not generating AI images or running Stable Diffusion farms. The dual RTX Pro 6000s (96 GB VRAM each) are a dedicated render node that processes overnight animation batches and path-traced scenes while our design team stays productive on their own workstations. Cloud rendering costs add up absurdly fast at our project volume. Owning the hardware pays for itself in months. OctaneRender and Redshift scale linearly across both GPUs, which turns 12+ hour VR renders into something we can actually deliver on client deadlines.
Key Technical Advice & Actionables
Infrastructure Stack (Overwhelming Consensus)
Switch from Ollama to vLLM or llama.cpp
169 upvotes on "Tip #1 don't use Ollama"
109 upvotes on criticism of using Ollama with $25k hardware
vLLM is the top recommendation for multi-user concurrency (your 10-15 concurrent users scenario)
llama.cpp is acceptable for single-user or simpler setups, but vLLM wins for parallelization
Use Linux instead of Windows
266 upvotes on "Tip #2 use Linux"
Ubuntu LTS 24.04 most recommended for NVIDIA driver support
Debian headless for maximum resource efficiency
Debate exists: some claim Windows CUDA drivers are 2-3% faster for pure VRAM inference, but Linux wins for stability and virtual memory handling
Model Recommendations
Stop using Llama 3.1 70B (described as "ancient" and "severely outdated")
Minimax M2.7 (230B MoE, 10B active) with NVFP4 quantization — perfect fit for your dual 96GB setup
Qwen 3.5/3.6 series (27B, 35B MoE, 122B) — excellent dense models, great for agentic tasks
Gemma 4 — recommended if you need "western" models (some companies ban Chinese models)
Mistral Medium 3.5 (119B MoE) or new Mistral 128B dense — good for massive context windows
Critical Configuration Settings
Use Tensor Parallelism (tp=2)
Splits model across both GPUs for unified inference
Doubles speed and allows models up to ~180-190GB total
Essential command: --tp 2 in vLLM or llama.cpp
Use NVFP4 Quantization
Hardware-accelerated 4-bit format specifically for Blackwell architecture
Minimax M2.7 NVFP4 fits in 130.6GB (down from 230GB)
Multiple users emphasized this is purpose-built for your cards
Optimize for Concurrency
Use litellm as a model router in front of vLLM for rate limiting and monitoring
Set --gpu-memory-utilization 0.9 or higher to maximize KV cache
SGLang recommended over vLLM if team works on same projects (prefix caching with RadixAttention)
For 60-person team: expect 5-8 simultaneous users per card on 70B Q4 before throughput drops
System Architecture
Cooling & Power Management
GPU spacing: minimum 2 slots apart for adequate airflow
Consider power limiting cards to reduce heat and increase stability
Script fixed clock times (10MHz below stock) to prevent PCIe bus spikes
Heat management is critical for sustained inference loads
RAM Requirements
Minimum 256GB system RAM
Recommendation: 2× VRAM = 384-512GB system RAM for optimal performance
Essential for virtual memory handling during large context operations
Frontend & User Access
Open WebUI is acceptable for team deployment (contrary to one dismissive comment)
Alternative: Set up litellm for monitoring, rate limiting, API key generation
Some debate about OpenWebUI in 2026, but no clear superior alternative mentioned for your use case
llama-swap tool for dynamic model switching without container restarts
Hiring & Operational Advice
Top upvoted wisdom (113+ votes on original thread you referenced): "Storage, model management, permissions, and user access become more important than the GPUs after week one. Hire someone experienced with this stack."
I have spent about $10,000 on Ollama credits and have developed over six projects which use its API.
When I got in touch with support to complain about the interruptions to the API, my account was put on restriction. I am not able to prove that the complaint caused the restriction, but the fact that it happened at that time is highly questionable.
Ollama didn't explain the restriction; instead it refunded the balance I still had to the original payment method, with a processing time of 5 to 10 business days.
I never asked for a refund since I need the API access that I paid for.
I put my funds set aside to keep my business going. At the moment my projects are in danger and the time I have to wait—up to 10 business days—for money which I had never asked to be refunded could seriously affect my operations.
I wrote to the support team to find out why my account had been restricted, what is meant by 'extra usage balance', and whether the restriction was connected to my complaint. I have not yet received a substantial reply.
I have put a great deal of money into your platform, so I am entitled to a clear explanation and a genuine solution, not an arbitrary restriction and an unwanted refund.
Have any other people had something similar when using Ollama?
I have just tried GPT-OSS:20b on my machine. This is the stupidest COT MOE model I have ever interacted with. Open AI chose to shit on the open-source community by releasing this abomination of a model.
Cannot perform basic arithmetic reasoning tasks, Thinks too much, and thinking traits remind me of deepseek-distill:70b, Would have been a great model 3 generations ago. As of today there are a ton of better models out there GLM is a far better alternative. Do not even try this model, Pure shit spray dried into fine powder.
I've been experimenting with local AI models, even "high-end ones" like the recent DeepSeek-R1 32B, using Open WebUI. I expected them to be weaker than online models, but the gap is just ridiculous. Even for the simplest questions, they either fail, give nonsense answers, or completely misunderstand the input.
I’ve set the parameters and all the settings at the best i could, tried different setups, system prompts, and still , even after parsing a basic document just a few pages long, is a struggle. If it already fails here, how am I supposed to use it for hundreds of internal company documents?
The crazy part? No one talks about this! Instead, i see every video in youtube saying : "How to run locally (modelname) much better than chat-gpt" "Local Deepseek beats Chat-gpt" Than the question they ask to those local models are : How many 'R' are in the word Strawberry and the model answer: 2 ... lol
Why is the performance so bad, even on 32B models?
Why are there no proper guides to get the best out of local AI? Having a big hardware such as the Nvidia project DIGITS will make a big model work close to the online Chat-gpt 3 or 4 ? I see those has 175b parameters.
What are we missing?
I really want to make local AI work as close as the online models, even buying bigger and stronger hardware, but, right now, it just feels like a waste of time. Has anyone actually succeeded in making these models work well? If so, how? And , what do you intend for Working Well for a local Model ?
Come on guys, we could get deepseek cheaper with more usage directly with them if we wanted. WHERE IS KIMI K3 WITHOUT EXTRA USAGE NEEDED !!? This is pissing me off like crazy
Ollama has officially started work on MLX support! For those who don't know, this is huge for anyone running models locally on their Mac. MLX is designed to fully utilize Apple's unified memory and GPU. Expect faster, more efficient LLM training, execution and inference speeds.
TL;DR: The open-source tool that lets local LLMs watch your screen launches tonight! Thanks to your feedback, it now has a 1-command install (completely offline no certs to accept), supports any OpenAI-compatible API, and has mobile support. I'd love your feedback!
You guys are so amazing! After all the feedback from my last post, I'm very happy to announce that Observer AI is almost officially launched! I want to thank everyone for their encouragement and ideas.
For those who are new, Observer AI is a privacy-first, open-source tool to build your own micro-agents that watch your screen (or camera) and trigger simple actions, all running 100% locally.
What's New in the last few days(Directly from your feedback!):
✅ 1-Command 100% Local Install: I made it super simple. Just run docker compose up --build and the entire stack runs locally. No certs to accept or "online activation" needed.
✅ Universal Model Support: You're no longer limited to Ollama! You can now connect to any endpoint that uses the OpenAI v1/chat standard. This includes local servers like LM Studio, Llama.cpp, and more.
✅ Mobile Support: You can now use the app on your phone, using its camera and microphone as sensors. (Note: Mobile browsers don't support screen sharing).
My Roadmap:
I hope that I'm just getting started. Here's what I will focus on next:
Standalone Desktop App: A 1-click installer for a native app experience. (With inference and everything!)
Discord Notifications
Telegram Notifications
Slack Notifications
Agent Sharing: Easily share your creations with others via a simple link.
And much more!
Let's Build Together:
This is a tool built for tinkerers, builders, and privacy advocates like you. Your feedback is crucial.
There are moments in life that are monumental and game-changing. This is one of those moments for me.
Background: I’m a 53-year-old attorney with virtually zero formal coding or software development training. I can roll up my sleeves and do some basic HTML or use the Windows command prompt, for simple "ipconfig" queries, but that's about it. Many moons ago, I built a dual-boot Linux/Windows system, but that’s about the greatest technical feat I’ve ever accomplished on a personal PC. I’m a noob, lol.
AI. As AI seemingly took over the world’s consciousness, I approached it with skepticism and even resistance ("Great, we're creating Skynet"). Not more than 30 days ago, I had never even deliberately used a publicly available paid or free AI service. I hadn’t tried ChatGPT or enabled AI features in the software I use. Probably the most AI usage I experienced was seeing AI-generated responses from normal Google searches.
The Awakening. A few weeks ago, a young attorney at my firm asked about using AI. He wrote a persuasive memo, and because of it, I thought, "You know what, I’m going to learn it."
So I went down the AI rabbit hole. I did some research (Google and YouTube videos), read some blogs, and then I looked at my personal gaming machine and thought it could run a local LLM (I didn’t even know what the acronym stood for less than a month ago!). It’s an i9-14900k rig with an RTX 5090 GPU, 64 GBs of RAM, and 6 TB of storage. When I built it, I didn't even think about AI – I was focused on my flight sim hobby and Monster Hunter Wilds. But after researching, I learned that this thing can run a local and private LLM!
Today. I devoured how-to videos on creating a local LLM environment. I started basic: I deployed Ubuntu for a Linux environment using WSL2, then installed the Nvidia toolkits for 50-series cards. Eventually, I got Docker working, and after a lot of trial and error (5+ hours at least), I managed to get Ollama and Open WebUI installed and working great. I settled on Gemma3 12B as my first locally-run model.
I am just blown away. The use cases are absolutely endless. And because it’s local and private, I have unlimited usage?! Mind blown. I can’t even believe that I waited this long to embrace AI. And Ollama seems really easy to use (granted, I’m doing basic stuff and just using command line inputs).
So for anyone on the fence about AI, or feeling intimidated by getting into the OS weeds (Linux) and deploying a local LLM, know this: If a 53-year-old AARP member with zero technical training on Linux or AI can do it, so can you.
Today, during the firm partner meeting, I’m going to show everyone my setup and argue for a locally hosted AI solution – I have no doubt it will help the firm.
EDIT: I appreciate everyone's support and suggestions! I have looked up many of the plugins and suggested apps that folks have suggested and will undoubtedly try out a few (e.g,, MCP, Open Notebook Tika Apache, etc.). Some of the recommended apps seem pretty technical because I'm not very experienced with Linux environments (though I do love the OS as it seems "light" and intuitive), but I am learning! Thank you and looking forward to being more active on this sub-reddit.
So, I was tinkering around and did a little digging on Shodan.io (you know, as one does on a lazy afternoon). Using the filter port:11434 html:"Ollama", I checked out the first 1,000 results out of a whopping 4,963. And guess what? It's like a free-for-all out there! Tons of Ollama servers are wide open without any authentication.
Hold on to your keyboards, because here's the Top 10 most used models on these open servers:
llama2- 186 occurrences
llama3.1- 176 occurrences
llama3- 166 occurrences
llama3.2- 152 occurrences
nomic-embed-text- 126 occurrences
qwen:0.5b - 104 occurrences
mistral- 97 occurrences
llama3.1:8b - 70 occurrences
llama3.1:70b - 62 occurrences
gemma2- 61 occurrences
Now, back to the serious (but still fun) stuff:
Why should you care?
Unwanted Guests: Leaving your Ollama server door wide open is like hosting a party and letting random strangers raid your fridge. They're munching on your compute resources without even saying "Thanks!"
Security Shenanigans: Without authentication, anyone could potentially mess with your models, steal sensitive data, or, worst-case scenario, turn your server into a meme-generating monster. Yikes!
What can you do?
Lock the Door (Enable Authentication): Just like you wouldn't leave your front door unlocked (unless you trust your neighborhood squirrels), don't leave your servers open. Implement authentication to keep the party invite-only.
Hide and Seek (Network Segmentation): Consider setting up firewalls or VPNs. Make those unauthorized users work harder than finding Waldo in a candy cane factory.
Check Yourself Before You Wreck Yourself (Regular Audits): Periodically review your server settings. Think of it as a health check-up but for your server—minus the awkward small talk.
Final Thoughts:
Let's keep our Ollama community safe and efficient. After all, nobody wants to be that person who unknowingly hosts the world's slowest and least secure AI rave.
P.S. The data wrangling for this post was done using o1-preview, but the genius idea (and any bad jokes) are all mine!
Edit: I double-checked the model card on Ollama(https://ollama.com/library/deepseek-r1), and it does mention DeepSeek R1 Distill Qwen 7B in the metadata. So this is actually a distilled model. But honestly, that still impresses me!
Just discovered DeepSeek R1 and I'm pretty hyped about it. For those who don't know, it's a new open-source AI model that matches OpenAI o1 and Claude 3.5 Sonnet in math, coding, and reasoning tasks.
You can check out Reddit to see what others are saying about DeepSeek R1 vs OpenAI o1 and Claude 3.5 Sonnet. For me it's really good - good enough to be compared with those top models.
And the best part? You can run it locally on your machine, with total privacy and 100% FREE!!
I've got it running locally and have been playing with it for a while. Here's my setup - super easy to follow:
(Just a note: While I'm using a Mac,this guide works exactly the same for Windows and Linux users*! 👌)*
1) Install Ollama
Quick intro to Ollama: It's a tool for running AI models locally on your machine. Grab it here: https://ollama.com/download
2) Next, you'll need to pull and run the DeepSeek R1 model locally.
Ollama offers different model sizes - basically, bigger models = smarter AI, but need better GPU. Here's the lineup:
1.5B version (smallest):
ollama run deepseek-r1:1.5b
8B version:
ollama run deepseek-r1:8b
14B version:
ollama run deepseek-r1:14b
32B version:
ollama run deepseek-r1:32b
70B version (biggest/smartest):
ollama run deepseek-r1:70b
Maybe start with a smaller model first to test the waters. Just open your terminal and run:
ollama run deepseek-r1:8b
Once it's pulled, the model will run locally on your machine. Simple as that!
Note: The bigger versions (like 32B and 70B) need some serious GPU power. Start small and work your way up based on your hardware!
3) Set up Chatbox - a powerful client for AI models
Quick intro to Chatbox: a free, clean, and powerful desktop interface that works with most models. I started it as a side project for 2 years. It’s privacy-focused (all data stays local) and super easy to set up—no Docker or complicated steps. Download here: https://chatboxai.app
In Chatbox, go to settings and switch the model provider to Ollama. Since you're running models locally, you can ignore the built-in cloud AI options - no license key or payment is needed!
Then set up the Ollama API host - the default setting is http://127.0.0.1:11434, which should work right out of the box. That's it! Just pick the model and hit save. Now you're all set and ready to chat with your locally running Deepseek R1! 🚀
Hope this helps! Let me know if you run into any issues.
---------------------
Here are a few tests I ran on my local DeepSeek R1 setup (loving Chatbox's artifact preview feature btw!) 👇
Explain TCP:
Honestly, this looks pretty good, especially considering it's just an 8B model!
Make a Pac-Man game:
It looks great, but I couldn’t actually play it. I feel like there might be a few small bugs that could be fixed with some tweaking. (Just to clarify, this wasn’t done on the local model — my mac doesn’t have enough space for the largest deepseek R1 70b model, so I used the cloud model instead.)
---------------------
Honestly, I’ve seen a lot of overhyped posts about models here lately, so I was a bit skeptical going into this. But after testing DeepSeek R1 myself, I think it’s actually really solid. It’s not some magic replacement for OpenAI or Claude, but it’s surprisingly capable for something that runs locally. The fact that it’s free and works offline is a huge plus.
What do you guys think? Curious to hear your honest thoughts.
Just wanted to share a personal project I've been working on in my freetime. I'm trying to build an interactive, voice-driven Live2D avatar.
The basic idea is: my voice goes in -> gets transcribed locally with Whisper -> that text gets sent to the Ollama api (along with history and a personality prompt) -> the response comes back -> gets turned into speech with a local TTS -> and finally animates the Live2D character (lipsync + emotions).
My main goal was to see if I could get this whole chain running smoothly locally on my somewhat old GTX 1080 Ti. Since I also like being able to use latest and greatest models + ability to run bigger models on mac or whatever, I decided to make this work with ollama api so I can just plug and play that.
Getting the character (I included a demo model, Aria) to sound right definitely takes some fiddling with the prompt in the personality.txt file. Any tips for keeping local LLMs consistently in character during conversations?
The whole thing's built in C#, which was a fun departure from the usual Python AI world for me, and the performance has been pretty decent.