r/SelfHostedAI • u/KungFuSaifooo • 1h ago
I made a voice assistant for the Mac notch that runs Qwen 3.5 4B locally, most commands skip the LLM entirely
r/SelfHostedAI • u/KungFuSaifooo • 1h ago
r/SelfHostedAI • u/Historical-Singer771 • 42m ago
Have old computers or spare GPUs sitting around?
Llama RPC Farm links them into one local AI cluster. You get one dashboard and one API that works with OpenAI and Anthropic tools. There's no cloud bill, and your data stays on your own machines.
It's free, open source, and starts with a single command.
r/SelfHostedAI • u/ZikporxAI • 58m ago
r/SelfHostedAI • u/Fearless_Strain_6161 • 3h ago
I run a business and use several AI tools every day: Grok Bot, Claude, ChatGPT and Manus, and I'm thinking about adding Muse. Each one is great at something, but they don't talk to each other, so I end up being the one copying context between them.
My dream is one interface I talk to, by voice or text, that knows what's going on across everything, hands each job to the right AI, brings the results back to me, and only interrupts me for what actually needs me. Nothing important happens without my OK. I'm open to running my own AI on a server if that's what it takes.
For anyone who's built something like this:
How did you connect the different AIs? APIs, MCP, n8n or Zapier, something custom?
Did you self-host a central "brain," and what does it run on?
How does it decide which AI gets which job?
How do you keep shared memory and context across all of them?
What would you do differently if you started over?
Real setups, diagrams or screenshots are very welcome. Thank you!
r/SelfHostedAI • u/House__Music • 7h ago
r/SelfHostedAI • u/House__Music • 5h ago
Hey everyone,
I’ve been researching local AI hardware for a while, particularly 128GB Strix Halo mini PCs, but I’m wondering whether it’s actually worth investing several thousand dollars or if hosted models would be better for my use case.
**My main use cases:**
**Cybersecurity:** Penetration testing, red/purple teaming, vulnerability research, malware analysis, bug bounty research and security tool development.
**AI security:** Prompt injection, jailbreak research, adversarial testing and evaluating LLM vulnerabilities.
**Coding:** Building applications, Python tools, debugging and analysing large codebases.
**AI agents:** Autonomous or semi-autonomous workflows, tool calling and multi-step technical tasks.
**General AI:** Research, documentation analysis and everyday assistance.
I’d prefer open-weight models with minimal refusals for legitimate security research. Privacy and keeping sensitive data local are also important.
**Current hardware:** MacBook Pro M4 Pro with 48GB unified memory.
**Hardware I’m considering:**
Bosgame M5, Ryzen AI Max+ 395, 128GB RAM
GMKtec EVO-X2 / MINISFORUM MS-S1 Max, 128GB
NVIDIA DGX Spark / ASUS Ascent GX10
Dedicated NVIDIA or AMD GPU workstations
**Models I’m interested in:** GPT-OSS 120B, Qwen, DeepSeek, GLM and uncensored/abliterated fine-tunes.
**My main questions:**
Is local AI genuinely capable of complex cybersecurity research, coding and agentic workflows, or will hosted frontier models significantly outperform it?
Is 128GB Strix Halo a good investment, especially regarding inference speed, context length and quantisation?
Would an NVIDIA GPU setup be better due to CUDA and software compatibility?
Are local models reliable enough for coding agents, tool calling and autonomous workflows?
Would I be better off sticking with my 48GB MacBook and paying for hosted inference when needed?
—-
I’m based in Australia, where these machines cost roughly A$4,000 to A$6,000+. ($4600 specifically for the bogame m5)
I don’t mind spending the money if it genuinely improves my workflows, but I don’t want expensive hardware that ends up being less useful than a cloud subscription.
**For those running local AI for cybersecurity, coding or agents, what hardware/models are you using, and would you make the same investment again?**
Thanks!
r/SelfHostedAI • u/Majestic681 • 6h ago
For a small home inference server used by 2/3 users, what is the best way to query it remotely? Users are currently using LM Link which is very convenient, but LM Studio doesn’t seem to support the Splash M1 port. Is it Tailscale or Wireguard or something else?
r/SelfHostedAI • u/tanafras • 23h ago
Whelp. Progress Update - 3x RTX Pro 5k 72gb e 216GB received. 1 turned out to be a EDU card. Took a bit but PNY fully backs it as a warranty-covered card. Yay that. No functional, hardware/software difference. Was sold by an authorized VAR in their records.
So, I now have 1.25 GB of AI capacity. I've also ordered the next server today an HPE DL380 / 512 GB ECC RAm, I cried with the bank account a little bit. I also received the Dell T320 server and the HPE Z8 dual Xeon workstation as well. It has 96GB and 18! open memory slots. I'll have to check ATech out and see if I can grab some more RAM. I also have the UPS. oming, 4kva double conversion LifePO4 setup.
I also am using the Intel currenet generation NICs for 2x10Gbps since that is sufficient for AI workloading. Nonetheless I am bridging them so 20Gbps along with the integrated adapters, and yes, I am using 10Gbps switches.At some point I'll consider MMF 28 25gbps or the g40 200Gbps setup. If I go with G40s I'll likely get 5 of them. Still on the fence about that since it really doesn't get me anything beyond what I already have so unlikely Will likely just keep snapping up the peo 5k 2gbs as I find them 300w at 72GB is a tantalizing proposition after all.
The next step is the HPE DL380 G12 with an 8kva UPS and that will go into the. 100 amp breaker panels. Thus. I am cotinuing my journey of discovery and learning as I build up my Tier 3 equivalent data center.
r/SelfHostedAI • u/martinkoistinen • 14h ago
Hey guys and gals. This new project is perfect for this sub. Protecting access to open weights models and legislation that fosters domestic abundance of hardware to lower costs are among the groups primary goals.
I was able to use this to create a personalized letter to my representative in about 30 seconds. I just had to print it out along with the mailing label and put a stamp on it. Super easy and did it with my phone.
I would encourage everyone here to do the same! It only works for incumbent Democrat and Independent Senators and Representatives right now but it reads like Republicans and the recently elected are to be supported soon.
Anyway, I’ve done my part. Mailed off the letter to my representative and shared it with you all. Keep it going!
r/SelfHostedAI • u/NIDO_007 • 22h ago
r/SelfHostedAI • u/Whoisjunn • 1d ago
r/SelfHostedAI • u/Whoisjunn • 1d ago
r/SelfHostedAI • u/Mecworks • 1d ago
My 10-inch active cooling rack for GB10 class computes is opensource and available. Check the link in the original post.
r/SelfHostedAI • u/Legal-Bridge-8332 • 1d ago
r/SelfHostedAI • u/SufficientAbility821 • 1d ago
r/SelfHostedAI • u/fluffymeowmeowkitty • 1d ago
Hello everyone, I wanted to ask if anybody has a local model I can run on my machine that can process forms with checkmarks open the questions. Ability to press next ability to remember each form also able to invent and react to the questions that’s on the questionnaire. I’ve tried qweens, gpt oss. Tried plugging in sonnet and that ate away the $5 with 5 forms. I found muse ember effective and cheap. But I’m still trying to go cheaper and have this run on my machine. Any advice would really help appreciated.
r/SelfHostedAI • u/fluffymeowmeowkitty • 1d ago
Hello everyone, I wanted to ask if anybody has a local model I can run on my machine that can process forms with checkmarks open the questions. Ability to press next ability to remember each form also able to invent and react to the questions that’s on the questionnaire. I’ve tried qweens, gpt oss. Tried plugging in sonnet and that ate away the $5 with 5 forms. I found muse ember effective and cheap. But I’m still trying to go cheaper and have this run on my machine. Any advice would really help appreciated.
r/SelfHostedAI • u/fluffymeowmeowkitty • 1d ago
Hello everyone, I wanted to ask if anybody has a local model I can run on my machine that can process forms with checkmarks open the questions. Ability to press next ability to remember each form also able to invent and react to the questions that’s on the questionnaire. I’ve tried qweens, gpt oss. Tried plugging in sonnet and that ate away the $5 with 5 forms. I found muse ember effective and cheap. But I’m still trying to go cheaper and have this run on my machine. Any advice would really help appreciated.
r/SelfHostedAI • u/julianfromstagewise • 1d ago
r/SelfHostedAI • u/messy-setup • 1d ago
Exo fails at the last layer when deploying on two nodes
Hello fellow Local AI enthusiasts,
I have been trying to deploy LLMs to two nodes using pipeline over TCP/UDP
Machine 1 specs:
CPU: Ryzen 5900x
Ram: 32gb
GPU: RTX 4080 super (16 VRAM)
Machine 2 specs:
CPU: Core i7 4770
Ram:16gb
GPU: GTX 1080 (8 VRAM)
both machines are running cachy os
both nodes discover each other normally and when I start a deployment it works fine all the way till the last layer it would get stuck at 16/17 layer
models that I have tried:
mlx community/Nvidia nemotron 3.5 lightning 30b A3B 4bit
mlx community/qwen3.6 35b A3B 4bit
youssofal/qwen3.6 35b A3B 4bit abliterated heretic mlx 4bit
any help or guidance to the right direction would be greatly appreciated
thank you
r/SelfHostedAI • u/pwap_official • 1d ago
Making local AI accessible to users across operating systems and various PC specs poses significant challenges, as well as, the often complex software installation or technical know how average people need to get those local weights working. That is the core issue "1-Click AI In-Browser Assistant" is trying to address. An extension and vector database that keeps your data securely on your device with no ongoing subscription fees or cloud telemtry. Ideal for the budget conscious user seeking a privacy-first local AI experience while improving productivity.
r/SelfHostedAI • u/_pleeply • 1d ago
Hey everyone!
We've been working on Pleeply, a project exploring the idea of a personal AI ecosystem.
The idea is to bring local AI models, optional cloud intelligence, and eventually personal AI agents into one desktop experience.
We're still at an early stage. Right now, we have a landing page and an early access waitlist — we're not claiming to have a finished product.
We'd love to understand:
How do you currently manage your local AI models?
Do you switch between local and cloud models?
What is the most frustrating part of your current setup?
We're building Pleeply and would genuinely appreciate feedback from people who use local AI.
Project: https://pleeply.vercel.app
Disclosure: I'm part of the pleeply team)