r/LocalAIStack • u/ruffus_or • 4h ago
r/LocalAIStack • u/fly-ute • 20h ago
[Project] La Gaceta del LLM Local – Nº0: measuring Strata on RDNA3 (gfx1100) with real numbers
reddit.comr/LocalAIStack • u/Legacy1320 • 1d ago
The Smallest Explorer in the Most Dangerous Desert
A tiny desert explorer discovers something much bigger than expected. This unusual creature encounter was imagined with wizstar ai.
r/LocalAIStack • u/SpeculationAllTheWay • 1d ago
We just released BaseQuant V1 for Qwen3.6-35B-A3B and Gemma 4 26B-A4B, benchmarked against your UD quants. Would love feedback
Hey everyone, we just released BaseQuant V1 for two MoE models at Base Compute: Qwen3.6-35B-A3B and Gemma 4 26B-A4B-it. The goal is to preserve more of the original model without making the files bigger.
We compared against Unsloth’s UD quants of the same size. On Gemma 4, our 16.93 GB GGUF had ~2.4% lower KL divergence from BF16 than UD-Q4_K_XL (~4.8% lower with our baseRT .base bundle). On Qwen3.6, our 22.13 GB GGUF is on par with UD-Q4_K_M overall. Our .base bundle is ~20% lower on long context and ~10% lower on agentic coding. UD still wins in some categories (chat on Gemma, math on Qwen). Full per-category tables are on the model cards.
Would love for you to try them and tell us how they hold up in your everyday workloads!
r/LocalAIStack • u/UniqueImplements • 1d ago
What is the best way forward for my boomer parents?
reddit.comr/LocalAIStack • u/Honest_Sea6861 • 1d ago
Quokka: a hardware-aware router for local AI. What would you need to trust automatic model selection?
r/LocalAIStack • u/Fit-Blueberry-5390 • 1d ago
Basalt ported to NVIDIA V100: Qwen3.8-Flash-Next at 91 tok/s (140 tok/s on long answers) on 2x V100 32GB
r/LocalAIStack • u/a-pyctam • 1d ago
Which inference engine squeezes max performance out of my setup without sacrificing precision? (Coding + image gen)
reddit.comr/LocalAIStack • u/refractr • 2d ago
Tested small models on document extraction with a score that punishes made-up values. Only covered a few local ones, help me add more
r/LocalAIStack • u/alez0077 • 2d ago
modelship 0.11: Vulkan (AMD/Intel) nodes, arm64 CUDA for DGX Spark, and an sd-server backend for image gen
I've been building modelship, a self-hosted, OpenAI-compatible server. It puts chat, embeddings, STT, TTS and image generation behind one /v1 API. Underneath it runs vLLM, llama.cpp, stable-diffusion.cpp, whisper.cpp, sherpa-onnx and diffusers, scheduled on Ray, so you can spread models across several machines. 0.11 is out today.
What's new
Vulkan variant for AMD and Intel GPUs.
Install with
mship bootstrap --vulkanon Linux, or use the-vulkanDocker image with--device /dev/dri. There's alsovariant: vulkanin the Helm chart.llama.cpp and stable-diffusion.cpp run on the GPU.
Integrated GPUs like Strix Halo are sized from system memory, so you don't have to guess VRAM.
arm64 CUDA. The
-cudaimage is now multi-arch, and llama_server runs on the GPU on DGX Spark / GB10.stable-diffusion.cpp now runs as an
sd-serversubprocess.It supports GPU on cuda, vulkan and metal, and a fractional
num_gpusto share a card.Requests for more than 8 images are split into several jobs.
Images also take
output_format(png/jpeg/webp).
llama_server.
It's on llama.cpp b12465.
MoE models that don't fit on the GPU get expert tensor overrides (
-ot) automatically.Added the
f8_e4m3KV cache type andmedia_allow_private_urls.A GPU deploy whose CPU-side part won't fit in RAM is now refused up front instead of loading and swapping.
Responses API.
truncation: autotrims the oldest input items.max_output_tokensnow bounds the whole MCP tool loop, and responses cut off by the token limit end asincomplete.vLLM 0.33 (torch 2.14). Multi-choice responses are parsed per choice,
parallel_tool_calls: falseis honoured, and a tool call cut off by the token limit reportsfinish_reason: length.
To upgrade a native install, run uv tool upgrade mship, then mship bootstrap.
Known issues
Vision models (mmproj) on a dedicated GPU are sized without the projector, so a model sized close to the VRAM limit can OOM at load. Lowering
n_ctxworks around it.Fedora native installs can hit an SSL error in replicas, so use the Docker images there.
f8_e4m3KV cache aborts on the Vulkan backend.
Repo and full notes: https://github.com/modelship-ai/modelship
Happy to hear about hardware I haven't tested, especially Intel Arc and older AMD cards on Vulkan.
r/LocalAIStack • u/Headcrab_Raiden • 2d ago
I built LLMLifeline: one install that sets up GGUF chat + ComfyUI image/video, sized to your GPU, and shares it to every device in your house ($11.11 one-time, no cloud)
Solo dev here. I built LLMLifeline because I wanted to contribute to the utter destruction of the companies that have been stealing out data from the beginning, getting a slap on the wrist before being allowed to continue stealing, and putting the power back in the hands of the people who should have had it from the beginning. US.
I wanted a local stack my whole household could actually use, not just me.
What it does:
• One installer (Windows/macOS/Linux). First run sets everything up itself: chat, plus ComfyUI for image/video generation
• Chat tiers matched to your GPU: Qwen 27B (large), Gemma 4 12B (medium), Qwen 4B (small) — it starts on the best tier your card runs well and moves up if you upgrade
• Chat with any .gguf model, including models bigger than your VRAM, with streaming replies; swap models freely, or bring a Hugging Face folder and it converts for you
• Image/video: Z-Image Turbo / RealVisXL 5 / Realistic Vision 6 tiers, Wan video models in three sizes
• The part I actually built it for: whole-home sharing. One machine runs the model; phones (tested with trash phones to prove it. If it's got a browser that can point to the local ip, then it works), tablets, Chromebooks and other PCs connect over your LAN through the browser, each device approved with a six-digit code. Per-person profiles, document indexing/quoting, and a computer-use agent that asks before each action. Handles projects, can increase context for better responses, with a button click, and you can delete anything generated from the app or from the program folder with easily found folders.
• No cloud, no account, no subscription, no telemetry, no auto-updates. License check is offline after activation; one key covers 3 computers
"Why not just Ollama/LM Studio" answer: if you enjoy assembling and tuning your own stack, you don't need this, and I mean that.
LLMLifeline is for the step before that — people (and households) who want local AI without becoming their own sysadmin. It's a baseline that sets itself up around the hardware you have today and it takes little brain power to use since were already using similar systems. This has the chatbot video and image generator built in to the same system, manages your swappable profiles in memory so you don't have to sign in every time, does computer use over the network to collaborate, and uses clever techniques like hidden markov modelling and paging tricks to make your hardware more efficient with heavier models that don't actually fit on your GPU.
Best part? It scales with you. You don't have to download new software later; as you update your PC it will just get faster.
Caveats, up front: Windows is where I build and use it daily. macOS/Linux builds pass automated checks every release but I haven't tested on real Mac/Linux hardware yet. Models are uncensored by default but you can drag and drop your own GGUFs or model folders and it will read them immediately.
Small models are noticeably worse at instruction-following and computer use. - Not my problem.
First-run downloads are big (about 63 GB of models; skippable if you're bringing your own). This is the easiest path however because I tested the models to find the most usable ones for each path (chat/code, image/video)
$11.11 one-time. If you figure out how to rip it, idc share it with the world. I'm hoping to make a little money off it since it took me months of work but the ultimate goal is sticking the finger to the AI corps. That's why it's $11.11 one-time with no forced updates that might screw you up. Not to mention if you do update, there is a contingency to ensure that if it breaks, you can revert to your previous version.
Demo, including it running offline: https://youtu.be/Kw0H1FlHotE
Full details and requirements: https://hector-gonzalez.nyctitropist.workers.dev/llmlifeline/
Genuinely here for the harsh questions: what's missing, what would make you not trust it, what would you need to see before running a closed installer on your machine?
I'm happy to hand out some free keys for power users to mess around with it.
r/LocalAIStack • u/company_url_finder • 2d ago
Stuck on 12GB VRAM but need 27B-level agent reasoning? SenseNova-Loop-9B loops its layers to match Qwen3.8-27B, official GGUF & AWQ out now
galleryr/LocalAIStack • u/CooperDK • 2d ago
Got Basalt running with an RTX 3060 beside a 5060 Ti: 42 to 80 tok/s, and the same build takes 40xx cards
reddit.comr/LocalAIStack • u/Beautiful-QGI-6166 • 2d ago
Can a local AI model beat Claude Haiku and GPT-6 Luna?
reddit.comr/LocalAIStack • u/AimCode • 3d ago
AimStack — a fully self-built local AI stack in pure C: ternary inference (zero third-party runtime), BitNet 2B at 0.87GB on a 2018 laptop, plus a GPU backend that matches the C engine token-for-token
Repo: https://github.com/tanxiansheng221/aimstack-sanzhidiji (AGPL-3.0)
Everything is hand-written — C inference core, plugin bus, multi-engine crawler with quality gates, FTS knowledge base, training pipeline, and a 117-chapter build log written as it happened (positive and negative results shipped together). Weights are Microsoft BitNet b1.58 2B (MIT), converted to a custom AIM-W packing format.
What ternary quantization buys you (CPU side)
All transformer matrices are ternary {-1, 0, +1}, packed 2 bits/weight, plus Q8 embeddings → the whole model is 0.87GB instead of ~4.8GB (−78%)
The C core never multiplies a float weight: it's table lookup + integer accumulation, SIMD fast path measured at 85.53 tok/s (41× over baseline)
BitNet 2B full model runs 3.20 tok/s dual-worker on a 2018 dual-core 8GB laptop — full chain (inference + KB + plugins + web search) offline, no cloud relay
What ternary quantization buys you (GPU side)
The GPU backend loads the same 0.87GB ternary file — weights stay in {-s, 0, +s}, dequantized on load, matmul in fp32
On a Radeon Pro W7900: 13.68 tok/s (~8× the laptop C engine), 256/256 greedy tokens bit-identical to the C engine. Two totally different implementations (integer lookup core vs FP matmul) producing the same logits is the proof the quantization is faithful
Verified by a 3-engine × 3-question × 3-test matrix (C-Linux/gcc13, torch fp32, torch bf16): 17/18 exact — the one miss is documented, not hidden
The honest boundaries (what's NOT done yet)
bf16 compute is 2.4× faster (32.96 tok/s) but flips argmax on near-tie logits — reported, not blessed; only fp32 is parity-verified
The GPU path gets the storage benefit of ternary (78% less memory) but not yet the compute benefit — a true 2-bit integer kernel (Triton/CUDA) is the next stage
The UI server drives the C engine (CPU) only; the GPU backend is a separate CLI
Linux engine build works and is token-identical to the Windows build, but runs serial (the worker pool is Windows-only for now)
No frameworks, no PyTorch in the product chain — PyTorch exists only in the optional GPU backend. Demo video, docs (EN/中文), and the full research log are in the repo.
r/LocalAIStack • u/jackalka • 3d ago
We built a chip prototype that runs Qwen 27B. You can try it live.
r/LocalAIStack • u/kind_entropy • 3d ago
A comparable local setup for someone using $100 monthly claude subscription
reddit.comr/LocalAIStack • u/Aware-Blacksmith1467 • 4d ago
Building Kevin, my home alone agent. 🔫🙂 A persistent agent In Pine that wakes on cam detect, runs fingerprinting, tracks who's visiting.
x.comr/LocalAIStack • u/Due_Initiative2641 • 4d ago
I built an open-source platform that pools a team's workstations (Macs, Windows and Linux boxes) into one OpenAI-compatible endpoint
My co-founder and I have been working on this for a few months and are launching today. Disclosure: it's our company; the desktop client is Apache-2.0, the control plane is hosted, and it's free up to 5 devices.
What it does: you install Nodea Desktop on the machines in your org, switch on sharing, and get one OpenAI-compatible endpoint. Behind it, a scheduler fits the models you assign onto whichever machines have free memory, and moves them as laptops come and go.
Stack: Rust/Tauri app, llama.cpp under the hood (Metal on Apple Silicon, Vulkan on NVIDIA/AMD/Intel, CPU fallback). Any GGUF model llama.cpp can serve that fits the machine; models are assigned per org, downloaded and cached per device. Signed auto-updates.
Honest limitations:
Requests go through our API to reach your machines (not logged or stored). No direct LAN routing yet.
One model runs on one machine; no splitting a model across machines yet.
Windows/Linux run on the GPU through Vulkan (tested on NVIDIA) and are still labelled preview. Machines without a supported GPU fall back to CPU, which is slow for now (no AVX2 yet).
Why not exo / GPUStack / llama.cpp RPC? Those are great if you run your own cluster. We're aiming at teams with 20–200 laptops and nobody to run one: sign in, toggle on, done, with an org, roles and API keys on top.
Repo: https://github.com/nodea-sh/nodea-client
Site: https://getnodea.com
Happy to answer anything, and especially keen on what models/quantizations you'd want on 16–36 GB machines.
r/LocalAIStack • u/WritHerAI • 4d ago

