r/LocalAIStack • • 4h ago

Anyone running one large LLM across Apple Silicon + NVIDIA GPUs?

Thumbnail reddit.com
1 Upvotes

r/LocalAIStack • • 14h ago

Better Local LLMs

1 Upvotes

I am working on a project to build better local LLMs (more info in the pictures below)! If you are interested in helping with compute, funding, outreach, or development please message me or comment below!


r/LocalAIStack • • 20h ago

[Project] La Gaceta del LLM Local – Nº0: measuring Strata on RDNA3 (gfx1100) with real numbers

Thumbnail reddit.com
1 Upvotes

r/LocalAIStack • • 21h ago

256 ARM Cores + 1TB RAM

1 Upvotes

r/LocalAIStack • • 1d ago

The Smallest Explorer in the Most Dangerous Desert

0 Upvotes

A tiny desert explorer discovers something much bigger than expected. This unusual creature encounter was imagined with wizstar ai.


r/LocalAIStack • • 1d ago

We just released BaseQuant V1 for Qwen3.6-35B-A3B and Gemma 4 26B-A4B, benchmarked against your UD quants. Would love feedback

Thumbnail
huggingface.co
7 Upvotes

Hey everyone, we just released BaseQuant V1 for two MoE models at Base Compute: Qwen3.6-35B-A3B and Gemma 4 26B-A4B-it. The goal is to preserve more of the original model without making the files bigger.

We compared against Unsloth’s UD quants of the same size. On Gemma 4, our 16.93 GB GGUF had ~2.4% lower KL divergence from BF16 than UD-Q4_K_XL (~4.8% lower with our baseRT .base bundle). On Qwen3.6, our 22.13 GB GGUF is on par with UD-Q4_K_M overall. Our .base bundle is ~20% lower on long context and ~10% lower on agentic coding. UD still wins in some categories (chat on Gemma, math on Qwen). Full per-category tables are on the model cards.

Would love for you to try them and tell us how they hold up in your everyday workloads!


r/LocalAIStack • • 1d ago

What is the best way forward for my boomer parents?

Thumbnail reddit.com
1 Upvotes

r/LocalAIStack • • 1d ago

Quokka: a hardware-aware router for local AI. What would you need to trust automatic model selection?

Thumbnail
reddit.com
1 Upvotes

r/LocalAIStack • • 1d ago

Basalt ported to NVIDIA V100: Qwen3.8-Flash-Next at 91 tok/s (140 tok/s on long answers) on 2x V100 32GB

Thumbnail
reddit.com
1 Upvotes

r/LocalAIStack • • 1d ago

Which inference engine squeezes max performance out of my setup without sacrificing precision? (Coding + image gen)

Thumbnail reddit.com
1 Upvotes

r/LocalAIStack • • 1d ago

4xM(x) Studio Apple Cluster

Post image
1 Upvotes

r/LocalAIStack • • 2d ago

Tested small models on document extraction with a score that punishes made-up values. Only covered a few local ones, help me add more

Thumbnail
reddit.com
1 Upvotes

r/LocalAIStack • • 2d ago

modelship 0.11: Vulkan (AMD/Intel) nodes, arm64 CUDA for DGX Spark, and an sd-server backend for image gen

1 Upvotes

I've been building modelship, a self-hosted, OpenAI-compatible server. It puts chat, embeddings, STT, TTS and image generation behind one /v1 API. Underneath it runs vLLM, llama.cpp, stable-diffusion.cpp, whisper.cpp, sherpa-onnx and diffusers, scheduled on Ray, so you can spread models across several machines. 0.11 is out today.

What's new

  • Vulkan variant for AMD and Intel GPUs.

    • Install with mship bootstrap --vulkan on Linux, or use the -vulkan Docker image with --device /dev/dri. There's also variant: vulkan in the Helm chart.

    • llama.cpp and stable-diffusion.cpp run on the GPU.

    • Integrated GPUs like Strix Halo are sized from system memory, so you don't have to guess VRAM.

  • arm64 CUDA. The -cuda image is now multi-arch, and llama_server runs on the GPU on DGX Spark / GB10.

  • stable-diffusion.cpp now runs as an sd-server subprocess.

    • It supports GPU on cuda, vulkan and metal, and a fractional num_gpus to share a card.

    • Requests for more than 8 images are split into several jobs.

    • Images also take output_format (png/jpeg/webp).

  • llama_server.

    • It's on llama.cpp b12465.

    • MoE models that don't fit on the GPU get expert tensor overrides (-ot) automatically.

    • Added the f8_e4m3 KV cache type and media_allow_private_urls.

    • A GPU deploy whose CPU-side part won't fit in RAM is now refused up front instead of loading and swapping.

  • Responses API. truncation: auto trims the oldest input items. max_output_tokens now bounds the whole MCP tool loop, and responses cut off by the token limit end as incomplete.

  • vLLM 0.33 (torch 2.14). Multi-choice responses are parsed per choice, parallel_tool_calls: false is honoured, and a tool call cut off by the token limit reports finish_reason: length.

To upgrade a native install, run uv tool upgrade mship, then mship bootstrap.

Known issues

  • Vision models (mmproj) on a dedicated GPU are sized without the projector, so a model sized close to the VRAM limit can OOM at load. Lowering n_ctx works around it.

  • Fedora native installs can hit an SSL error in replicas, so use the Docker images there.

  • f8_e4m3 KV cache aborts on the Vulkan backend.

Repo and full notes: https://github.com/modelship-ai/modelship

Happy to hear about hardware I haven't tested, especially Intel Arc and older AMD cards on Vulkan.


r/LocalAIStack • • 2d ago

I built LLMLifeline: one install that sets up GGUF chat + ComfyUI image/video, sized to your GPU, and shares it to every device in your house ($11.11 one-time, no cloud)

Thumbnail
reddit.com
1 Upvotes

Solo dev here. I built LLMLifeline because I wanted to contribute to the utter destruction of the companies that have been stealing out data from the beginning, getting a slap on the wrist before being allowed to continue stealing, and putting the power back in the hands of the people who should have had it from the beginning. US.

I wanted a local stack my whole household could actually use, not just me.

What it does:
• One installer (Windows/macOS/Linux). First run sets everything up itself: chat, plus ComfyUI for image/video generation
• Chat tiers matched to your GPU: Qwen 27B (large), Gemma 4 12B (medium), Qwen 4B (small) — it starts on the best tier your card runs well and moves up if you upgrade
• Chat with any .gguf model, including models bigger than your VRAM, with streaming replies; swap models freely, or bring a Hugging Face folder and it converts for you
• Image/video: Z-Image Turbo / RealVisXL 5 / Realistic Vision 6 tiers, Wan video models in three sizes

• The part I actually built it for: whole-home sharing. One machine runs the model; phones (tested with trash phones to prove it. If it's got a browser that can point to the local ip, then it works), tablets, Chromebooks and other PCs connect over your LAN through the browser, each device approved with a six-digit code. Per-person profiles, document indexing/quoting, and a computer-use agent that asks before each action. Handles projects, can increase context for better responses, with a button click, and you can delete anything generated from the app or from the program folder with easily found folders.

• No cloud, no account, no subscription, no telemetry, no auto-updates. License check is offline after activation; one key covers 3 computers

"Why not just Ollama/LM Studio" answer: if you enjoy assembling and tuning your own stack, you don't need this, and I mean that.

LLMLifeline is for the step before that — people (and households) who want local AI without becoming their own sysadmin. It's a baseline that sets itself up around the hardware you have today and it takes little brain power to use since were already using similar systems. This has the chatbot video and image generator built in to the same system, manages your swappable profiles in memory so you don't have to sign in every time, does computer use over the network to collaborate, and uses clever techniques like hidden markov modelling and paging tricks to make your hardware more efficient with heavier models that don't actually fit on your GPU.

Best part? It scales with you. You don't have to download new software later; as you update your PC it will just get faster.

Caveats, up front: Windows is where I build and use it daily. macOS/Linux builds pass automated checks every release but I haven't tested on real Mac/Linux hardware yet. Models are uncensored by default but you can drag and drop your own GGUFs or model folders and it will read them immediately.

Small models are noticeably worse at instruction-following and computer use. - Not my problem.

First-run downloads are big (about 63 GB of models; skippable if you're bringing your own). This is the easiest path however because I tested the models to find the most usable ones for each path (chat/code, image/video)

$11.11 one-time. If you figure out how to rip it, idc share it with the world. I'm hoping to make a little money off it since it took me months of work but the ultimate goal is sticking the finger to the AI corps. That's why it's $11.11 one-time with no forced updates that might screw you up. Not to mention if you do update, there is a contingency to ensure that if it breaks, you can revert to your previous version.

Demo, including it running offline: https://youtu.be/Kw0H1FlHotE
Full details and requirements: https://hector-gonzalez.nyctitropist.workers.dev/llmlifeline/

Genuinely here for the harsh questions: what's missing, what would make you not trust it, what would you need to see before running a closed installer on your machine?

I'm happy to hand out some free keys for power users to mess around with it.


r/LocalAIStack • • 2d ago

Stuck on 12GB VRAM but need 27B-level agent reasoning? SenseNova-Loop-9B loops its layers to match Qwen3.8-27B, official GGUF & AWQ out now

Thumbnail gallery
1 Upvotes

r/LocalAIStack • • 2d ago

All AI Tools In One Website Free

Thumbnail interconnectd.com
1 Upvotes

r/LocalAIStack • • 2d ago

Got Basalt running with an RTX 3060 beside a 5060 Ti: 42 to 80 tok/s, and the same build takes 40xx cards

Thumbnail reddit.com
0 Upvotes

r/LocalAIStack • • 2d ago

Can a local AI model beat Claude Haiku and GPT-6 Luna?

Thumbnail reddit.com
1 Upvotes

r/LocalAIStack • • 3d ago

AimStack — a fully self-built local AI stack in pure C: ternary inference (zero third-party runtime), BitNet 2B at 0.87GB on a 2018 laptop, plus a GPU backend that matches the C engine token-for-token

Thumbnail
github.com
1 Upvotes

Repo: https://github.com/tanxiansheng221/aimstack-sanzhidiji (AGPL-3.0)

Everything is hand-written — C inference core, plugin bus, multi-engine crawler with quality gates, FTS knowledge base, training pipeline, and a 117-chapter build log written as it happened (positive and negative results shipped together). Weights are Microsoft BitNet b1.58 2B (MIT), converted to a custom AIM-W packing format.

What ternary quantization buys you (CPU side)

All transformer matrices are ternary {-1, 0, +1}, packed 2 bits/weight, plus Q8 embeddings → the whole model is 0.87GB instead of ~4.8GB (−78%)

The C core never multiplies a float weight: it's table lookup + integer accumulation, SIMD fast path measured at 85.53 tok/s (41× over baseline)

BitNet 2B full model runs 3.20 tok/s dual-worker on a 2018 dual-core 8GB laptop — full chain (inference + KB + plugins + web search) offline, no cloud relay

What ternary quantization buys you (GPU side)

The GPU backend loads the same 0.87GB ternary file — weights stay in {-s, 0, +s}, dequantized on load, matmul in fp32

On a Radeon Pro W7900: 13.68 tok/s (~8× the laptop C engine), 256/256 greedy tokens bit-identical to the C engine. Two totally different implementations (integer lookup core vs FP matmul) producing the same logits is the proof the quantization is faithful

Verified by a 3-engine × 3-question × 3-test matrix (C-Linux/gcc13, torch fp32, torch bf16): 17/18 exact — the one miss is documented, not hidden

The honest boundaries (what's NOT done yet)

bf16 compute is 2.4× faster (32.96 tok/s) but flips argmax on near-tie logits — reported, not blessed; only fp32 is parity-verified

The GPU path gets the storage benefit of ternary (78% less memory) but not yet the compute benefit — a true 2-bit integer kernel (Triton/CUDA) is the next stage

The UI server drives the C engine (CPU) only; the GPU backend is a separate CLI

Linux engine build works and is token-identical to the Windows build, but runs serial (the worker pool is Windows-only for now)

No frameworks, no PyTorch in the product chain — PyTorch exists only in the optional GPU backend. Demo video, docs (EN/中文), and the full research log are in the repo.


r/LocalAIStack • • 3d ago

We built a chip prototype that runs Qwen 27B. You can try it live.

Thumbnail
reddit.com
1 Upvotes

r/LocalAIStack • • 3d ago

A comparable local setup for someone using $100 monthly claude subscription

Thumbnail reddit.com
1 Upvotes

r/LocalAIStack • • 4d ago

I measured how easily a local 7B model reverse-engineers a binary, then built hardening that targets the LLM layer specifically

9 Upvotes

Most anti-reverse-engineering assumes a human in a disassembler. The threat I care about now is a local, unrestricted model driving those tools. So besides the classic stuff, the harness includes defenses aimed at the model: inflating the analysis surface so it burns context, decoy identifiers that mislead semantic guesses, and honeytokens that detect when a crack was AI-generated.

My own before/after is honest but small: one run, a 4-bit 7B on an 8 GB GPU. The unprotected build was cracked in 3 steps and about 1,300 tokens. The hardened one burned roughly 13x the tokens and still wasn't cracked. I'd really like people with bigger rigs to run it. It's one command: make slice-k-run MODEL=<your-model>, and it prints a scored before/after. or online modals

Most of the code was written with Claude agents under my direction. Happy to talk through any design choice.

Repo: https://github.com/seifheakal/blue-harness (AGPL-3.0)


r/LocalAIStack • • 4d ago

Building Kevin, my home alone agent. 🔫🙂 A persistent agent In Pine that wakes on cam detect, runs fingerprinting, tracks who's visiting.

Thumbnail x.com
1 Upvotes

r/LocalAIStack • • 4d ago

I built an open-source platform that pools a team's workstations (Macs, Windows and Linux boxes) into one OpenAI-compatible endpoint

Thumbnail
reddit.com
1 Upvotes

My co-founder and I have been working on this for a few months and are launching today. Disclosure: it's our company; the desktop client is Apache-2.0, the control plane is hosted, and it's free up to 5 devices.

What it does: you install Nodea Desktop on the machines in your org, switch on sharing, and get one OpenAI-compatible endpoint. Behind it, a scheduler fits the models you assign onto whichever machines have free memory, and moves them as laptops come and go.

Stack: Rust/Tauri app, llama.cpp under the hood (Metal on Apple Silicon, Vulkan on NVIDIA/AMD/Intel, CPU fallback). Any GGUF model llama.cpp can serve that fits the machine; models are assigned per org, downloaded and cached per device. Signed auto-updates.

Honest limitations:

  • Requests go through our API to reach your machines (not logged or stored). No direct LAN routing yet.

  • One model runs on one machine; no splitting a model across machines yet.

  • Windows/Linux run on the GPU through Vulkan (tested on NVIDIA) and are still labelled preview. Machines without a supported GPU fall back to CPU, which is slow for now (no AVX2 yet).

Why not exo / GPUStack / llama.cpp RPC? Those are great if you run your own cluster. We're aiming at teams with 20–200 laptops and nobody to run one: sign in, toggle on, done, with an org, roles and API keys on top.

Repo: https://github.com/nodea-sh/nodea-client

Site: https://getnodea.com

Happy to answer anything, and especially keen on what models/quantizations you'd want on 16–36 GB machines.


r/LocalAIStack • • 4d ago

I added Qwen3.6-35B-A3B to my C inference engine: 10–13 tok/s in real chat on a GTX 1650 4GB + 16GB RAM

Post image
1 Upvotes