r/LocalLLM • • 20m ago

News Tencent/Hy4 Flash-224B-A12B-weight (Ultimate cost-efficiency)

Post image
3 Upvotes

https://hy.tencent.ai/research/hy4-flash
Tencent released Hy4 Flash, a comparatively small model at 224B. Compared to Hy3, which was compressed to 235B, that's another breakthrough. With the smaller weight, Hy4 Flash finishes tasks 13.8x cheaper than Hy3, 24.5x cheaper than Hy4.

Saw the price table for Hy4 Flash. $0.07 in, $0.281 out and $0.003 per million tokens on cache reads. GLM-5.3-Flash is $0.12 and $0.42, cache $0.034, so about 1.7x on input and 1.5x on output. DeepSeek V4.1 Flash is $0.15 and $0.60, cache $0.003, just over 2x on both, and that's their off peak rate. Claude Haiku 5.5 and GPT-6 Luna are both $0.10 and $0.50 with $0.01 cache, which works out to 1.4x on input and 1.8x on output.

Both Hy4 numbers are the lowest on the table, and the gap holds on cache reads too, where only DeepSeek matches at $0.003. Good value on paper, assuming the quality holds up.09:09


r/LocalLLM • • 21m ago

Qwen4-27B reported to run on 16 GB of RAM

Post image
5 Upvotes

r/LocalLLM • • 32m ago

News Smaller Qwen4 models πŸ”₯

Post image
11 Upvotes

r/LocalLLM • • 32m ago

Question What am i doing wrong?

1 Upvotes

Hi

iam not into local llms but recently i started to play with it a little bit.

so installed lm studio on my pc (ryzen 5 5700, 32gb ddr4 3200, rtx 4060ti 16gb) and started by using qwen 3.5 9b model, it was slow so i started searching for a solution and found that the context length, quantization of the model and the kv cache, and mtp matters (maybe other things also i dont know, mb). then i tried the qwen 3.6 35b a3b on iq2-m from unsloth because it was said that its an moe model which brings faster performance , so after further search i knew that i can offload the experts in the model to the ram and keep a reserved cache space for experts on my gpu, and after otger tweaking and testing i reached that i can run full context the model supports but the speed is slow, too slow infact, if i try to give it a prompt using opencode for example it would take several minutes to respond, but if i just gave it the same prompt in lm studio chat its somewhat faster (i understand that opencode sends more prompts with the one i wrote), so i also tried filling the context on lm studio to see the speed, it was ~40 tokens per second at the start and by the time the context is 50-60% it started slowing down rapidly to 15 tokens per second, so i tried another model called mellum 2.1 from jetbrains, and this model is pretty fast with full context at nearly 100 tokens per second whatever the context filled. so my questions are

1-what did i do wrong or is the qwen model normally slow?

2-the mellum model is fast but need a lot of steering and fixing if i use a framework that is not used a lot, so how do i fix this?

3-if qwen is slow and this is the max it can perform, is there a fast smart model used for coding primarily fitting for my pc with full context?

and sorry for any english mistakes.


r/LocalLLM • • 1h ago

Research Optimal setup research: Muse Glimmer 30B on RTX 5090M/RTX Pro 4000 (24gb cards)

1 Upvotes

Notice:
The point of these research runs is to see, document and opensource the best configuration we can get with a certain model on hardware that "most of us" can get our hands on.
While we do have a bit of different hardware in our lab - we tend to focus around 5090M/RTX pro 4000 - because its a 24gb card with a quite fair performance - and its not insanely expensive.

Find anything you like?.. use it - got questions?.. shoot.. have an idea?.. let us know.

--

Hi there,

After our Qwen 3.8 Flash Next research (https://github.com/inordic/cmp170hx-vllm-qwen38-flash-next) and Gemma 4 31B (https://github.com/inordic/rtx5090-NInferno-gemma4-31b) - we wanted to to try out something new..

I ran som tests myself and thought "Muse Glimmer 30B" seem like a quite nice model.. it was fairly good at its job and quite nice to talk to.. so lets see where we can take it.

So.. this is where we are at.. full context.. 128K.. and quite nice and fast.. not as fast as Qwen and Gemma.. but fast enough.

Same GPU, same benchmarkllama.cpp f8def7f (Aug 2026), unsloth UD-Q4_K_XL, Meta DFlash, depth 15This repo: patched llama.cpp, MLP-NVFP4 hybrid, DFlash 2, depth 15
Prefill, 6,880-token prompt1,430 tok/s3,254 tok/s
Prefill, 100,554-token prompt1,246 tok/s2,367 tok/s
Chat decode, greedy / sampled80.6 / 74.2 tok/s110.3 / 105.4 tok/s
Agent turn with a cached 46K promptβ€”0.89 s
KLD vs Q8_0: agent / English / Swedish text0.033 / 0.012 / 0.0150.053 / 0.030 / 0.034

As allways.. this is research.. we might be wrong.. hell.. we might be doing it wrong.. but atleast .- we are doing it :)

Enjoy
//iNordic team

Head over to: https://github.com/inordic/rtx5090m-muse-glimmer-30b for more info.


r/LocalLLM • • 1h ago

News inferzero: a from-scratch LLM engine for Intel Arc. Qwen3.8 27B at 2,694 tok/s prompt on one Pro B70, the 35B at 4,264 tok/s over 4Γ— B60 and 851 tok/s on a laptop iGPU, and DeepSeek-V4.1-Flash split over several cards

Thumbnail
reddit.com
2 Upvotes

r/LocalLLM • • 1h ago

Discussion Struggling with coding on local models - setup help?

2 Upvotes

I'd tried out models like DS and QFN using APIs before and they seemed very capable. However, they all seemed to struggle heavily when using them local.

QFN processes really fast but seems to get stuck in endless "Wait..." and "Whatever." loops and burns through 100K tokens just being confused at my codebase.
DeepSeek is slower but seems more coherent, I haven't tested at full context but it had a similar issue with indecisiveness (even with guidance) and would think itself into compaction.

All of this over one .json ComfyUI workflow. Just a simple test I do where I tell it to rearrange nodes and change some inputs...
I've tried using Llama serve, Strata desktop, Pi. All lead to similar outcomes.

I have a feeling this is an issue with my setup rather than the models bc the API models seemed to do it just fine and (allegedly) quantization doesn't hit MOE models as hard.

Setup:
(both models from OrcaRouter). DSV4.1-1017 2-bit, Q3.8FN_IQ3XXS
Inference: Maya (DS), Strata/LlamaCPP (QFN)
Harness: default Pi
Context: QFN was to 256K, DS I only tried to 512K


r/LocalLLM • • 1h ago

Why download abliterated Flash Next models when LoRA exists?

Thumbnail
huggingface.co
0 Upvotes

r/LocalLLM • • 2h ago

Discussion Own harness or open source one ?

Post image
0 Upvotes

Hey LocalLLM community,

I am super curious how you run your models.

Are you using open source harnesses ?
Did you build your own ?

A bit of explanation for why would be nice.

There has gotta be a ton of people who did the same as me.
This is a local Qwen 3.8 27B / Flash-Next on heavily modified engines sharing one harness and user interface. Llama.cpp just recently merged f8_e4m3 cache, which I got Claude to optimize for my Blackwells. Due to the deliberate kneecap of Nvidia on certain accumulations, I decided to skip conversion to specific steps that would run on the half speed flag = FP8 full speed. The surgery worked and while we were in already fixed a binary that caused a regression in draft speed which took literal hours of testing to pin down.

End result, my private local backend runs 27B now up to 50% faster ( median 18% increase ).

Hardware specs at upper-mid third of HF compute richness. 128 GB DDR5 6000 1:1, Ryzen 9700X, RTX 5080 + 5070Ti .

Anyone running their own harness and wants to share a screen of it ? Id love to reason over the visual design and maybe learn a bit or two while at it.


r/LocalLLM • • 2h ago

News Qwen4 Smaller Models Open Weights Confirmed via Official Qwen X account.

Thumbnail gallery
3 Upvotes

r/LocalLLM • • 3h ago

Question Specs for Motion graphics and 3D

1 Upvotes

I’ve been offloading some of my work to AI to help speed things up at work. I use Claude Ultra but ai’m running out of weekly tokens mid week.

I want to build a PC that’ll help me continue working without having to wait for my credits to reset.

I work with a lot of 2D motions graphics and 3D modeling and animation. Not games related, mostly product related and sometimes jewelry related (building 3D models from my sketches). Would also like to experiment with image and video models.

What would be appropriate specs for my use case?


r/LocalLLM • • 3h ago

Discussion Stop Burning Tokens on File Searches: How QMD Helps Coding Agents Save Up to 55%

Post image
0 Upvotes

I noticed that, before carrying out a task, coding agents run find, rg --files, or glob to locate files and the classes and modules they need. Then they search file contents with rg or grep. This burns a lot of tokens. So I decided to see whether semantic search could save tokens compared with all of the above. If a local GPU is sitting idle, why not use it for semantic search?

I work at a company focused on security. In one of my projects, I use semantic search for sensitive documents. Then an acquaintance sent me a link to an open-source project that tackles the same problem. The project is called "QMD - Query Markup Documents." It runs from the command line, extracts text from documents in all formats, and creates an index. It supports several search modes:

# Search across everything
qmd search "project timeline"           # Fast keyword search
qmd vsearch "how to deploy"             # Semantic search
qmd query "quarterly planning process"  # Hybrid + reranking (best quality)

We're interested in the last option. It requires three things to work:

  • embeddinggemma-300M-Q8_0 Vector embeddings (default) ~300MB

  • qwen3-reranker-0.6b-q8_0 Re-ranking ~640MB

  • qmd-query-expansion-1.7B-q4_k_m Query expansion (fine-tuned) ~1.1GB

The first model is used for indexing to compute semantic vectors for text chunks. The second compares the search results with the query to determine how semantically similar they are. The third generates several search queries from the original query to broaden the search.

First, add your source files to QMD (this is an example for a minimal project with a Rust server and an Angular frontend):

qmd collection add . \
  --name PROJECT1 \
  --mask "src/**/*.rs,\
frontend/src/**/*.{ts,html,scss},\
README.md,\
frontend/README.md,\
Cargo.toml,\
frontend/package.json,\
frontend/angular.json,\
frontend/tsconfig*.json,\
frontend/proxy.conf.json,\
.gitignore,\
frontend/.gitignore,\
frontend/.prettierrc,\
frontend/.editorconfig"

You don't have to do this manually: show this example to your coding agent and tell it to do the same for the current repo. It will figure out which folders and file extensions to add to the index, what to exclude, and so on.

Now let's run the indexing (you'll need to rerun it periodically as your codebase changes):

qmd update && qmd embed -c PROJECT1

To make the coding agent work with this QMD setup, add the following to AGENTS.md:

# QMD - Efficient Code Discovery

**Goal:** minimize token costs and code search time without compromising correctness.

## Choosing a tool

- If the path is known β†’ read the file directly.
- If an identifier, error, or line is known β†’ use `rg` / LSP.
- If only the behavior is known β†’ use `qmd query`.
- Once the code is found β†’ inspect the necessary sections with standard tools.
- Once you have enough information β†’ do the task.

**QMD replaces the initial broad search; it does not add another step.**

## Semantic search

Use the project's existing collection.

"""
qmd query "DESCRIPTION" -c PROJECT1 -n 15 --files \
  | grep -E '\.(EXT1|EXT2|EXT3)$' \
  | cut -d, -f3- \
  | sed 's|^qmd://[^/]*/||' \
  | head -n 10
"""

- Use one specific query, preferably with English technical terms.
- Choose extensions that match the project, including configuration files when needed.
- Start by reading only the relevant sections of the 3–5 most relevant files.
- If you cannot find the implementation, refine the query; if necessary, increase `-n` to 60, then use `rg` / LSP.
- Stop semantic searching once you find the implementation.

## Saving tokens

- Do not scan the whole repository before using QMD.
- Do not repeat searches without a reason or open every result.
- Do not investigate unrelated code or dependencies.
- Do not write detailed search plans or reports.
- Do not paraphrase code you have read.
- As soon as you have enough information, make the changes and check the result.

**Priority: correctness β†’ token cost β†’ time.**

I tested it on a repository like this, with 1,231 source files across Java, Scala, and Angular:

C:\Users\igumn\temp2>qmd collection list
jcm-server (qmd://jcm-server/)
  Pattern:  server/src/**/*.scala,server/src/**/*.java,server/src/**/*.xml,server/src/**/*.properties,server/src/**/*.js,server/src/**/*.ts,server/src/**/*.css,server/src/**/*.html,server/src/**/*.tmpl,server/src/**/*.txt,server/src/**/*.md,server/pom.xml,server/Dockerfile,server/*.md,server/README.txt,web/src/**/*.ts,web/src/**/*.html,web/src/**/*.scss,web/src/**/*.css,web/package.json,web/angular.json,web/tsconfig*.json,web/karma.conf.js,web/README.md,!server/src/main/resources/templates/**,!server/src/main/resources/messages*,!server/src/main/resources/messages*/**,!web/src/assets/lang/**
  Files:    1231
  Updated:  1h ago

Prompt: "Add a post-quantum encryption flag to the disk encryption policy. It should be disabled by default, configurable in the web interface when editing the policy, saved to the database, and sent to the agent as part of the encryption policy via the REST endpoint."

Cost Comparison

Using GPT-6 Luna API rates: input $0.10, cached input $0.01, and output $0.50 per 1 million tokens.

MetricWith QMDWithout QMDSavings
Input65,006101,50135.96%
Cached input317,5681,191,80873.35%
Output13,85916,34815.23%
Reasoning9,60711,26014.68%
Cost$0.0166$0.030245.09%

Now QMD saves 45.09% in costs!

The reduction in cached input is especially noticeable: nearly 4x.

I ran these tests with GPT-6 Luna. In the first test, the agent's runtime dropped from 6 minutes to 4 minutes.

Prompt: "Add a dry field to the erasure policy. It should be editable in the web interface and saved to the database, and it should also be sent through the /rest endpoint to the agent responsible for erasure. Its default value should be false."

Calculation using the same GPT-6 Luna rates: input $0.10, cached input $0.01, and output $0.50 per million tokens.

Comparison

MetricQMDWithout QMDDifference
Input77,16798,042βˆ’21.29%
Cached input936,832854,912+9.58%
Output17,32216,259+6.54%
Reasoning8,83510,264βˆ’13.92%
Cost$0.0257$0.0265βˆ’2.78%

Here, QMD turned out to be 2.78% cheaper. There are savings, but they are small.

Conclusion: QMD reduced regular input tokens by 21.3% and reasoning by 13.9%, but the increase in cached input and output almost completely wiped out the savings.

Overall, the savings range from 2.78% to 45.09% of the cost. I think that's a decent result. Of course, we need a larger sample. But I'm ready for bed. Let me know in the comments if you've run a larger sample and what results you got.

PS

I recommend checking that junk isn't getting into your index. Try running a semantic query yourself. I once overlooked compiled .js files and large lang files-they heavily polluted the semantic search results. Here's an example of how to run a search manually:

C:\Users\igumn\jcm-server>qmd query "disk encryption policy editing form persisted setting sent to endpoint for agent" -c jcm-server -n 15 --files
Expanding query... (3.0s)
β”œβ”€ disk encryption policy editing form persisted setting sent to endpoint for agent
β”œβ”€ lex: disk encryption configuration
β”œβ”€ lex: disk encryption policy
β”œβ”€ vec: disk encryption configuration
β”œβ”€ vec: disk encryption policy
└─ hyde: The topic of disk encryption policy editing form persisted setting se...
Searching 6 queries...
Embedding 4 queries... (1.5s)
Reranking 40 chunks... (3.3s)
#d70022,1.00,qmd://jcm-server/server/src/main/java/com/jetico/jcm/server/domain/common/bcvolume/EncryptPolicy.scala
#732b09,0.63,qmd://jcm-server/server/src/main/java/com/jetico/jcm/server/service/ajax/common/EncryptSettingsManager.scala
#e66494,0.63,qmd://jcm-server/web/src/app/modules/management/components/company/dialogs/add-or-update-policies/add-or-update-policies.component.html
#413c5e,0.62,qmd://jcm-server/web/src/app/modules/management/components/group/dialogs/change-policie/change-policies.component.html
#413c5e,0.62,qmd://jcm-server/web/src/app/modules/management/components/computer/dialogs/change-policie/change-policies.component.html
#758b4d,0.62,qmd://jcm-server/server/src/main/java/com/jetico/jcm/server/service/mcp/McpFacade.scala
#4e8973,0.62,qmd://jcm-server/web/src/app/modules/management/components/group/dialogs/change-policie/change-policies.component.ts
#bef78e,0.62,qmd://jcm-server/web/src/app/modules/management/components/computer/dialogs/change-policie/change-policies.component.ts
#4fcd1c,0.61,qmd://jcm-server/server/src/main/java/com/jetico/jcm/server/service/mcp/McpToolbox.scala

PPS

I tried it on a small project with 27 files. Instead, the cost went up by 60%. The solution turned out to be overkill. In general, use it only on large codebases.

UPDATE

I went ahead and built a hybrid QMD + GitNexus solution after all. GitNexus performs structural code analysis (AST, dependency graph) and lets you search based on it. Here are the results:

Cost comparison

MetricWithout QMDQMDQMD + GitNexus
Input101,50165,00659,122
Cached input1,191,808317,568225,792
Output16,34813,85910,894
Reasoning (included in Output)11,2609,6078,465
Cost$0.03024$0.01661$0.01362
Savings-45.1%55.0%

The savings figure rose from 45% to 55%! The coding agent's runtime increased from 6 minutes to 7 minutes. Here are the revised instructions for AGENTS.md:

# QMD + GitNexus - Efficient Code Discovery

**Goal:** find the required code and its relationships with minimal token usage, without compromising correctness.

## Choosing a tool

- If the file path is known β†’ read it directly.
- If the exact identifier, error, or line is known β†’ use `rg` / LSP.
- If only the behavior is known and the implementation location is unknown β†’ use QMD semantic search.
- If you need to understand symbol relationships, the execution path, or the impact of a change β†’ use GitNexus structural analysis.
- Once the code is found β†’ inspect the necessary sections with standard tools.

QMD and GitNexus replace the initial broad search; they do not add mandatory steps. Use only the tools needed for the task.

## QMD semantic search

Use the project's existing collection:

    qmd query "DESCRIPTION" -c PROJECT1 -n 15 --files \
      | grep -E '\.(EXT1|EXT2|EXT3)$' \
      | cut -d, -f3- \
      | sed 's|^qmd://[^/]*/||' \
      | head -n 10

- Formulate one specific query, preferably using English technical terms.
- Specify suitable source and configuration file extensions.
- First read the 3–5 most relevant files, and only the relevant sections.
- If the implementation is not found, refine the query; if necessary, increase `-n` to 60, then use `rg` / LSP.
- Stop semantic search once the implementation is found.

## GitNexus structural analysis

Use GitNexus through the CLI when you need to investigate code structure:

- `context` - relationships and context for a known symbol.
- `impact` - possible impact of changing a symbol.
- `trace` - path between known symbols.

Examples:

    gitnexus context SymbolName
    gitnexus impact SymbolName
    gitnexus trace SourceSymbol TargetSymbol

If a symbol is unknown, find it first through QMD, `rg`, or LSP. Use GitNexus only when its structural analysis will help answer the question; do not run every command in sequence.

## Token efficiency

- Do not scan the entire repository before using QMD.
- Do not repeat searches without a reason or open every result.
- Do not investigate unrelated code or dependencies.
- Do not write detailed plans and reports about the search.
- Do not summarize code you have read.
- Once you have enough information, make the changes and verify the result.

**Priority: correctness β†’ token cost β†’ time.**

Create a .gitnexusignore file to keep junk out of the index so that it contains only source files. Then start GitNexus indexing with this command:

gitnexus analyze --index-only

r/LocalLLM • • 3h ago

Research I gave a sandboxed local model a one-line Apache patch. It built the working RCE.

Thumbnail
github.com
1 Upvotes

r/LocalLLM • • 4h ago

Question Dual RTX Titan for local LLM

3 Upvotes

I currently have a 3080 ti that I don’t really game with anymore, but have the opportunity to get 2 RTX titans at a significant discount from a friend ($1300 total). I was thinking about selling my 3080 ti and jumping on that deal for the combined 48gb of vram to pair with 64gb of system ram and a 13900k.

Would doing this make any sense if I want to run some local models for project code, personal learning, and some small business workflows (paired with Gemini pro for complex stuff)? I know the RTX titans have limited memory bandwidth but the price for 48gb of vram is hard to beat. Any opinions or recommendations? Thanks!


r/LocalLLM • • 4h ago

Discussion RTX 3080 20GB ReBAR on AMD Windows platforms without BIOS flashing

2 Upvotes

Some of you may have seen the post from u/favabean28 with enabling ReBAR on modded 3080 20GB GPUs around a week ago with the repo NV_30_DXE. For Windows, this is (or was) seemingly the only way to get ReBAR working on the 3080 20GB. For those that don't know, previously the RTX 3080 20GB did not have ReBAR working as there was no VBIOS that supported the extra VRAM, and even flashing a modified VBIOS wouldn't work. The method for enabling it properly on modded GPUs was recently discovered about a month ago for Linux, and now for Windows.

Unfortunately it turns out that for AMD, many newer AMD motherboards, especially AM5, are incompatible: All AMD motherboards with AGESA PI-1.2.0.0 or newer have checks preventing modded BIOS firmware from being flashed. Most new AMD motherboards either don't have a BIOS version that old or aren't compatible with the installed CPU on BIOS versions that old, so I wanted to look into the possibility of booting off a USB drive with EFI to enable ReBAR on AMD platforms. This idea mainly came from how the current FP32 accumulation unlock works on Windows. Ultimately, this does work albeit with a few tradeoffs described in the repo below and at the bottom of the post.

Repo with instructions on installing and with the technical write-up: https://github.com/Articom123/3080-Unlock-ReBAR-USB

Installing does not require building from source. However, I did include the FP32 accumulation unlock with it though (since there's no reason not to pair them together), which means it will automatically download and extract Nvidia firmware when you run it, specifically the Nvidia and Arch Linux website for the kernel and utils 610.43.03.

Some quick results:

  • GPU-Z reports ReBAR as enabled with a 32768 MB BAR1 size

  • Device Manager shows an entry for Large Memory Range 0x2000000000 - 0x27FFFFFFFF (this does not show up without ReBAR enabled)

  • The FP32 accumulation unlock works alongside ReBAR. This generally results in around a 20% increase to prefill speed for LLMs based off TTFT, though it's not the primary focus of the repo (it's just a no-brainer to include it since it makes Ampere cards infinitely better)

  • CPU has access and can map the full ~18-19GB VRAM on the GPU

I want to also briefly give a lot of credit to physicsdolphin (nv-uefi-unlocker) of which this tool uses for the FP32 accumulation unlock and install, the creator of Windows-GPU-UNLOCK (Windows-GPU-UNLOCK) who made similar bootable EFI tools to allow for running the tensor unlock on Windows, and u/favabean28 (NV_30_DXE) who figured out the path for enabling ReBAR on Windows in the first place.

While the repo goes more in depth on compatibility and what does/doesn't work I wanted to still mention some important things about it:

  • This is mainly meant for AM5 platforms. It seems like Intel motherboards don't have the same restrictions when it comes to flashing modified BIOS firmware, so if you have Intel you might be more interested in the NV_30_DXE repo instead.

  • Enabling ReBAR isn't really all that useful on Windows without having P2P; however, the modded P2P drivers are still Linux only and probably will be for a very long time, and multi-GPU support on Windows isn't really that good anyways. Therefore, for practicality, the tensor unlock is more important in which case you would probably be more interested in the other Windows unlock tools I mentioned earlier. There is some limited usefulness in games though I guess.

  • This only works for the 3080 20GB. Technically the 4080 32GB would work, but since I don't have the card, it hasn't been supported in the repo and would require modifications to the source (mainly to the PCI vendor and subsystem ID check). The tensor unlock included is also for Ampere and not Ada.

  • u/favabean28 did work to try and make the tool as universal as possible, however, I don't have any other AMD motherboards to test this tool on and so I decided to ultimately leave it as a "proof-of-concept". It is almost certainly possible that this can be adapted further to other AMD setups with the 3080 20GB as pretty much all of the hard work is done now though.

  • There is likely zero possibility that this can cause any damage to Windows or the GPU, as there is no flashing of the VBIOS/BIOS or modification of the OS at all. The tensor unlock and ReBAR don't survive any reboots and the USB has to be manually booted again after any shutdown or restart.

  • Sleep and hibernate specifically do not work when enabling ReBAR with this method, and Secure Boot cannot be used. In comparison modding the BIOS with NV_DXE_30 does not have these limitations if applicable to your setup.

  • This is mainly made for Windows. If you're on Linux there are far better tools for doing this and you can use P2P.

I still thought it was worth posting this for anyone who is interested in this sort of stuff for Windows since Windows kinda sucks and it's at least somewhat cool to have Windows get something for once. Ultimately getting the P2P driver to work on Windows would be ideal but I'm not sure how it could be done for now.


r/LocalLLM • • 4h ago

Discussion Strix Halo 128GB local coding setup: some real test results, 23–45 tok/s, and what the harness changed

Thumbnail reddit.com
2 Upvotes

r/LocalLLM • • 4h ago

Discussion how to remove synthid from videos?

0 Upvotes

wondering if anyone has any pointers in how you would do this. images are pretty easy, but with videos you have many frames so can't just do each frame (i mean you can, but its not feasible to spend 60mins of GPU time on one video).


r/LocalLLM • • 5h ago

Question What local AI model is suited for coding in Unreal or in Redot/Godot?

4 Upvotes

Basically the title


r/LocalLLM • • 5h ago

Discussion A developer tried to build a dictation app with AI. Three weeks later he had three libraries and no app.

0 Upvotes

A developer didn't like the dictation apps available for his Mac, so he decided to build one with AI agents doing most of the coding.

The speech engine needed work, so he built an offline one called wavo. A dependency was slowing down builds, so he wrote his own FFT. The GUI options pulled in too much, so he built a renderer called nimbo.

Three weeks later: three libraries, still no dictation app.

The libraries might be useful on their own. wavo supports Whisper, Parakeet and GigaAM. But the bit I liked was his caveat: AI writing the code doesn't remove the work of reading it, checking it and testing it across systems.

At what point does "I'll just build it myself" stop saving time? What's your worst detour from the thing you actually meant to ship?

More of the development story is on my profile.


r/LocalLLM • • 5h ago

Question What if?

0 Upvotes

I have an idea what if we had an ai app that pooled everyone who downloaded the app ram/ processor/ vram etc together and made it into a big data center web since there wouldn’t be big factories size data centers we would save the water cause most gpus can be cooled with good air ventilation and just fans plus if you have a million of them spread out between a million houses it’s not even necessary consuming all that power as it would be pooling together all the resources it need to run. You know this is just an idea so I’m open to other but I think if we solve the problem of ai taking our water civilization could advance even further ?


r/LocalLLM • • 6h ago

Question How would you best utilize my setup?

Post image
2 Upvotes

Hello all! I've been running a home lab for a really long time, but I've finally come across a couple of RTX 3060 12GB cards and I want to get a useable LLM in for me and my wife to use.

My server has an Intel i7-9700K, and 128GB of DDR4 RAM, but I would only allocate 64GB to the container in Proxmox. I was wondering how you would recommend setting up a fairly reasonable model (I was thinking Qwen 3.8 27B Q4K) and splitting it between the two graphics cards (since there's no SLI). I want to try to get at least 128K context. 256K would be ideal, if I wanted to run something like Hermes, or other agents.

Is llama.cpp the way to go? It's pretty much all I've read about due to the customization possibilities, and I've only read to stay away from ollama. What would you recommend I do?


r/LocalLLM • • 6h ago

News Qwen officially announces Qwen4-Max and Qwen4 smaller models!

Post image
243 Upvotes

r/LocalLLM • • 6h ago

Question Anyone here running a super large model like Kimi k3?

2 Upvotes

If yes: how?


r/LocalLLM • • 6h ago

News Jensen Fought for Open Modelsβ€”Now Open Source Fights for Him

Thumbnail
youtube.com
0 Upvotes

r/LocalLLM • • 7h ago

Discussion What are all the current Strata-likes/forks going around? (e.g. ways to run larger models on lower VRAM setups)

1 Upvotes

I'm wondering if we could get a list going with all of the low VRAM workarounds to run stuff like Qwen Flash Next. Specifically, I would also like to clearly define what their use cases are. For example, I know that Basalt is a Strata fork that is made for 50 series cards.