r/LocalLLM • u/a-pyctam • 2d ago
Question Which inference engine squeezes max performance out of my setup without sacrificing precision? (Coding + image gen)
I'm trying to figure out the best inference stack for my hardware and use cases. Here's what I've got:
2x
RTX 9700R97001x RTX 5070 Ti
AMD EPYC 7302
128GB RAM
Linux
My main focus is coding-related LLM work (code completion, review, refactoring) with potential image generation down the road. I want to get as much throughput and speed as possible out of these three GPUs without resorting to heavy quantization.
I'm particularly interested in using all three GPUs effectively. I know about Vulkan multi-GPU support, but I'm also curious about a two-phase approach where the 5070 Ti handles orchestration and delegation while the dual 9700s do the heavy lifting for actual inference. Is that a realistic setup?
What's everyone running for this kind of configuration? I'm weighing options like llama.cpp, radiance, or maybe something else entirely. Any advice on how to split workloads across the cards would be great too.
2
u/Strong_Technician626 2d ago
No easy way to pool AMD and Nvidia cards into one model. Put the coding model on the R9700s with llama.cpp and put ComfyUI or a small completion model on the 5070 Ti. LiteLLM running on the cpu will handle routing, no need for a dedicated orchestration gpu. I'd push back on quant, a bigger model at Q5 usually performs better than a smaller one at Q8. Below Q4 is where you really see a difference.
1
u/a-pyctam 2d ago
This is something I might try out. Now I need to get a starting point to explore how to "handle routing, no need for a dedicated orchestration gpu". Is LiteLLM about this https://docs.litellm.ai/docs/proxy/docker_quick_start ?
1
u/Strong_Technician626 2d ago
Yeah thats it. Go to the "Running without a database" section near the bottom of that page, you can skip the Postgres/UI stuff for now. Run one llama-server for the R9700s and one for the 5070Ti, each on its own port. In the config, add each server with an openai/ model name and api_base set to its url. LiteLLM routes by model name. Reply here if you get stuck, happy to help
1
u/Independent_Rock347 2d ago
With a 50 series card I believe you can definitely start using basalt along with your 9700s for coding tasks. I'm not really sure about image generation though since that's not really what I look for.
1
u/a-pyctam 2d ago
This https://github.com/jesdga95/basalt ? And, 5070ti is the smallest in my stack.
1
u/Independent_Rock347 2d ago
That is correct, it pretty much has the optimizations for a 50 series cards and all other remaining experts can live in your other VRAM
1
u/a-pyctam 2d ago
How basalt will talk to/deal with other, remaining, experts on dual R9700s?
1
u/Independent_Rock347 2d ago
I'm honestly pretty lazy and just let my $20 claude sub set it up for me, once it's done it's pretty much good to go. I saw this video as a reference and was interested in it and it actually performed fairly well https://www.youtube.com/watch?v=wC6jsOU4jII&t=1305s
1
1
u/Necessary-Orange6903 2d ago
it seems like as the days go by the correct answer is starting to be to get Opus 5.5 to write you a custom engine for your own hardware lmao
1
u/a-pyctam 2d ago
Yeah, sounds like other than Vulkan there is no way to run all three of them together
1
u/Necessary-Orange6903 2d ago
True but also you might not need to - depending on your use case, strata is gonna load most of the relevant experts to VRAM anyways - I have like 95% of my relevant experts loaded in vram most of the time, so adding a third (was thinking of adding an RDNA3 card) wouldn't benefit me, probs same with you
1
u/Necessary-Orange6903 2d ago
also on your stack, try strata for RDNA4, use the unsloth Q6_K_XL model as a starting point and see how far you get - it should be pretty good. I get about 40-60t/s decode and my prefill is at around 300 ish on two radeon pro v620s which are RDNA2 and lack alot of FP instructions whereas strata is very well optimized for RDNA4 so you could get much better numbers
1
u/OurManInHavana 2d ago
"without resorting to heavy quantization" is an issue. You can tune towards performance, or quality: but you don't get both without spending a lot of money :) . I'd lean towards slower/larger models so you don't waste time with subpar answers.
I'd run a Strata/variant of Qwen 3.8 Flash Next on your dual 9700's (UD-IQ4_XS? IQ3_S?) and give your 5070 Ti to your image model. Load something like Qwen Image 2.1 or MiniMax H3 Image into ComfyUI.
2
u/Late-Coast6973 2d ago
The catch is that the two 9700s are AMD and the 5070 Ti is NVIDIA, so anything built on CUDA or ROCm tensor parallel can only use one side. llama.cpp with the Vulkan backend can split one model across all three by layers. I'd run the coding model on the 9700s and give the 5070 Ti its own job, like image gen.