Here is how I made a website builder using the viral REA reverse engineer repo.

llama.cpp merged PR #29887 two weeks ago and I think it is the most interesting thing to land in local inference this year, so I went through the PR, the benchmarks and the review thread and wrote up how it actually works. Short version below, long version linked at the end.

The problem

Qwen3.8-Flash-Next at Q4_0 is 93.7 GiB, and 65.4 GiB of that is experts. It is 125B total but only 6B active: 512 experts per layer, the router picks 10 (plus 1 shared) per token. One expert is 3 x 2560 x 640 = 4.9M params = about 2.6 MiB at 4-bit. 10 experts x 48 layers = 480 experts per token = about 1.3 GB of weights touched per token.

The observation

Expert routing has locality of reference. Token N+1 mostly picks the same experts as token N. A handful of experts are hot, most are almost never touched. That is the same property your CPU cache, your OS page cache, your browser cache, every CDN and Redis allkeys-lru all rely on.

The fix

PR #29887 (by am17an, a port of Tether’s qvac-fabric work, merged by ggerganov on Oct 7) sets aside a slice of VRAM as an LRU cache of experts. Hit: compute from VRAM, zero bytes moved. Miss: copy that one 2.6 MiB expert over PCIe and evict the least recently used one. The log prints the hit rate at exit.

Numbers from the PR description, same model file, same machine:

The 5090 gains more for two reasons: PCIe 5.0 halves the cost of every miss, and 32 GB means a bigger cache and fewer misses.

How to use it

llama-server -m Qwen3.8-Flash-Next-Q4_0.gguf -fa on -c 32768 -b 2048 -ub 2048 -cmoe --moe-cache-mib 11000

The PR author’s rule of thumb is about 10% of your expert bytes (65.4 GiB -> 6544 MiB). Env var form is LLAMA_ARG_MOE_CACHE_MIB.

SGLang merged the same idea 12 days later as “paged experts” in PR #42557

The part I find most interesting is LRU , Locality of reference go back to 1960s.

And i find locality of reference principle highly underrated.

Full write-up with the memory math script, an expert cache simulator you can run, and the benchmark table: https://cloudmash.blog/posts/moe-expert-offload-lru-cache-consumer-gpu/

llama.cpp PR #29887, MoE expert cache: https://github.com/ggml-org/llama.cpp/pull/29887

SGLang PR #42557, paged experts: https://github.com/sgl-project/sglang/pull/42557

Hi everyone,

I completed my BSc in Computer Science and currently have a 2-year career gap. My goal is to become a Full Stack Developer and get into the software industry.

I'm confused about what would be the better choice:

- Should I pursue an MCA?

- Or should I spend the next few months building strong full-stack skills, creating projects, and applying for jobs?

I'm worried that my 2-year gap might affect my chances of getting hired.

Has anyone been in a similar situation? If yes, what did you do, and what would you recommend?

I'd really appreciate advice from recruiters, developers, or anyone who has gone through a similar journey. Thanks!

Hii ,I am thinking about to take chai code cohort by hitesh Choudhary, I want to know about it . Is it good for beginner ,I am just know javascript pls give your pov on it

I’m making a free TypeScript course for people coming from JS and had to make a call early: strict on or off for the first lessons.

I went with strict on. The main reason is that the most useful catch in my intro lesson only exists with it. users.find(...) followed by found.name. Without strictNullChecks the undefined just gets swallowed by the type and TS says nothing, which is exactly the crash beginners hit in real code. Showing TS without strict basically teaches them it protects more than it does.

Same with noImplicitAny. A function like addTax(price) that gets “100” from a form returns 10020 because the plus concatenates. With strict, the untyped parameter is already an error, so beginners are pushed to add the annotation before they even know why.

The cost is real though. More red lines on day one, and some of those error messages are rough for someone who just learned what a type is.

The other thing I hammer early is that types are erased. Even now that Node can run .ts files directly, it just strips the types, it doesn’t check them. So anything coming from a server or a form still needs runtime validation, and I think a lot of people learn that way too late.

If you’ve onboarded juniors to TS, did you start strict or ease them in?

It’s crazy to think that the entire modern AI revolution started with researchers just trying to mathematically mimic a single biological neuron. When you zoom out, the timeline of AI architectures looks exactly like an evolutionary tree.

We went from struggling to solve basic logic puzzles (shoutout to the XOR problem) to building early neural networks, realizing they forgot things too quickly, battling the vanishing gradient problem, inventing LSTMs to give models actual memory, and eventually landing on Transformers with attention, the architecture that essentially ate the world. Every major roadblock forced a mutation that led to a massive paradigm shift.

But looking at this evolutionary timeline brings up a massive question: If Transformers are just the current "apex predator" of AI architecture, what exactly is the next evolutionary leap?

Are we going to see a completely new paradigm that makes self attention look as primitive as a basic AND gate? How do we even begin to perceive or predict what lies beyond our current models when the jumps are this exponential?

I went down a massive rabbit hole mapping out this entire evolutionary timeline. I wanted to see the exact breadcrumbs that led us here, so I put together a deep dive covering the journey from the first perceptrons to the modern era, complete with the foundational paper citations and the historical tweets that marked these shifts in real-time.

If you want to look at the complete map of how we got here to help predict where we are going, I compiled the whole history here: https://cloudmash.blog/posts/evolution-of-ai-perceptron-to-text-to-video/

Would love to hear your theories on what does the post Transformer era actually look like ? Like what can we imagine , how this intelligence on our command can do for us , how enterprise and big players can use it , what can happen ?

Hey Folks, I am looking to start my web design business where i will use no-code/low-code platforms and one of the platform that keeps coming on my feed is ODOO

But all the odoo websites i have seen so far are like 2015 Wordpress. So has anyone of you tried it and sold it to a real client?

Also how would you improve ODOO website builders websites?

Thanks

Right now, the AI space feels entirely focused on massive datacenter clusters and renting H100s by the hour. But after spending way too much time looking at the actual footprint of these models, I realized that 90% of use cases are completely over engineered.

You don’t always need a multi GPU setup. The AI ecosystem is actually a massive spectrum.

I recently sat down and mapped out the exact tiers of AI models based on their size, the hardware needed to run them, and the point of diminishing returns.

Here are the two extremes and the sweet spot in the middle:

• The 100KB Extreme (TinyML) (Tensorflow Lite , sensor anamoly detection models): We are talking models that run on microcontrollers drawing single-digit milliwatts. They run on kilohertz processors using ultra-quantized integer math. You can run basic sensor anomaly detection or wake-word detection on a device powered by a coin cell battery.

• The Local Sweet Spot (4GB to 40GB) (Mistral 7B, Gemma 2 9B/27B, Qwen 2.5 14B/32B): This is where the magic happens for most devs right now. You can run highly capable 7B to 35B parameter models (like Llama 3 or Qwen) at 4-bit quantization on a standard Mac or a consumer GPU (like an RTX 3060 or 4090). It’s perfect for local RAG, coding assistance, and uncensored chat. VRAM is your only real bottleneck here.

• The 2.5TB Behemoths (Deepseek, Llama , Kimi k3): State of the art massive Mixture of Experts (MoE) routing. To even load these, you need dedicated power infrastructure and server racks of specialized accelerators drawing thousands of watts.

The missing piece: Figuring out the exact math for your hardware

The hardest part about building right now is looking at a model on Hugging Face and trying to calculate exactly how much VRAM you need, what quantization to use, and whether your CPU/GPU will choke on the context window.

So, I wrote a complete deep dive breaking down the math for all tiers of the AI spectrum.

If you want to see the architectural differences at each scale, and a cheat sheet for matching the right model size to your specific hardware, I put the full breakdown on my blog here:

https://cloudmash.blog/posts/ai-model-size-memory-hardware-guide/

Let me know what you guys think especially if you've found any ultra efficient small models/technique that punch above their weight on consumer hardware. And also I would love to hear whether quantization have resulted in major difference in quality , like if anyone have that kind of experience in that.

EDIT: Hello everyone, taking advice from your comments, I have added new model examples and kept old ones as they fit in the already assigned tiers, for reference , year of launch is added alongside the model.

I have written a blog regarding evolution of AI from simple perceptron to today's transformers who can generate a Video.

Along the chronological path I have also embedded relevant research papers and tweets regarding the respective AI tech. Whole blog is very very fun to read and I am sure you will like it.

https://cloudmash.blog/posts/evolution-of-ai-perceptron-to-text-to-video/