I read The bitter lesson by Rich Sutton recently which talks about it.

Summary:

Rich Sutton’s essay The Bitter Lesson explains that over 70 years of AI research, methods that leverage massive computation have consistently outperformed approaches relying on human-designed knowledge. This is largely due to the exponential decrease in computation costs, enabling scalable techniques like search and learning to dominate. While embedding human knowledge into AI can yield short-term success, it often leads to methods that plateau and become obstacles to progress. Historical examples, including chess, Go, speech recognition, and computer vision, demonstrate how general-purpose, computation-driven methods have surpassed handcrafted systems. Sutton argues that AI development should focus on scalable techniques that allow systems to discover and learn independently, rather than encoding human knowledge directly. This “bitter lesson” challenges deeply held beliefs about modeling intelligence but highlights the necessity of embracing scalable, computation-driven approaches for long-term success.

Read: https://www.cs.utexas.edu/~eunsol/courses/data/bitter_lesson.pdf

What do we think about this? It is super interesting.

I've been getting a bit more into evolutionary algorithms again, so I was testing some loss landscapes where evolution beats vanilla gradient descent (while also trying to make some cool visuals).

Round 1, rugged hillside: gradient descent gets stuck in a dip, and evolution reaches the bottom after 750 evaluations.

Round 2, smooth slope: gradient descent wins, 108 steps against 570 evaluations.

Round 3, flat plateau: the slope is zero, so gradient descent never moves, and evolution reaches the bottom after 840 evaluations.

Edit:
"evolution" here means truncation selection (keep best 30 of 120) plus Gaussian mutation, no crossover.

I’ve spent the last half of 2025 in interview hell. I walked into my first few rounds prepared for deep math proofs, Transformer internals, and heavy LeetCode, but almost none of that came up. 

What they asked was way more practical, and I failed the first three rounds because I was over-preparing for the wrong things. Recruiters don't want a lecture on attention mechanisms anymore, they want to hear about your decisions.

Whenever I walked through a project, the questions were always: "Why RAG instead of fine-tuning for this?" or "How did you actually evaluate the hallucinations?" I failed early on because I’d just say, "I built a PDF chat app." Now, I lead with the trade-offs. 

I explain that I chose RAG because fine-tuning was too expensive for the dataset, used MiniLM for speed, and implemented a semantic chunking strategy that dropped the hallucination rate by 40%. That shift in how I talked about my work changed everything.

Another huge factor is cost and latency. I got my best offer because I could explain exactly how I cut inference costs by 60% using a hybrid local/cloud setup with Phi-3.5-mini and aggressive request caching. 

Companies want to know you aren't just burning GPU credits for fun. During live coding, they usually just had me "build a simple retriever" or fix a hallucination. I used to code in silence and fail; now, I narrate the whole time. 

If I’m using a FAISS flat index, I explain it’s for a small dataset but mention I’d pivot to HNSW for speed if we hit a million vectors. They don't want perfect code, they want to hear you architecting out loud.

The next time you’re in a technical round, don't just describe what you built. Describe why you didn't build it the other way. Showing that you weighed the cost of tokens against the accuracy of the model is exactly what separates a hobbyist from a senior engineer.

deepseek has done something world changing. it's really not about them as a company. nor is it about their being based in china.

deepseek showed the world that, through reinforcement learning and several other algorithmic breakthroughs, a powerful reasoning ai can be distilled from a base model using a fraction of the gpus, and at a fraction of the cost, of ais built by openai, meta, google and the other ai giants.

but that's just part of what they did. the other equally important part is that they open sourced r1. they gave it away as an amazing and wonderful gift to our world!

google has 180,000 employees. open source has over a million engineers and programmers, many of them who will now pivot to distilling new open source models from r1. don't underestimate how quickly they will move in this brand new paradigm.

deepseek built r1 in 2 months. so our world shouldn't be surprised if very soon new open source frontier ais are launched every month. we shouldn't be surprised if soon after that new open source frontier ais are launched every week. that's the power of more and more advanced algorithms and distillation.

we should expect an explosion of breakthroughs in reinforcement learning, distillation, and other algorithms that will move us closer to agi with a minimum of data, a minimum of compute, and a minimum of energy expenditure. that's great for fighting global warming. that's great for creating a better world for everyone.

deepseek has also shifted our 2025 agentic revolution into overdrive. don't be surprised if open source ai developers now begin building frontier artificial narrow superintelligent, (ansi) models designed to powerfully outperform humans in specific narrow domains like law, accounting, financial analysis, marketing, and many other knowledge worker professions.

don't be surprised if through these open source ansi agents we arrive at the collective equivalent of agi much sooner than any of us would have expected. perhaps before the end of the year.

that's how big deepseek's gift to our world is!

for those afraid of using a chinese ai or want to more easily build more powerful ais based on deepseek's r1:

"The release of DeepSeek-R1 is an amazing boon for the community, but they didn’t release everything—although the model weights are open, the datasets and code used to train the model are not.

The goal of Open-R1 is to build these last missing pieces so that the whole research and industry community can build similar or better models using these recipes and datasets. And by doing this in the open, everybody in the community can contribute!.

As shown in the figure below, here’s our plan of attack:

Step 1: Replicate the R1-Distill models by distilling a high-quality reasoning dataset from DeepSeek-R1.

Step 2: Replicate the pure RL pipeline that DeepSeek used to create R1-Zero. This will involve curating new, large-scale datasets for math, reasoning, and code.

Step 3: Show we can go from base model → SFT → RL via multi-stage training.

The synthetic datasets will allow everybody to fine-tune existing or new LLMs into reasoning models by simply fine-tuning on them. The training recipes involving RL will serve as a starting point for anybody to build similar models from scratch and will allow researchers to build even more advanced methods on top."

https://huggingface.co/blog/open-r1?utm_source=tldrai#what-is-deepseek-r1