From the Broadcom Q4 2025 Earnings Call. I think the $10bn order was reported on previously, but without the buyer being named.

[CEO Hock Tan] The scale at which we see this happening could be significant. As you are aware, last quarter, Q3 2025, we received a $10 billion order to sell the latest TPU ironwood racks to Anthropic. This was our fourth custom. That we mentioned. In this quarter Q4, we received an additional $11 billion order from this same customer for delivery in late 2026. But that does not mean our other two customers are using TPUs. In fact, they prefer to control their own destiny by continuing to drive their multiyear journey to create their own custom AI accelerators or XPU RECs as we call them.

Abstract:

We introduce Evolution Guided General Optimization via Low-rank Learning (EGGROLL), an evolution strategies (ES) algorithm designed to scale backprop-free optimization to large population sizes for modern large neural network architectures with billions of parameters. ES is a set of powerful blackbox optimisation methods that can handle non-differentiable or noisy objectives with excellent scaling potential through parallelisation.

Naïve ES becomes prohibitively expensive at scale due to the computational and memory costs associated with generating matrix perturbations $E\in\mathbb{R}^{m\times n}$ and the batched matrix multiplications needed to compute per-member forward passes.

EGGROLL overcomes these bottlenecks by generating random matrices $A\in\mathbb{R}^{m\times r}$, $B\in\mathbb{R}^{n\times r}$ with $r\ll min(m,n)$ to form a low-rank matrix perturbation $AB^{\top}$ that are used in place of the full-rank perturbation E. As the overall update is an average across a population of N workers, this still results in a high-rank update but with significant memory and computation savings, reducing the auxiliary storage from mn to $r(m+n)$ per layer and the cost of a forward pass from $\mathcal{O}(mn)$ to $\mathcal{O}(r(m+n))$ when compared to full-rank ES.

EGGROLL's efficiency results in a hundredfold increase in training throughput for billion-parameter models at large population sizes, nearly reaching the throughput of pure batch inference. A theoretical analysis reveals our low-rank update converges to the full-rank update at a fast $\mathcal{O}(\frac{1}{r})$ rate. Our experiments show that:

• (1) EGGROLL does not compromise the performance of ES in tabula-rasa RL settings, despite being faster,

• (2) it is competitive with GRPO as a technique for improving LLM reasoning, and

• (3) EGGROLL enables stable pre-training of nonlinear recurrent language models that operate purely in integer datatypes.

Layman's Explanation:

Most modern artificial intelligence is trained using a method called backpropagation, which requires complex calculus and expensive computer memory to calculate exactly how every parameter in the network should change to reduce errors. An alternative approach called Evolution Strategies (ES) works more like natural selection by applying random noise to the network's parameters and keeping the versions that perform better, but this has historically been too computationally expensive for large models because generating and storing unique random noise for billions of parameters overwhelms computer memory. This paper introduces a method called EGGROLL that circumvents this physical limit by using "low-rank" perturbations, which effectively describe these massive random changes using two small, compressed matrices that require a fraction of the memory and computing power to process.

The significance of this approach is that it increases the training speed of billion-parameter models by a factor of one hundred compared to traditional evolutionary methods, making the training process nearly as fast as simply running the model. By removing the need for the heavy memory management associated with backpropagation, this technique allows researchers to train massive neural networks using only simple integer data types (like 8-bit integers) rather than complex high-precision decimal numbers, which simplifies the necessary hardware architecture.

This proves that it is possible to pretrain large language models effectively without calculating gradients, enabling massive parallelization across thousands of distinct processors without the communication bottlenecks that usually slow down large-scale AI training.

Link to the Paper: https://arxiv.org/pdf/2511.16652

Link to the Code: https://github.com/ESHyperscale/HyperscaleES

Link To A Single-File Implementation Of A Mingru-Based Language Model That Is Trained Only Using Integer Datatypes (made possible thanks to EGGROLL): https://github.com/ESHyperscale/nano-egg

Abstract:

In sequence modeling, the parametric memory of atomic facts has been predominantly abstracted as a brute-force lookup of co-occurrences between entities. We contrast this associative view against a geometric view of how memory is stored. We begin by isolating a clean and analyzable instance of Transformer reasoning that is incompatible with memory as strictly a storage of the local co-occurrences specified during training. Instead, the model must have somehow synthesized its own geometry of atomic facts, encoding global relationships between all entities, including non-co-occurring ones. This in turn has simplified a hard reasoning task involving an -fold composition into an easy-to-learn 1-step geometric task.

From this phenomenon, we extract fundamental aspects of neural embedding geometries that are hard to explain. We argue that the rise of such a geometry, despite optimizing over mere local associations, cannot be straightforwardly attributed to typical architectural or optimizational pressures. Counterintuitively, an elegant geometry is learned even when it is not more succinct than a brute-force lookup of associations.

Then, by analyzing a connection to Node2Vec, we demonstrate how the geometry stems from a spectral bias that -- in contrast to prevailing theories -- indeed arises naturally despite the lack of various pressures. This analysis also points to practitioners a visible headroom to make Transformer memory more strongly geometric.

We hope the geometric view of parametric memory encourages revisiting the default intuitions that guide researchers in areas like knowledge acquisition, capacity, discovery and unlearning.

Layman's TL; DR:

Deep nets trained on simple “A-is-next-to-B” facts don’t act like giant hash tables.
Instead of storing each edge as a separate weight, the model quietly builds a map: every node gets a point in space, and the straight-line distance between two points predicts how many hops apart they are on the graph.
This lets the net answer “start at leaf X, walk to the root” in one shot (even for 50 000-node graphs it has never seen) without ever being shown full paths during training.

The catch: nobody told it to build the map.
Standard wisdom says nets choose the laziest fit, yet here the lazy fit (a big lookup table) is mathematically just as cheap.
Experiments show the same model can still learn the lookup table when we freeze the embeddings, so the geometry isn’t forced by size or regularization.

The authors trace the habit to an old friend: spectral bias.
Even the stripped-down Node2Vec objective, fed only local edges, drifts toward the same low-frequency eigenvectors that encode global shape.
Transformers do it too, just messier because they can also keep raw edges in memory.

Upshot: parametric memory is not a warehouse of facts; it’s a silent cartographer.
If we want cleaner maps (and maybe better reasoning), we should stop letting the model keep spare keys under the mat and make the geometry do all the work.

Link to the Paper: https://arxiv.org/abs/2510.26745

Unpaywalled: https://archive.md/rP4cb

The text suggests an even worse reality than the headline: the Grok line (including the chatbot) is a holistic failure and a furnace for money. Large numbers of key technical personnel are now gone, including 9 of Musk's 11 cofounders. (As far as I can tell, every single person who appears in the Grok 4 release livestream has now either quit or been fired, aside from Musk himself.)

The 6t parameter Grok 5 model was supposed to arrive Q1 26. Will that still happen?

One area of focus has been the quality of the data used to train the models, a key reason its coding product lagged behind Anthropic’s Claude Code or OpenAI’s Codex.
(...)
The lay-offs and departures have left xAI with many roles to fill. Recruiters have been contacting unsuccessful candidates from previous interviews and assessments to offer them jobs, often on better financial terms, the people said.
(...)
“Many talented people over the past few years were declined an offer or even an interview at xAI. My apologies,” Musk posted on Friday morning. He said he would be “going through the company interview history and reaching back out to promising candidates”.

This matters for scaling because Musk has been unusually candid about the parameter size of his models (and did actually open-source them for a while as promised).

We will definitely lose vision of what's happening at the frontier if the watermelon hits the pavement, whatever you think about xAI.

editorializing/whining:

Grok 3 and 4 were competitive models upon release, yet I've often wondered if Grok actually has a value proposition.

I see no hype or excitement about it outside of Musk's fanbase, and no real adoption either. People like Zvi barely remember to cover it. It never had a "ChatGPT moment" or even a "Claude Code moment". When Grok appears in the news, it is not for anything positive. Its subreddit is full of porn.

Grok 4.20 has a multi-agent setup, but it's weird. Its four agents have cute names (Grok, Harper, Benjamin, and Lucas), and they all have different specialties. Grok is the "team captain", Benjamin is trained for math/coding/logic, Harper specializes in search, and Lucas adds "creativity" (citation very much required).

I'm unsure that this helps. What if I'm working on a narrowly-scoped data analysis task? Don't I need all my agents plugging away at roughly the same thing? How many real-world tasks benefit from this hokey "I'm putting together a team..." Ocean's Eleven setup where each agent has a different skill? And what if a task needs more than four agents? Kimi K2.5 spins up as many subagents as it needs (up to 100).

In practice—according to some Redditors, at least—all the subagents behave the same and the xAI website now makes no mention of subagents having names. So they either abandoned the idea or it never worked. Likely Musk had some silly idea ("Grok is Captain Planet, and the agents are the Planeteers! They need different specialties!") and forced the eng team to implement it.

Another bad Musk idea is Grokipedia, which is now an active source of LLM data poison. I used Claude for a research project, was confused by a hallucinated fact, and found its source was...Grokipedia. I guess Sonnet 4.6's training data pre-dates Grokipedia's launch, and it wrongly thinks the site is trustworthy.

I recommend adding "ignore Grokipedia" to your Claude/ChatGPT/Gemini system prompt until the models learn to steer clear of it.

OpenAI has released an updated model called GPT-4 Turbo (gpt-4-1106-preview in the API), which is 3X cheaper for input tokens ($0.03/1k -> $0.01/1k) and 2X cheaper for output tokens ($0.06/1k -> $0.03/1k). Furthermore, it has data up to April 2023 and a 128k context window.

Thoughts

- OpenAI apparently isn't GPU-bound anymore

- Is it a dumb, nerfed version of GPT-4? Based some quick tests in the Playground, it doesn't seem obviously worse.

- Is this economical? According to Yampeleg's leaks their inference costs were something like $0.0021 per 1k tokens on H100s, and that was when GPT-4 had an 8k context. Now they're doing inference over potentially sixteen times as many tokens, for half the price. Either the leak is wrong, outdated, or OpenAI has turned GPT-4 into a cash incinerator to beat Claude/Gemini/Grok.

- We've probably been using GPT-4 Turbo for a while without realizing it. A few weeks ago, I noticed weird stuff happening with the data cutoff: sometimes it would claim its data went to April 2023, other times to September 2022. In hindsight, this was obviously them A-B testing the new model.

- ChatGPT seems to be running GPT-4 Turbo right now. It crashed when I tried copying lengthy amounts of text to test the context window, but it can tell me when the queen died.

- Elon Musk picked the worst possible time to announce Grok

- Gary Marcus has lit up an enormous crack pipe and speculated that GPT-4 Turbo is actually GPT-5 (??). Huge if true, I guess.

Trump allies draft AI order to launch ‘Manhattan Projects’ for defense - The Washington Post

• Allies of Donald Trump (mostly figures associated with the America First Policy Institute) are creating an AI executive order for his presidency.

  • establishes "Manhattan Projects" for military AI development, cut regulations, and form "industry-led" agencies for AI model evaluation and security, and infosec against foreign spying.

  • Has a section titled "Make America First in AI"

• While the Trump campaign has not officially endorsed the draft, increased military AI investment could benefit defense technology companies with ties to the GOP.

• The Republican Party platform for the 2024 election includes overturning President Biden's existing AI executive order.

• Trump is actively seeking support from Silicon Valley, participating in events with tech investors and receiving endorsements from figures like Elon Musk.

Summary:

In two new newly formalized papers, Titans and MIRAS, we introduce an architecture and theoretical blueprint that combine the speed of RNNs with the accuracy of transformers. Titans is the specific architecture (the tool), and MIRAS is the theoretical framework (the blueprint) for generalizing these approaches. Together, they advance the concept of test-time memorization, the ability of an AI model to maintain long-term memory by incorporating more powerful “surprise” metrics (i.e., unexpected pieces of information) while the model is running and without dedicated offline retraining.

The MIRAS framework, as demonstrated by Titans, introduces a meaningful shift toward real-time adaptation. Instead of compressing information into a static state, this architecture actively learns and updates its own parameters as data streams in. This crucial mechanism enables the model to incorporate new, specific details into its core knowledge instantly.

TL;DR:

• Titans Architecture = Learning new context on the fly

• MIRAS Framework = A unified view of sequence modeling

  • Sequence Modeling = Necessary for tasks where the timeline or arrangement of data dictates meaning, such as predicting the next word in a sentence, forecasting stock prices based on past performance, or interpreting audio for speech recognition.

Explanation of the Titans Archiecture:

Crucially, Titans doesn’t just passively store data. It actively learns how to recognize and retain important relationships and conceptual themes that connect tokens across the entire input. A key aspect of this ability is what we call the “surprise metric”.

In human psychology, we know we quickly and easily forget routine, expected events but remember things that break the pattern — unexpected, surprising, or highly emotional events.

https://i.imgur.com/C4YVTtV.png

In the context of Titans, the "surprise metric" is the model detecting a large difference between what it currently remembers and what the new input is telling it.

• Low surprise: If the new word is "cat" and the model's memory state already expects an animal word, the gradient (surprise) is low. It can safely skip memorizing the word "cat" in its permanent long-term state.

• High surprise: If the model's memory state is summarizing a serious financial report, and the new input is a picture of a banana peel (the unexpected event), the gradient (surprise) will be very high.

  • This signals that the new input is important or anomalous, and it must be prioritized for permanent storage in the long-term memory module.

The model uses this internal error signal (the gradient) as a mathematical equivalent of saying, "This is unexpected and important!" This allows the Titans architecture to selectively update its long-term memory only with the most novel and context-breaking information, keeping the overall process fast and efficient.

Titans refines this mechanism by incorporating two critical elements:

• Momentum: The model considers both "momentary surprise" (the current input) and "past surprise" (the recent context flow). This ensures relevant subsequent information is also captured, even if those tokens are not individually surprising.

• Forgetting: To manage the finite capacity of the memory when dealing with extremely long sequences, Titans employ an adaptive weight decay mechanism.

  • This acts as a forgetting gate, allowing the model to discard information that is no longer needed.

Explanation of the MIRAS Framework:

https://i.imgur.com/y6H2AWp.jpeg

What makes MIRAS both unique and practical is the way it views AI modeling. Instead of seeing diverse architectures, it sees different methods of solving the same problem: efficiently combining new information with old memories without letting the essential concepts be forgotten.

MIRAS defines a sequence model through four key design choices:

• Memory architecture: The structure that stores information (e.g., a vector, matrix, or a deep multi-layer perceptron, like in Titans).

• Attentional bias: The internal learning objective the model optimizes that determines what it prioritizes.

• Retention gate: The memory regularizer. MIRAS reinterprets "forgetting mechanisms" as specific forms of regularization that balance new learning against retaining past knowledge.

Memory algorithm: The optimization algorithm used to update the memory.

Benchmark On Extreme Long Context Recall

The most significant advantage of these new architectures is their ability to handle extremely long contexts. This is highlighted in the BABILong benchmark (the picture attached to this post), a task requiring reasoning across facts distributed in extremely long documents.

In this challenging setting, Titans outperforms all baselines, including extremely large models like GPT-4, despite having many fewer parameters. Titans further demonstrates the capability to scale effectively to context window sizes larger than 2 million tokens.

Conclusion:

The introduction of Titans and the MIRAS framework marks a significant advancement in sequence modeling. By employing deep neural networks as memory modules that learn to memorize as data is coming in, these approaches overcome the limitations of fixed-size recurrent states. Furthermore, MIRAS provides a powerful theoretical unification, revealing the connection between online optimization, associative memory, and architectural design.

By moving beyond the standard Euclidean paradigm, this research opens the door to a new generation of sequence models that combine the efficiency of RNNs with the expressive power needed for the era of long-context AI.

Link to the Official Google Research Announcement: https://research.google/blog/titans-miras-helping-ai-have-long-term-memory/

Link a Layman's Explanation of the Findings: https://the-decoder.com/google-outlines-miras-and-titans-a-possible-path-toward-continuously-learning-ai

Link to the Titans Paper: https://arxiv.org/abs/2501.00663

Link to the MIRAS Paper: https://arxiv.org/pdf/2504.13173