Karpathy dropped microgpt a few weeks ago and a 200-line pure Python GPT built on scalar autograd. Beautiful project. I wanted to see what happens when you throw the tape away entirely and derive every gradient analytically at the matrix level.

The result: ~20 BLAS calls instead of ~57,000 autograd nodes. Same math, none of the overhead. Fastest batch=1 implementation out there. The gap to EEmicroGPT is batching, f32 vs f64, and hand-tuned SIMD not the algorithm.

Repo + full benchmarks: https://github.com/ssrhaso/microjpt

Also working on a companion blog walking through all the matrix calculus and RMSNorm backward, softmax Jacobian, the dK/dQ asymmetry in attention. The main reason for this is because I want to improve my own understanding through Feynmann Learning whilst also explaining the fundamental principles which apply to almost all modern deep learning networks. Will post when its completed and please let me know if you have any questions or concerns I would love to hear your opinions!

I just released the second edition of my book Numerical Methods for Scientific Computing. The digital version of the book is free at the link. The print edition is available from Amazon. The book covers the mathematical theory and practical considerations of the essential numerical methods used in scientific computing. Julia is used throughout, with Python and Matlab/Octave included in the back matter. Jupyter notebooks of the code are available on GitHub.

I’m releasing the book with an agile publishing mindset—get it out quickly and cheaply with minimal errors so that it can be of use, and then iterate and improve with feedback. The book is designed for senior undergraduate and first-year graduate students and as a self-study for anyone with a working knowledge of multivariate calculus and linear algebra. Any feedback on errors, omissions, or suggestions is appreciated.

The code is meant to help the reader better connect the dots to the math concepts—something in the spirit of Nick Trefethen’s ten-digit algorithms. Moreover, the methods discussed in the book are typically already available in optimized Julia packages. That said, I'm by no means fluent in Julia (or Python or Matlab, for that matter), and I don’t want to cultivate weird, wrong, or bad Julia practices. I would be thankful for any critical comments. Feel free to DM me. My email is also on the edition notice page of the book.

Yesterday I made a post that claimed that jluna, the modern C++ <-> Wrapper, had a thread pool that was 80% faster than std::thread. That benchmark had a methodological error, I have corrected it today and the new figure is ~4% speedup, which is much more conservative. At least it isn't any slower than C++, which was my goal to begin with.

The new benchmark can be found here.

I know not many people saw it to begin with, but I wanted to be honest and not lie to people, so I deleted the old post.

If you would still like to check out the 1.0 version of jluna, it is available for free under MIT license here. I also announced it on discourse.

We took a poll the other day to decide whether to include Julia in Erdos (the open source data science IDE we've built), and between the polling results and comments we got on other subs, we decided to do it. In Erdos, there's now a Julia console, and the Julia runtime connects to the plot system, the documentation system, the variables explorer, and the package manager. Julia scripts can be executed in part or in full with Cmd/Ctrl-Enter, and jupyter notebooks with Julia also still work. You can try it out at https://www.lotas.ai/erdos - we're happy to hear any feedback!

(Hopefully it's clear we've added this to help Julia users since a lot of people have said or voted they want something like this and that we're not just self promoting.)

There's a new "Terminal" app that is native to some Pixel devices. I've enabled it in developer settings and just used juliaup to install the latest Julia and it worked! It just asked for some build dependencies that can be installed with apt.

I think that this feature should also roll out in the future for other non Pixel devices.