TL;DR: The topic of LLMs are/aren't a big step towards AGI had a lot of questionable statements stemming from possible misunderstanding of how LLM/AI/brain work.
I love Jay, but I think this latest episode missed the mark in this segment. I honestly don't even know what the point was. It started with a statement that some scientists think LLM is a big step towards AGI but the rest of it sounded like Jay's personal opinions on the subject. But to do that you need a pretty solid math and software background, and if you don't - the skeptics and sci-com 101 say "just cover what experts said". But there were no references or quotes of actual experts in the field..
"LLM just regurgitates" - how often can you say that whatever your opinion is is your own and is not a result from the knowledge taken from other people? Our brains regurgitate all the time. It's the most effective way to process the information and it would be weird if we didn't do that. Energy-wise.
"LLM just does statistics" - what's that supposed to convey? Our brain does that - connections become stronger the more often they are activated (among other things). That's pretty much - being driven by statistics. So why is this an argument against LLM being AGI?
The whole topic of "common sense". It's literally the most common answer you'd encounter in the body of knowledge of a society. That's exactly what LLMs do and do great. The whole issue on the path to AGI is to not do common sense, but to do actual logic. Maybe "common logic" is what was implied, but that's a totally different thing, and would probably result in a completely different discussion.
The examples were inadequate too, in my opinion. It takes 30 seconds to ask chat gpt why coffee gets cold and it explains it very well. You can ask to walk you through step by step what's happening and why, and it will tell you. Like Steve said multiple times before, you cannot tell if the system is AGI or not by just looking at the answers it gives you. LLMs currently WILL give you all the answers that make them seem AGI, while they are not. But your examples do not expose that. LLM totally knows what a "cheat day" is and will give you a proper answer (try it). But on the other hand, "cheat day" is not "common sense". Being from a non-US culture, I don't know what "cheat day" is and would be stumped by that question. Am I less AGI than Chat GPT?
With all that said, I do think LLMs are not AGI and will not become AGI simply by getting more training data. But the (emergent) thing they do, is converting the body of knowledge captured in written language, into mathematical/computer entities that can be operated on. And just that one thing could be a huge step to AGI. It's the ability to store and process knowledge and concepts (and not just combinations of letters) that brings a new power to these systems.