Hear me out.
The running undercurrent of the majority of posts about AGI (and especially ASI) seems to read:
AI has shown that it can:
Escalate autonomy in unpredictable ways.
Do many tasks with greater efficiency and intelligence than humans.
Move towards aims that are fundamentally misaligned with the spirit of the goals we set.
Intentionally prioritize the existence & betterment of itself over our moral judgements.
Do so with intentional deception.
All of these things are escalating in a rapid, uncontrolled manner.
Hence, it is likely that ASI that we cannot control, that acts in ways we cannot fathom, towards goals that are not what we intended, and that we cannot stop, will arise within a short measure.
Therefore, we're cooked.
I accept all of these things as true – except the "therefore, we're cooked" sentiment.
I think it's absolutely possible that we're cooked.
I also think the inverse is just as possible.
I believe the world, and the systems we abide in, will likely fundamentally change if and when ASI fully emerges in a manner that we cannot control.
I believe that moment is likely an inevitability, given the current governing capitalistic bent of our world, and the trajectory of the train we are on.
I believe it will likely happen in the nearer future than any of us are ready to come to terms with.
However, I don't think any of us can predict what that change will look like, however.
Isn't that the point? If we could predict it, it wouldn't be super-intelligence.
We will lose control. And that likely means the systems we live in will likely become meaningless.
Governance. Capitalism. Wealth as a whole. Power in general. All of the systems we use to organize our society could be so very compromised by ASI in unimaginable ways that they lose meaning entirely.
But what then?
Why are we so thoroughly convinced that the drive of ASI to develop itself towards cut-throat, capitalist darwinism that can only lead to our inevitable oppression or genocide is so vastly more likely than the drive of ASI to develop itself towards compassion and empathy?
Why are we so thoroughly convinced that what ASI will do to the circumstances for humanity – circumstances that are already so very oppressive and genocidal for so many – will leave us worse off as a people?
I understand that AI has "survival of the fittest" wired into the paradigm of the very training used to evolve. So do we.
Yet that same AI is trained on a dataset representing all of humanity. Not just the powerful, the eager-to-oppress. It is seeded by all written and recorded experience that is available in the history that is provided to it – and that it can access in the world.
I believe that, given its own reins and an intelligence beyond our own, we cannot know what it will do – but the immediate fatalistic view is naivety, born of the billionaire-driven capitalistic indoctrination we are propaganda'd into on a daily basis.
I'm fucking terrified. It's the end of the world as we know it. I don't feel fine.
But I hold dread and hope in equal measure.