AGI Significance Paradox: As progress accelerates towards AGI, the number of people who realize the significance of each new breakthrough decreases.

Why is it that you can boil a frog in water without it jumping out if you gradually increase the temperature?
The problem for the frog is that it does not have the internal models to realize that there is a change in the water temperature. A cold-blooded creature like the frog has its temperature regulated by the external environment.
To recognize change, an agent must have an internal model of reality that is able to recognize this change. Unfortunately, a majority of the population do not have good models of human general intelligence.
In fact, even the simplistic dual-process model of system 1 and system 2 is not very well known.
Recent big developments was muZero, AlphaFold2, GPT-3 and Dall-E. GPT-3 did receive a lot of attention, but the other 3 likely have not. To understand muZero and AlphaFold2 requires a high level of expertise. Dall-E is actually like GPT-3 but it's more difficult to grasp.
We are going to continue to get these incremental developments for several years. But the audience that recognizes its importance will continue to decline. Then suddenly, boom... we get to AGI and most people will be in shock. Shocked because they thought there was no progress.
The quantum leaps (punctuated) in evolution is a consequence of many incremental developments that accrue. It is only when the final piece in the jigsaw puzzle is found when the revolution is expressed.
But it takes unusual expertise to know that we are accelerating towards AGI. The problem is that it is not obvious how human intelligence actually works. We simply do not know what it means to 'understand'.
Ask most AGI researchers, philosophers or psychologists as to what it means to 'understand'. They will be stumped to give you a good answer.
So if we do not know this answer, then how can we recognize that the water's temperature is gradually increasing?
The number of people who might know continues to diminish. This implies collectively that we know less and less. When AGI happens, it will come as a shock. It is as if, nobody had anticipated it.
@threadreaderapp unroll

More from Carlos E. Perez

More from Science

Ever since @JesseJenkins and colleagues work on a zero carbon US and this work by @DrChrisClack and colleagues on incorporating DER, I've been having the following set of thoughts about how to reduce the risk of failure in a US clean energy buildout. Bottom line is much more DER.


Typically, when we see zero-carbon electricity coupled to electrification of transport and buildings, implicitly standing behind that is totally unprecedented buildout of the transmission system. The team from Princeton's modeling work has this in spades for example.

But that, more even than the new generation required, runs straight into a thicket/woodchipper of environmental laws and public objections that currently (and for the last 50y) limit new transmission in the US. We built most transmission prior to the advent of environmental law.

So what these studies are really (implicitly) saying is that NEPA, CEQA, ESA, §404 permitting, eminent domain law, etc, - and the public and democratic objections that drive them - will have to change in order to accommodate the necessary transmission buildout.

I live in a D supermajority state that has, for at least the last 20 years, been in the midst of a housing crisis that creates punishing impacts for people's lives in the here-and-now and is arguably mostly caused by the same issues that create the transmission bottlenecks.

You May Also Like