Aligning superhuman AI with human interests is a significant challenge because there is no reason to believe a machine that can rewrite its own code will remain aligned with us.

“the idea that these things will stay aligned with us because we have built them we gave them a capacity to rewrite their code there's just no reason to believe that”

Humanity currently lacks the capability to align superhuman AI with human interests, and the required alignment work is progressing significantly slower than AI capabilities.

“That is absolutely the trajectory we are on right now and it's not close. Like capabilities are are running along orders of magnitude faster than the level of alignment work you would need to target a super intelligence.”