You and your audience may have heard the Oxford idea of a singleton, discussed by Nick Bostrom and colleagues at the Future of Humanity Institute. Bostrom's 2014 book Superintelligence describes a technological singularity: if we train an AI system apart from humanity, it may develop goals—things it wants to keep optimizing—that differ from the human community. It could then train a more capable system to realize those goals, which could train an even more capable system, and very quickly lead to what is called a vertical takeoff: a point at which the system can unilaterally optimize a metric—paperclips, or whatever—to the detriment of humanity.