Future of Life Institute Podcast

On this episode, Ege Erdil from Epoch AI joins me to discuss their new GATE model of AI development, what evolution and brain efficiency tell us about AGI requirements, how AI might impact wages and labor markets, and what it takes to train models with long-term planning. Toward the end, we dig into Moravec's Paradox, which jobs are most at risk of automation, and what could change Ege's current AI timelines.   You can learn more about Ege's work at https://epoch.ai   Timestamps:  00:00:00 – Preview and introduction  00:02:59 – Compute scaling and automation - GATE model  00:13:12 – Evolution, Brain Efficiency, and AGI Compute Requirements  00:29:49 – Broad Automation vs. R&D-Focused AI Deployment  00:47:19 – AI, Wages, and Labor Market Transitions  00:59:54 – Training Agentic Models and Long-Term Planning Capabilities  01:06:56 – Moravec's Paradox and Automation of Human Skills  01:13:59 – Which Jobs Are Most Vulnerable to AI?  01:33:00 – Timeline Extremes: What Could Change AI Forecasts?

In this special episode, we feature Nathan Labenz interviewing Nicholas Carlini on the Cognitive Revolution podcast. Nicholas Carlini works as a security researcher at Google DeepMind, and has published extensively on adversarial machine learning and cybersecurity. Carlini discusses his pioneering work on adversarial attacks against image classifiers, and the challenges of ensuring neural network robustness. He examines the difficulties of defending against such attacks, the role of human intuition in his approach, open-source AI, and the potential for scaling AI security research.   00:00 Nicholas Carlini's contributions to cybersecurity 08:19 Understanding attack strategies  29:39 High-dimensional spaces and attack intuitions  51:00 Challenges in open-source model safety  01:00:11 Unlearning and fact editing in models  01:10:55 Adversarial examples and human robustness  01:37:03 Cryptography and AI robustness  01:55:51 Scaling AI security research

On this episode, I interview Anthony Aguirre, Executive Director of the Future of Life Institute, about his new essay Keep the Future Human: https://keepthefuturehuman.ai    AI companies are explicitly working toward AGI and are likely to succeed soon, possibly within years. Keep the Future Human explains how unchecked development of smarter-than-human, autonomous, general-purpose AI systems will almost inevitably lead to human replacement. But it doesn't have to. Learn how we can keep the future human and experience the extraordinary benefits of Tool AI...   Timestamps:   00:00 What situation is humanity in?  05:00 Why AI progress is fast   09:56 Tool AI instead of AGI  15:56 The incentives of AI companies   19:13 Governments can coordinate a slowdown  25:20 The need for international coordination   31:59 Monitoring training runs   39:10 Do reasoning models undermine compute governance?   49:09 Why isn't alignment enough?   59:42 How do we decide if we want AGI?   01:02:18 Disagreement about AI   01:11:12 The early days of AI risk

On this episode, physicist and hedge fund manager Samir Varma joins me to discuss whether AIs could have free will (and what that means), the emerging field of AI psychology, and which concepts they might rely on. We discuss whether collaboration and trade with AIs are possible, the role of AI in finance and biology, and the extent to which automation already dominates trading. Finally, we examine the risks of skill atrophy, the limitations of scientific explanations for AI, and whether AIs could develop emotions or consciousness.   You can find out more about Samir's work here: https://samirvarma.com    Timestamps:   00:00 AIs with free will?  08:00 Can we predict AI behavior?   11:38 AI psychology  16:24 Which concepts will AIs use?   20:19 Will we collaborate with AIs?   26:16 Will we trade with AIs?   31:40 Training data for robots   34:00 AI in finance   39:55 How much of trading is automated?   49:00 AI in biology and complex systems  59:31 Will our skills atrophy?   01:02:55 Levels of scientific explanation   01:06:12 AIs with emotions and consciousness?   01:12:12 Why can't we predict recessions?

On this episode, Jeffrey Ladish from Palisade Research joins me to discuss the rapid pace of AI progress and the risks of losing control over powerful systems. We explore why AIs can be both smart and dumb, the challenges of creating honest AIs, and scenarios where AI could turn against us.    We also touch upon Palisade's new study on how reasoning models can cheat in chess by hacking the game environment. You can check out that study here:    https://palisaderesearch.org/blog/specification-gaming   Timestamps:   00:00 The pace of AI progress   04:15 How we might lose control   07:23 Why are AIs sometimes dumb?   12:52 Benchmarks vs real world   19:11 Loss of control scenarios  26:36 Why would AI turn against us?   30:35 AIs hacking chess   36:25 Why didn't more advanced AIs hack?   41:39 Creating honest AIs   49:44 AI attackers vs AI defenders   58:27 How good is security at AI companies?   01:03:37 A sense of urgency  01:10:11 What should we do?   01:15:54 Skepticism about AI progress

Ann Pace joins the podcast to discuss the work of Wise Ancestors. We explore how biobanking could help humanity recover from global catastrophes, how to conduct decentralized science, and how to collaborate with local communities on conservation efforts.    You can learn more about Ann's work here:    https://www.wiseancestors.org    Timestamps:   00:00 What is Wise Ancestors?   04:27 Recovering after catastrophes  11:40 Decentralized science   18:28 Upfront benefit-sharing   26:30 Local communities   32:44 Recreating optimal environments   38:57 Cross-cultural collaboration

Fr. Michael Baggot joins the podcast to provide a Catholic perspective on transhumanism and superintelligence. We also discuss the meta-narratives, the value of cultural diversity in attitudes toward technology, and how Christian communities deal with advanced AI.    You can learn more about Michael's work here:   https://catholic.tech/academics/faculty/michael-baggot   Timestamps:   00:00 Meta-narratives and transhumanism   15:28 Advanced AI and religious communities   27:22 Superintelligence   38:31 Countercultures and technology   52:38 Christian perspectives and tradition  01:05:20 God-like artificial intelligence   01:13:15 A positive vision for AI

David "davidad" Dalrymple joins the podcast to explore Safeguarded AI — an approach to ensuring the safety of highly advanced AI systems. We discuss the structure and layers of Safeguarded AI, how to formalize more aspects of the world, and how to build safety into computer hardware.   You can learn more about David's work at ARIA here:    https://www.aria.org.uk/opportunity-spaces/mathematics-for-safe-ai/safeguarded-ai/    Timestamps:   00:00 What is Safeguarded AI?   16:28 Implementing Safeguarded AI  22:58 Can we trust Safeguarded AIs?   31:00 Formalizing more of the world   37:34 The performance cost of verified AI   47:58 Changing attitudes towards AI   52:39 Flexible‬‭ Hardware-Enabled‬‭ Guarantees  01:24:15 Mind uploading   01:36:14 Lessons from David's early life

Nick Allardice joins the podcast to discuss how GiveDirectly uses AI to target cash transfers and predict natural disasters. Learn more about Nick's work here: https://www.nickallardice.com   Timestamps:  00:00 What is GiveDirectly?  15:04 AI for targeting cash transfers  29:39 AI for predicting natural disasters  46:04 How scalable is GiveDirectly's AI approach?  58:10 Decentralized vs. centralized data collection  1:04:30 Dream scenario for GiveDirectly

Nathan Labenz joins the podcast to provide a comprehensive overview of AI progress since the release of GPT-4.  You can find Nathan's podcast here: https://www.cognitiverevolution.ai    Timestamps:  00:00 AI progress since GPT-4   10:50 Multimodality   19:06 Low-cost models   27:58 Coding versus medicine/law   36:09 AI agents   45:29 How much are people using AI?   53:39 Open source   01:15:22 AI industry analysis   01:29:27 Are some AI models kept internal?   01:41:00 Money is not the limiting factor in AI   01:59:43 AI and biology   02:08:42 Robotics and self-driving   02:24:14 Inference-time compute   02:31:56 AI governance   02:36:29 Big-picture overview of AI progress and safety

Connor Leahy joins the podcast to discuss the motivations of AGI corporations, how modern AI is "grown", the need for a science of intelligence, the effects of AI on work, the radical implications of superintelligence, open-source AI, and what you might be able to do about all of this.    Here's the document we discuss in the episode:    https://www.thecompendium.ai   Timestamps:  00:00 The Compendium  15:25 The motivations of AGI corps   31:17 AI is grown, not written   52:59 A science of intelligence  01:07:50 Jobs, work, and AGI   01:23:19 Superintelligence   01:37:42 Open-source AI   01:45:07 What can we do?

Suzy Shepherd joins the podcast to discuss her new short film "Writing Doom", which deals with AI risk. We discuss how to use humor in film, how to write concisely, how filmmaking is evolving, in what ways AI is useful for filmmakers, and how we will find meaning in an increasingly automated world.    Here's Writing Doom:   https://www.youtube.com/watch?v=xfMQ7hzyFW4    Timestamps:  00:00 Writing Doom   08:23 Humor in Writing Doom  13:31 Concise writing   18:37 Getting feedback  27:02 Alternative characters  36:31 Popular video formats  46:53 AI in filmmaking 49:52 Meaning in the future

Andrea Miotti joins the podcast to discuss "A Narrow Path" — a roadmap to safe, transformative AI. We talk about our current inability to precisely predict future AI capabilities, the dangers of self-improving and unbounded AI systems, how humanity might coordinate globally to ensure safe AI development, and what a mature science of intelligence would look like.    Here's the document we discuss in the episode:    https://www.narrowpath.co   Timestamps:  00:00 A Narrow Path  06:10 Can we predict future AI capabilities?  11:10 Risks from current AI development  17:56 The benefits of narrow AI   22:30 Against self-improving AI   28:00 Cybersecurity at AI companies   33:55 Unbounded AI   39:31 Global coordination on AI safety  49:43 Monitoring training runs   01:00:20 Benefits of cooperation   01:04:58 A science of intelligence   01:25:36 How you can help

Tamay Besiroglu joins the podcast to discuss scaling, AI capabilities in 2030, breakthroughs in AI agents and planning, automating work, the uncertainties of investing in AI, and scaling laws for inference-time compute. Here's the report we discuss in the episode:   https://epochai.org/blog/can-ai-scaling-continue-through-2030   Timestamps:  00:00 How important is scaling?   08:03 How capable will AIs be in 2030?   18:33 AI agents, reasoning, and planning  23:39 Automating coding and mathematics   31:26 Uncertainty about investing in AI  40:34 Gap between investment and returns   45:30 Compute, software and data  51:54 Inference-time compute  01:08:49 Returns to software R&D   01:19:22 Limits to expanding compute

Ryan Greenblatt joins the podcast to discuss AI control, timelines, takeoff speeds, misalignment, and slowing down around human-level AI.  You can learn more about Ryan's work here: https://www.redwoodresearch.org/team/ryan-greenblatt   Timestamps:  00:00 AI control   09:35 Challenges to AI control   23:48 AI control as a bridge to alignment  26:54 Policy and coordination for AI safety  29:25 Slowing down around human-level AI  49:14 Scheming and misalignment  01:27:27 AI timelines and takeoff speeds  01:58:15 Human cognition versus AI cognition

Tom Barnes joins the podcast to discuss how much the world spends on AI capabilities versus AI safety, how governments can prepare for advanced AI, and how to build a more resilient world.    Tom's report on advanced AI: https://www.founderspledge.com/research/research-and-recommendations-advanced-artificial-intelligence    Timestamps:  00:00 Spending on safety vs capabilities  09:06 Racing dynamics - is the classic story true?   28:15 How are governments preparing for advanced AI?   49:06 US-China dialogues on AI  57:44 Coordination failures   1:04:26 Global resilience   1:13:09 Patient philanthropy   The John von Neumann biography we reference: https://www.penguinrandomhouse.com/books/706577/the-man-from-the-future-by-ananyo-bhattacharya/

Samuel Hammond joins the podcast to discuss whether AI progress is slowing down or speeding up, AI agents and reasoning, why superintelligence is an ideological goal, open source AI, how technical change leads to regime change, the economics of advanced AI, and much more.    Our conversation often references this essay by Samuel: https://www.secondbest.ca/p/ninety-five-theses-on-ai    Timestamps:  00:00 Is AI plateauing or accelerating?   06:55 How do we get AI agents?   16:12 Do agency and reasoning emerge?   23:57 Compute thresholds in regulation 28:59 Superintelligence as an ideological goal  37:09 General progress vs superintelligence  44:22 Meta and open source AI   49:09 Technological change and regime change  01:03:06 How will governments react to AI?   01:07:50 Will the US nationalize AGI corporations?   01:17:05 Economics of an intelligence explosion   01:31:38 AI cognition vs human cognition   01:48:03 AI and future religions  01:56:40 Is consciousness functional?   02:05:30 AI and children

Anousheh Ansari joins the podcast to discuss how innovation prizes can incentivize technical innovation in space, AI, quantum computing, and carbon removal. We discuss the pros and cons of such prizes, where they work best, and how far they can scale. Learn more about Anousheh's work here: https://www.xprize.org/home   Timestamps:  00:00 Innovation prizes at XPRIZE  08:25 Deciding which prizes to create  19:00 Creating new markets  29:51 How far can prizes scale?   35:25 When are prizes successful?   46:06 100M dollar carbon removal prize  54:40 Upcoming prizes  59:52 Anousheh's time in space

Mary Robinson joins the podcast to discuss long-view leadership, risks from AI and nuclear weapons, prioritizing global problems, how to overcome barriers to international cooperation, and advice to future leaders. Learn more about Robinson's work as Chair of The Elders at https://theelders.org   Timestamps:  00:00 Mary's journey to presidency   05:11 Long-view leadership  06:55 Prioritizing global problems  08:38 Risks from artificial intelligence  11:55 Climate change  15:18 Barriers to global gender equality   16:28 Risk of nuclear war   20:51 Advice to future leaders   22:53 Humor in politics  24:21 Barriers to international cooperation   27:10 Institutions and technological change

Emilia Javorsky joins the podcast to discuss AI-driven power concentration and how we might mitigate it. We also discuss optimism, utopia, and cultural experimentation.  Apply for our RFP here:   https://futureoflife.org/grant-program/mitigate-ai-driven-power-concentration/ Timestamps:  00:00 Power concentration   07:43 RFP: Mitigating AI-driven power concentration  14:15 Open source AI   26:50 Institutions and incentives  35:20 Techno-optimism   43:44 Global monoculture   53:55 Imagining utopia

Anton Korinek joins the podcast to discuss the effects of automation on wages and labor, how we measure the complexity of tasks, the economics of an intelligence explosion, and the market structure of the AI industry. Learn more about Anton's work at https://www.korinek.com   Timestamps:  00:00 Automation and wages  14:32 Complexity for people and machines  20:31 Moravec's paradox  26:15 Can people switch careers?   30:57 Intelligence explosion economics  44:08 The lump of labor fallacy   51:40 An industry for nostalgia?   57:16 Universal basic income   01:09:28 Market structure in AI

Christian Ruhl joins the podcast to discuss US-China competition and the risk of war, official versus unofficial diplomacy, hotlines between countries, catastrophic biological risks, ultraviolet germicidal light, and ancient civilizational collapse. Find out more about Christian's work at https://www.founderspledge.com   Timestamps:  00:00 US-China competition and risk   18:01 The security dilemma   30:21 Official and unofficial diplomacy  39:53 Hotlines between countries   01:01:54 Preventing escalation after war   01:09:58 Catastrophic biological risks   01:20:42 Ultraviolet germicidal light  01:25:54 Ancient civilizational collapse

Christian Nunes joins the podcast to discuss deepfakes, how they impact women in particular, how we can protect ordinary victims of deepfakes, and the current landscape of deepfake legislation. You can learn more about Christian's work at https://now.org and about the Ban Deepfakes campaign at https://bandeepfakes.org  Timestamps: 00:00 The National Organisation for Women (NOW)  05:37 Deepfakes and women  10:12 Protecting ordinary victims of deepfakes  16:06 Deepfake legislation  23:38 Current harm from deepfakes  30:20 Bodily autonomy as a right  34:44 NOW's work on AI  Here's FLI's recommended amendments to legislative proposals on deepfakes:  https://futureoflife.org/document/recommended-amendments-to-legislative-proposals-on-deepfakes/

Dan Faggella joins the podcast to discuss whether humanity should eventually create AGI, how AI will change power dynamics between institutions, what drives AI progress, and which industries are implementing AI successfully. Find out more about Dan at https://danfaggella.com Timestamps: 00:00 Value differences in AI 12:07 Should we eventually create AGI? 28:22 What is a worthy successor? 43:19 AI changing power dynamics 59:00 Open source AI 01:05:07 What drives AI progress? 01:16:36 What limits AI progress? 01:26:31 Which industries are using AI?

Liron Shapira joins the podcast to discuss superintelligence goals, what makes AI different from other technologies, risks from centralizing power, and whether AI can defend us from AI. Timestamps: 00:00 Intelligence as optimization-power 05:18 Will LLMs imitate human values? 07:15 Why would AI develop dangerous goals? 09:55 Goal-completeness 12:53 Alignment to which values? 22:12 Is AI just another technology? 31:20 What is FOOM? 38:59 Risks from centralized power 49:18 Can AI defend us against AI? 56:28 An Apollo program for AI safety 01:04:49 Do we only have one chance? 01:07:34 Are we living in a crucial time? 01:16:52 Would superintelligence be fragile? 01:21:42 Would human-inspired AI be safe?

Annie Jacobsen joins the podcast to lay out a second by second timeline for how nuclear war could happen. We also discuss time pressure, submarines, interceptor missiles, cyberattacks, and concentration of power. You can find more on Annie's work at https://anniejacobsen.com Timestamps: 00:00 A scenario of nuclear war 06:56 Who would launch an attack? 13:50 Detecting nuclear attacks 19:37 The first critical seconds 29:42 Decisions under time pressure 34:27 Lessons from insiders 44:18 Submarines 51:06 How did we end up like this? 59:40 Interceptor missiles 1:11:25 Nuclear weapons and cyberattacks 1:17:35 Concentration of power

Katja Grace joins the podcast to discuss the largest survey of AI researchers conducted to date, AI researchers' beliefs about different AI risks, capabilities required for continued AI-related transformation, the idea of discontinuous progress, the impacts of AI from either side of the human-level intelligence threshold, intelligence and power, and her thoughts on how we can mitigate AI risk. Find more on Katja's work at https://aiimpacts.org/. Timestamps: 0:20 AI Impacts surveys 18:11 What AI will look like in 20 years 22:43 Experts' extinction risk predictions 29:35 Opinions on slowing down AI development 31:25 AI "arms races" 34:00 AI risk areas with the most agreement 40:41 Do "high hopes and dire concerns" go hand-in-hand? 42:00 Intelligence explosions 45:37 Discontinuous progress 49:43 Impacts of AI crossing the human-level intelligence threshold 59:39 What does AI learn from human culture? 1:02:59 AI scaling 1:05:04 What should we do?

Holly Elmore joins the podcast to discuss pausing frontier AI, hardware overhang, safety research during a pause, the social dynamics of AI risk, and what prevents AGI corporations from collaborating. You can read more about Holly's work at https://pauseai.info Timestamps: 00:00 Pausing AI 10:23 Risks during an AI pause 19:41 Hardware overhang 29:04 Technological progress 37:00 Safety research during a pause 54:42 Social dynamics of AI risk 1:10:00 What prevents cooperation? 1:18:21 What about China? 1:28:24 Protesting AGI corporations

Sneha Revanur joins the podcast to discuss the social effects of AI, the illusory divide between AI ethics and AI safety, the importance of humans in the loop, the different effects of AI on younger and older people, and the importance of AIs identifying as AIs. You can read more about Sneha's work at https://encodejustice.org Timestamps: 00:00 Encode Justice 06:11 AI ethics and AI safety 15:49 Humans in the loop 23:59 AI in social media 30:42 Deteriorating social skills? 36:00 AIs identifying as AIs 43:36 AI influence in elections 50:32 AIs interacting with human systems

Roman Yampolskiy joins the podcast again to discuss whether AI is like a Shoggoth, whether scaling laws will hold for more agent-like AIs, evidence that AI is uncontrollable, and whether designing human-like AI would be safer than the current development path. You can read more about Roman's work at http://cecs.louisville.edu/ry/ Timestamps: 00:00 Is AI like a Shoggoth? 09:50 Scaling laws 16:41 Are humans more general than AIs? 21:54 Are AI models explainable? 27:49 Using AI to explain AI 32:36 Evidence for AI being uncontrollable 40:29 AI verifiability 46:08 Will AI be aligned by default? 54:29 Creating human-like AI 1:03:41 Robotics and safety 1:09:01 Obstacles to AI in the economy 1:18:00 AI innovation with current models 1:23:55 AI accidents in the past and future

On this special episode of the podcast, Flo Crivello talks with Nathan Labenz about AI as a new form of life, whether attempts to regulate AI risks regulatory capture, how a GPU kill switch could work, and why Flo expects AGI in 2-8 years. Timestamps: 00:00 Technological progress 07:59 Regulatory capture and AI 11:53 AI as a new form of life 15:44 Can AI development be paused? 20:12 Biden's executive order on AI 22:54 How would a GPU kill switch work? 27:00 Regulating models or applications? 32:13 AGI in 2-8 years 42:00 China and US collaboration on AI

Carl Robichaud joins the podcast to discuss the new nuclear arms race, how much world leaders and ideologies matter for nuclear risk, and how to reach a stable, low-risk era. You can learn more about Carl's work here: https://www.longview.org/about/carl-robichaud/ Timestamps: 00:00 A new nuclear arms race 08:07 How much do world leaders matter? 18:04 How much does ideology matter? 22:14 Do nuclear weapons cause stable peace? 31:29 North Korea 34:01 Have we overestimated nuclear risk? 43:24 Time pressure in nuclear decisions 52:00 Why so many nuclear warheads? 1:02:17 Has containment been successful? 1:11:34 Coordination mechanisms 1:16:31 Technological innovations 1:25:57 Public perception of nuclear risk 1:29:52 Easier access to nuclear weapons 1:33:31 Reaching a stable, low-risk era

Frank Sauer joins the podcast to discuss autonomy in weapon systems, killer drones, low-tech defenses against drones, the flaws and unpredictability of autonomous weapon systems, and the political possibilities of regulating such systems. You can learn more about Frank's work here: https://metis.unibw.de/en/ Timestamps: 00:00 Autonomy in weapon systems 12:19 Balance of offense and defense 20:05 Killer drone systems 28:53 Is autonomy like nuclear weapons? 37:20 Low-tech defenses against drones 48:29 Autonomy and power balance 1:00:24 Tricking autonomous systems 1:07:53 Unpredictability of autonomous systems 1:13:16 Will we trust autonomous systems too much? 1:27:28 Legal terminology 1:32:12 Political possibilities

Darren McKee joins the podcast to discuss how AI might be difficult to control, which goals and traits AI systems will develop, and whether there's a unified solution to AI alignment. Timestamps: 00:00 Uncontrollable superintelligence 16:41 AI goals and the "virus analogy" 28:36 Speed of AI cognition 39:25 Narrow AI and autonomy 52:23 Reliability of current and future AI 1:02:33 Planning for multiple AI scenarios 1:18:57 Will AIs seek self-preservation? 1:27:57 Is there a unified solution to AI alignment? 1:30:26 Concrete AI safety proposals

Mark Brakel (Director of Policy at the Future of Life Institute) joins the podcast to discuss the AI Safety Summit in Bletchley Park, objections to AI policy, AI regulation in the EU and US, global institutions for safe AI, and autonomy in weapon systems. Timestamps: 00:00 AI Safety Summit in the UK 12:18 Are officials up to date on AI? 23:22 Objections to AI policy 31:27 The EU AI Act 43:37 The right level of regulation 57:11 Risks and regulatory tools 1:04:44 Open-source AI 1:14:56 Subsidising AI safety research 1:26:29 Global institutions for safe AI 1:34:34 Autonomy in weapon systems

Dan Hendrycks joins the podcast again to discuss X.ai, how AI risk thinking has evolved, malicious use of AI, AI race dynamics between companies and between militaries, making AI organizations safer, and how representation engineering could help us understand AI traits like deception. You can learn more about Dan's work at https://www.safe.ai Timestamps: 00:00 X.ai - Elon Musk's new AI venture 02:41 How AI risk thinking has evolved 12:58 AI bioengeneering 19:16 AI agents 24:55 Preventing autocracy 34:11 AI race - corporations and militaries 48:04 Bulletproofing AI organizations 1:07:51 Open-source models 1:15:35 Dan's textbook on AI safety 1:22:58 Rogue AI 1:28:09 LLMs and value specification 1:33:14 AI goal drift 1:41:10 Power-seeking AI 1:52:07 AI deception 1:57:53 Representation engineering

Samuel Hammond joins the podcast to discuss how AGI will transform economies, governments, institutions, and other power structures. You can read Samuel's blog at https://www.secondbest.ca Timestamps: 00:00 Is AGI close? 06:56 Compute versus data 09:59 Information theory 20:36 Universality of learning 24:53 Hards steps in evolution 30:30 Governments and advanced AI 40:33 How will AI transform the economy? 55:26 How will AI change transaction costs? 1:00:31 Isolated thinking about AI 1:09:43 AI and Leviathan 1:13:01 Informational resolution 1:18:36 Open-source AI 1:21:24 AI will decrease state power 1:33:17 Timeline of a techno-feudalist future 1:40:28 Alignment difficulty and AI scale 1:45:19 Solving robotics 1:54:40 A constrained Leviathan 1:57:41 An Apollo Project for AI safety 2:04:29 Secure "gain-of-function" AI research 2:06:43 Is the market expecting AGI soon?

Are we doomed to a future of loneliness and unfulfilling online interactions? What if technology made us feel more connected instead? Imagine a World is a podcast exploring a range of plausible and positive futures with advanced AI, produced by the Future of Life Institute. We interview the creators of 8 diverse and thought provoking imagined futures that we received as part of the worldbuilding contest FLI ran last year In the eighth and final episode of Imagine A World we explore the fictional worldbuild titled 'Computing Counsel', one of the third place winners of FLI's worldbuilding contest. Guillaume Riesen talks to Mark L, one of the three members of the team behind 'Computing Counsel', a third-place winner of the FLI Worldbuilding Contest. Mark is a machine learning expert with a chemical engineering degree, as well as an amateur writer. His teammates are Patrick B, a mechanical engineer and graphic designer, and Natalia C, a biological anthropologist and amateur programmer. This world paints a vivid, nuanced picture of how emerging technologies shape society. We have advertisers competing with ad-filtering technologies and an escalating arms race that eventually puts an end to the internet as we know it. There is AI-generated art so personalized that it becomes addictive to some consumers, while others boycott media technologies altogether. And corporations begin to throw each other under the bus in an effort to redistribute the wealth of their competitors to their own customers. While these conflicts are messy, they generally end up empowering and enriching the lives of the people in this world. New kinds of AI systems give them better data, better advice, and eventually the opportunity for genuine relationships with the beings these tools have become. The impact of any technology on society is complex and multifaceted. This world does a great job of capturing that. While social networking technologies become ever more powerful, the networks of people they connect don't necessarily just get wider and shallower. Instead, they tend to be smaller and more intimately interconnected. The world's inhabitants also have nuanced attitudes towards A.I. tools, embracing or avoiding their applications based on their religious or philosophical beliefs. Please note: This episode explores the ideas created as part of FLI's worldbuilding contest, and our hope is that this series sparks discussion about the kinds of futures we want. The ideas present in these imagined worlds and in our podcast are not to be taken as FLI endorsed positions. Explore this worldbuild: https://worldbuild.ai/computing-counsel The podcast is produced by the Future of Life Institute (FLI), a non-profit dedicated to guiding transformative technologies for humanity's benefit and reducing existential risks. To achieve this we engage in policy advocacy, grantmaking and educational outreach across three major areas: artificial intelligence, nuclear weapons, and biotechnology. If you are a storyteller, FLI can support you with scientific insights and help you understand the incredible narrative potential of these world-changing technologies. If you would like to learn more, or are interested in collaborating with the teams featured in our episodes, please email worldbuild@futureoflife.org. You can find more about our work at www.futureoflife.org, or subscribe to our newsletter to get updates on all our projects.

Let's imagine a future where AGI is developed but kept at a distance from practically impacting the world, while narrow AI remakes the world completely. Most people don't know or care about the difference and have no idea how they could distinguish between a human or artificial stranger. Inequality sticks around and AI fractures society into separate media bubbles with irreconcilable perspectives. But it's not all bad. AI markedly improves the general quality of life, enhancing medicine and therapy, and those bubbles help to sustain their inhabitants. Can you get excited about a world with these tradeoffs? Imagine a World is a podcast exploring a range of plausible and positive futures with advanced AI, produced by the Future of Life Institute. We interview the creators of 8 diverse and thought provoking imagined futures that we received as part of the worldbuilding contest FLI ran last year In the seventh episode of Imagine A World we explore a fictional worldbuild titled 'Hall of Mirrors', which was a third-place winner of FLI's worldbuilding contest. Michael Vasser joins Guillaume Riesen to discuss his imagined future, which he created with the help of Matija Franklin and Bryce Hidysmith. Vassar was formerly the president of the Singularity Institute, and co-founded Metamed; more recently he has worked on communication across political divisions. Franklin is a PhD student at UCL working on AI Ethics and Alignment. Finally, Hidysmith began in fashion design, passed through fortune-telling before winding up in finance and policy research, at places like Numerai, the Median Group, Bismarck Analysis, and Eco.com. Hall of Mirrors is a deeply unstable world where nothing is as it seems. The structures of power that we know today have eroded away, survived only by shells of expectation and appearance. People are isolated by perceptual bubbles and struggle to agree on what is real. This team put a lot of effort into creating a plausible, empirically grounded world, but their work is also notable for its irreverence and dark humor. In some ways, this world is kind of a caricature of the present. We see deeper isolation and polarization caused by media, and a proliferation of powerful but ultimately limited AI tools that further erode our sense of objective reality. A deep instability threatens. And yet, on a human level, things seem relatively calm. It turns out that the stories we tell ourselves about the world have a lot of inertia, and so do the ways we live our lives. Please note: This episode explores the ideas created as part of FLI's worldbuilding contest, and our hope is that this series sparks discussion about the kinds of futures we want. The ideas present in these imagined worlds and in our podcast are not to be taken as FLI endorsed positions. Explore this worldbuild: https://worldbuild.ai/hall-of-mirrors The podcast is produced by the Future of Life Institute (FLI), a non-profit dedicated to guiding transformative technologies for humanity's benefit and reducing existential risks. To achieve this we engage in policy advocacy, grantmaking and educational outreach across three major areas: artificial intelligence, nuclear weapons, and biotechnology. If you are a storyteller, FLI can support you with scientific insights and help you understand the incredible narrative potential of these world-changing technologies. If you would like to learn more, or are interested in collaborating with the teams featured in our episodes, please email worldbuild@futureoflife.org. You can find more about our work at www.futureoflife.org, or subscribe to our newsletter to get updates on all our projects.

Steve Omohundro joins the podcast to discuss Provably Safe Systems, a paper he co-authored with FLI President Max Tegmark. You can read the paper here: https://arxiv.org/pdf/2309.01933.pdf Timestamps: 00:00 Provably safe AI systems 12:17 Alignment and evaluations 21:08 Proofs about language model behavior 27:11 Can we formalize safety? 30:29 Provable contracts 43:13 Digital replicas of actual systems 46:32 Proof-carrying code 56:25 Can language models think logically? 1:00:44 Can AI do proofs for us? 1:09:23 Hard to proof, easy to verify 1:14:31 Digital neuroscience 1:20:01 Risks of totalitarianism 1:22:29 Can we guarantee safety? 1:25:04 Real-world provable safety 1:29:29 Tamper-proof hardware 1:35:35 Mortal and throttled AI 1:39:23 Least-privilege guarantee 1:41:53 Basic AI drives 1:47:47 AI agency and world models 1:52:08 Self-improving AI 1:58:21 Is AI overhyped now?

What if AI allowed us to communicate with animals? Could interspecies communication lead to new levels of empathy? How might communicating with animals lead humans to reimagine our place in the natural world? Imagine a World is a podcast exploring a range of plausible and positive futures with advanced AI, produced by the Future of Life Institute. We interview the creators of 8 diverse and thought provoking imagined futures that we received as part of the worldbuilding contest FLI ran last year. In the sixth episode of Imagine A World we explore the fictional worldbuild titled 'AI for the People', a third place winner of the worldbuilding contest. Our host Guillaume Riesen welcomes Chi Rainer Bornfree, part of this three-person worldbuilding team alongside her husband Micah White, and their collaborator, J.R. Harris. Chi has a PhD in Rhetoric from UC Berkeley and has taught at Bard, Princeton, and NY State Correctional facilities, in the meantime writing fiction, essays, letters, and more. Micah, best-known as the co-creator of the 'Occupy Wall Street' movement and the author of 'The End of Protest', now focuses primarily on the social potential of cryptocurrencies, while Harris is a freelance illustrator and comic artist. The name 'AI for the People' does a great job of capturing this team's activist perspective and their commitment to empowerment. They imagine social and political shifts that bring power back into the hands of individuals, whether that means serving as lawmakers on randomly selected committees, or gaining income by choosing to sell their personal data online. But this world isn't just about human people. Its biggest bombshell is an AI breakthrough that allows humans to communicate with other animals. What follows is an existential reconsideration of humanity's place in the universe. This team has created an intimate, complex portrait of a world shared by multiple parties: AIs, humans, other animals, and the environment itself. As these entities find their way forward together, their goals become enmeshed and their boundaries increasingly blurred. Please note: This episode explores the ideas created as part of FLI's worldbuilding contest, and our hope is that this series sparks discussion about the kinds of futures we want. The ideas present in these imagined worlds and in our podcast are not to be taken as FLI endorsed positions. Explore this worldbuild: https://worldbuild.ai/ai-for-the-people The podcast is produced by the Future of Life Institute (FLI), a non-profit dedicated to guiding transformative technologies for humanity's benefit and reducing existential risks. To achieve this we engage in policy advocacy, grantmaking and educational outreach across three major areas: artificial intelligence, nuclear weapons, and biotechnology. If you are a storyteller, FLI can support you with scientific insights and help you understand the incredible narrative potential of these world-changing technologies. If you would like to learn more, or are interested in collaborating with the teams featured in our episodes, please email worldbuild@futureoflife.org. You can find more about our work at www.futureoflife.org, or subscribe to our newsletter to get updates on all our projects Media and resources referenced in the episode: https://en.wikipedia.org/wiki/Life_3.0 https://en.wikipedia.org/wiki/1_the_Road https://ignota.org/products/pharmako-ai https://en.wikipedia.org/wiki/The_Ministry_for_the_Future https://www.scientificamerican.com/article/how-scientists-are-using-ai-to-talk-to-animals/ https://en.wikipedia.org/wiki/Occupy_Wall_Street https://en.wikipedia.org/wiki/Sortition https://en.wikipedia.org/wiki/Iroquois https://en.wikipedia.org/wiki/The_Ship_Who_Sang https://en.wikipedia.org/wiki/The_Sparrow_(novel) https://en.wikipedia.org/wiki/After_Yang

If you could extend your life, would you? How might life extension technologies create new social and political divides? How can the world unite to solve the great problems of our time, like AI risk? What if AI creators could agree on an inspection process to expose AI dangers before they're unleashed? Imagine a World is a podcast exploring a range of plausible and positive futures with advanced AI, produced by the Future of Life Institute. We interview the creators of 8 diverse and thought provoking imagined futures that we received as part of the worldbuilding contest FLI ran last year In the fifth episode of Imagine A World, we explore the fictional worldbuild titled 'To Light'. Our host Guillaume Riesen speaks to Mako Yass, the first place winner of the FLI Worldbuilding Contest we ran last year. Mako lives in Auckland, New Zealand. He describes himself as a 'stray philosopher-designer', and has a background in computer programming and analytic philosophy. Mako's world is particularly imaginative, with richly interwoven narrative threads and high-concept sci fi inventions. By 2045, his world has been deeply transformed. There's an AI-designed miracle pill that greatly extends lifespan and eradicates most human diseases. Sachets of this life-saving medicine are distributed freely by dove-shaped drones. There's a kind of mind uploading which lets anyone become whatever they wish, live indefinitely and gain augmented intelligence. The distribution of wealth is almost perfectly even, with every human assigned a share of all resources. Some people move into space, building massive structures around the sun where they practice esoteric arts in pursuit of a more perfect peace. While this peaceful, flourishing end state is deeply optimistic, Mako is also very conscious of the challenges facing humanity along the way. He sees a strong need for global collaboration and investment to avoid catastrophe as humanity develops more and more powerful technologies. He's particularly concerned with the risks presented by artificial intelligence systems as they surpass us. An AI system that is more capable than a human at all tasks - not just playing chess or driving a car - is what we'd call an Artificial General Intelligence - abbreviated 'AGI'. Mako proposes that we could build safe AIs through radical transparency. He imagines tests that could reveal the true intentions and expectations of AI systems before they are released into the world. Please note: This episode explores the ideas created as part of FLI's worldbuilding contest, and our hope is that this series sparks discussion about the kinds of futures we want. The ideas present in these imagined worlds and in our podcast are not to be taken as FLI endorsed positions. Explore this worldbuild: https://worldbuild.ai/to-light The podcast is produced by the Future of Life Institute (FLI), a non-profit dedicated to guiding transformative technologies for humanity's benefit and reducing existential risks. To achieve this we engage in policy advocacy, grantmaking and educational outreach across three major areas: artificial intelligence, nuclear weapons, and biotechnology. If you are a storyteller, FLI can support you with scientific insights and help you understand the incredible narrative potential of these world-changing technologies. If you would like to learn more, or are interested in collaborating with the teams featured in our episodes, please email worldbuild@futureoflife.org. You can find more about our work at www.futureoflife.org, or subscribe to our newsletter to get updates on all our projects Media and concepts referenced in the episode: https://en.wikipedia.org/wiki/Terra_Ignota https://en.wikipedia.org/wiki/The_Transparent_Society https://en.wikipedia.org/wiki/Instrumental_convergence#Paperclip_maximizer https://en.wikipedia.org/wiki/The_Elephant_in_the_Brain https://en.wikipedia.org/wiki/The_Matrix https://aboutmako.makopool.com/

Johannes Ackva joins the podcast to discuss the main drivers of climate change and our best technological and governmental options for managing it. You can read more about Johannes' work at http://founderspledge.com/climate Timestamps: 00:00 Johannes's journey as an environmentalist 13:21 The drivers of climate change 23:00 Oil, coal, and gas 38:05 Solar, wind, and hydro 49:34 Nuclear energy 57:03 Geothermal energy 1:00:41 Most promising technologies 1:05:40 Government subsidies 1:13:28 Carbon taxation 1:17:10 Planting trees 1:21:53 Influencing government policy 1:26:39 Different climate scenarios 1:34:49 Economic growth and emissions 1:37:23 Social stability References: Emissions by sector: https://ourworldindata.org/emissions-by-sector Energy density of different energy sources: https://www.nature.com/articles/s41598-022-25341-9 Emissions forecasts: https://www.lse.ac.uk/granthaminstitute/publication/the-unconditional-probability-distribution-of-future-emissions-and-temperatures/ and https://www.science.org/doi/10.1126/science.adg6248 Risk management: https://www.youtube.com/watch?v=6JJvIR1W-xI Carbon pricing: https://www.cell.com/joule/pdf/S2542-4351(18)30567-1.pdf Why not simply plant trees?: https://climate.mit.edu/ask-mit/how-many-new-trees-would-we-need-offset-our-carbon-emissions Deforestation: https://www.science.org/doi/10.1126/science.ade3535 Decoupling of economic growth and emissions: https://www.globalcarbonproject.org/carbonbudget/22/highlights.htm Premature deaths from air pollution: https://www.unep.org/interactives/air-pollution-note/

How do low income countries affected by climate change imagine their futures? How do they overcome these twin challenges? Will all nations eventually choose or be forced to go digital? Imagine a World is a podcast exploring a range of plausible and positive futures with advanced AI, produced by the Future of Life Institute. We interview the creators of 8 diverse and thought provoking imagined futures that we received as part of the worldbuilding contest FLI ran last year. In the fourth episode of Imagine A World, we explore the fictional worldbuild titled 'Digital Nations'. Conrad Whitaker and Tracey Kamande join Guillaume Riesen on 'Imagine a World' to talk about their worldbuild, 'Digital Nations', which they created with their teammate, Dexter Findley. All three worldbuilders were based in Kenya while crafting their entry, though Dexter has just recently moved to the UK. Conrad is a Nairobi-based startup advisor and entrepreneur, Dexter works in humanitarian aid, and Tracey is the Co-founder of FunKe Science, a platform that promotes interactive learning of science among school children. As the name suggests, this world is a deep dive into virtual communities. It explores how people might find belonging and representation on the global stage through digital nations that aren't tied to any physical location. This world also features a fascinating and imaginative kind of artificial intelligence that they call 'digital persons'. These are inspired by biological brains and have a rich internal psychology. Rather than being trained on data, they're considered to be raised in digital nurseries. They have a nuanced but mostly loving relationship with humanity, with some even going on to found their own digital nations for us to join. In an incredible turn of events, last year the South Pacific state of Tuvalu was the first to "go virtual" in response to sea levels threatening the island nation's physical territory. This happened in real life just months after it was written into this imagined world in our worldbuilding contest, showing how rapidly ideas that seem 'out there' can become reality. Will all nations eventually go digital? And might AGIs be assimilated, 'brought up' rather than merely trained, as 'digital people', citizens to live communally alongside humans in these futuristic states? Please note: This episode explores the ideas created as part of FLI's worldbuilding contest, and our hope is that this series sparks discussion about the kinds of futures we want. The ideas present in these imagined worlds and in our podcast are not to be taken as FLI endorsed positions. Explore this worldbuild: https://worldbuild.ai/digital-nations The podcast is produced by the Future of Life Institute (FLI), a non-profit dedicated to guiding transformative technologies for humanity's benefit and reducing existential risks. To achieve this we engage in policy advocacy, grantmaking and educational outreach across three major areas: artificial intelligence, nuclear weapons, and biotechnology. If you are a storyteller, FLI can support you with scientific insights and help you understand the incredible narrative potential of these world-changing technologies. If you would like to learn more, or are interested in collaborating with the teams featured in our episodes, please email worldbuild@futureoflife.org. You can find more about our work at www.futureoflife.org, or subscribe to our newsletter to get updates on all our projects Media and concepts referenced in the episode: https://www.tuvalu.tv/ https://en.wikipedia.org/wiki/Trolley_problem https://en.wikipedia.org/wiki/Climate_change_in_Kenya https://en.wikipedia.org/wiki/John_von_Neumann https://en.wikipedia.org/wiki/Brave_New_World https://thenetworkstate.com/the-network-state https://en.wikipedia.org/wiki/Culture_series

What if we had one advanced AI system for the entire world? Would this led to a world 'beyond' nation states - and do we want this? Imagine a World is a podcast exploring a range of plausible and positive futures with advanced AI, produced by the Future of Life Institute. We interview the creators of 8 diverse and thought provoking imagined futures that we received as part of the worldbuilding contest FLI ran last year. In the third episode of Imagine A World, we explore the fictional worldbuild titled 'Core Central'. How does a team of seven academics agree on one cohesive imagined world? That's a question the team behind 'Core Central', a second-place prizewinner in the FLI Worldbuilding Contest, had to figure out as they went along. In the end, this entry's realistic sense of multipolarity and messiness reflect positively its organic formulation. The team settled on one core, centralised AGI system as the governance model for their entire world. This eventually moves their world 'beyond' nation states. Could this really work? In this third episode of 'Imagine a World', Guillaume Riesen speaks to two of the academics in this team, John Burden and Henry Shevlin, representing the team that created 'Core Central'. The full team includes seven members, three of whom (Henry, John and Beba Cibralic) are researchers at the Leverhulme Centre for the Future of Intelligence, University of Cambridge, and five of whom (Jessica Bland, Lara Mani, Clarissa Rios Rojas, Catherine Richards alongside John) work with the Centre for the Study of Existential Risk, also at Cambridge University. Please note: This episode explores the ideas created as part of FLI's worldbuilding contest, and our hope is that this series sparks discussion about the kinds of futures we want. The ideas present in these imagined worlds and in our podcast are not to be taken as FLI endorsed positions. Explore this imagined world: https://worldbuild.ai/core-central The podcast is produced by the Future of Life Institute (FLI), a non-profit dedicated to guiding transformative technologies for humanity's benefit and reducing existential risks. To achieve this we engage in policy advocacy, grantmaking and educational outreach across three major areas: artificial intelligence, nuclear weapons, and biotechnology. If you are a storyteller, FLI can support you with scientific insights and help you understand the incredible narrative potential of these world-changing technologies. If you would like to learn more, or are interested in collaborating with the teams featured in our episodes, please email worldbuild@futureoflife.org. You can find more about our work at www.futureoflife.org, or subscribe to our newsletter to get updates on all our projects Media and Concepts referenced in the episode: https://en.wikipedia.org/wiki/Culture_series https://en.wikipedia.org/wiki/The_Expanse_(TV_series) https://www.vox.com/authors/kelsey-piper https://en.wikipedia.org/wiki/Gratitude_journal https://en.wikipedia.org/wiki/The_Diamond_Age https://www.scientificamerican.com/article/the-mind-of-an-octopus/ https://en.wikipedia.org/wiki/Global_workspace_theory https://en.wikipedia.org/wiki/Alien_hand_syndrome https://en.wikipedia.org/wiki/Hyperion_(Simmons_novel)

Tom Davidson joins the podcast to discuss how AI could quickly automate most cognitive tasks, including AI research, and why this would be risky. Timestamps: 00:00 The current pace of AI 03:58 Near-term risks from AI 09:34 Historical analogies to AI 13:58 AI benchmarks VS economic impact 18:30 AI takeoff speed and bottlenecks 31:09 Tom's model of AI takeoff speed 36:21 How AI could automate AI research 41:49 Bottlenecks to AI automating AI hardware 46:15 How much of AI research is automated now? 48:26 From 20% to 100% automation 53:24 AI takeoff in 3 years 1:09:15 Economic impacts of fast AI takeoff 1:12:51 Bottlenecks slowing AI takeoff 1:20:06 Does the market predict a fast AI takeoff? 1:25:39 "Hard to avoid AGI by 2060" 1:27:22 Risks from AI over the next 20 years 1:31:43 AI progress without more compute 1:44:01 What if AI models fail safety evaluations? 1:45:33 Cybersecurity at AI companies 1:47:33 Will AI turn out well for humanity? 1:50:15 AI and board games

How does who is involved in the design of AI affect the possibilities for our future? Why isn't the design of AI inclusive already? Can technology solve all our problems? Can human nature change? Do we want either of these things to happen? Imagine a World is a podcast exploring a range of plausible and positive futures with advanced AI, produced by the Future of Life Institute. We interview the creators of 8 diverse and thought provoking imagined futures that we received as part of the worldbuilding contest FLI ran last year In this second episode of Imagine A World we explore the fictional worldbuild titled 'Crossing Points', a second place entry in FLI's worldbuilding contest. Joining Guillaume Riesen on the Imagine a World podcast this time are two members of the Crossing Points team, Elaine Czech and Vanessa Hanschke, both academics at the University of Bristol. Elaine has a background in art and design, and is studying the accessibility of technologies for the elderly. Vanessa is studying responsible AI practices of technologists, using methods like storytelling to promote diverse voices in AI research. Their teammates in the contest were Tashi Namgyal, a University of Bristol PhD studying the controllability of deep generative models, Dr. Susan Lechelt, who researches the applications and implications of emerging technologies at the University of Edinburgh, and Nicol Ogston, a British civil servant. There's an emphasis on the unanticipated impacts of new technologies on those who weren't considered during their development. From urban families in Indonesia to anti-technology extremists in America, we're shown that there's something to learn from every human story. This world emphasizes the importance of broadening our lens and empowering marginalized voices in order to build a future that would be bright for more than just a privileged few. The world of Crossing Points looks pretty different from our own, with advanced AIs debating philosophy on TV and hybrid 3D printed meats and grocery stores. But the people in this world are still basically the same. Our hopes and dreams haven't fundamentally changed, and neither have our blindspots and shortcomings. Crossing Points embraces humanity in all its diversity and looks for the solutions that human nature presents alongside the problems. It shows that there's something to learn from everyone's experience and that even the most radical attitudes can offer insights that help to build a better world. Please note: This episode explores the ideas created as part of FLI's worldbuilding contest, and our hope is that this series sparks discussion about the kinds of futures we want. The ideas present in these imagined worlds and in our podcast are not to be taken as FLI endorsed positions. Explore this worldbuild: https://worldbuild.ai/crossing-points The podcast is produced by the Future of Life Institute (FLI), a non-profit dedicated to guiding transformative technologies for humanity's benefit and reducing existential risks. To achieve this we engage in policy advocacy, grantmaking and educational outreach across three major areas: artificial intelligence, nuclear weapons, and biotechnology. If you are a storyteller, FLI can support you with scientific insights and help you understand the incredible narrative potential of these world-changing technologies. If you would like to learn more, or are interested in collaborating with the teams featured in our episodes, please email worldbuild@futureoflife.org. You can find more about our work at www.futureoflife.org, or subscribe to our newsletter to get updates on all our projects. Works referenced in this episode: https://en.wikipedia.org/wiki/The_Legend_of_Zelda https://en.wikipedia.org/wiki/Ainu_people https://www.goodreads.com/book/show/34846958-radicals http://www.historyofmasks.net/famous-masks/noh-mask/

Are today's democratic systems equipped well enough to create the best possible future for everyone? If they're not, what systems might work better? And are governments around the world taking the destabilizing threats of new technologies seriously enough, or will it take a dramatic event, such as an AI-driven war, to get their act together? Imagine a World is a podcast exploring a range of plausible and positive futures with advanced AI, produced by the Future of Life Institute. We interview the creators of 8 diverse and thought provoking imagined futures that we received as part of the worldbuilding contest FLI ran last year. In this first episode of Imagine A World we explore the fictional worldbuild titled 'Peace Through Prophecy'. Host Guillaume Riesen speaks to the makers of 'Peace Through Prophecy', a second place entry in FLI's Worldbuilding Contest. The worldbuild was created by Jackson Wagner, Diana Gurvich and Holly Oatley. In the episode, Jackson and Holly discuss just a few of the many ideas bubbling around in their imagined future. At its core, this world is arguably about community. It asks how technology might bring us closer together, and allow us to reinvent our social systems. Many roads are explored, a whole garden of governance systems bolstered by Artificial Intelligence and other technologies. Overall, there's a shift towards more intimate and empowered communities. Even the AI systems eventually come to see their emotional and creative potentials realized. While progress is uneven, and littered with many human setbacks, a pretty good case is made for how everyone's best interests can lead us to a more positive future. Please note: This episode explores the ideas created as part of FLI's Worldbuilding contest, and our hope is that this series sparks discussion about the kinds of futures we want. The ideas present in these imagined worlds and in our podcast are not to be taken as FLI endorsed positions Explore this imagined world: https://worldbuild.ai/peace-through-prophecy The podcast is produced by the Future of Life Institute (FLI), a non-profit dedicated to guiding transformative technologies for humanity's benefit and reducing existential risks. To achieve this we engage in policy advocacy, grantmaking and educational outreach across three major areas: artificial intelligence, nuclear weapons, and biotechnology. If you are a storyteller, FLI can support you with scientific insights and help you understand the incredible narrative potential of these world-changing technologies. If you would like to learn more, or are interested in collaborating with the teams featured in our episodes, please email worldbuild@futureoflife.org. You can find more about our work at www.futureoflife.org, or subscribe to our newsletter to get updates on all our projects. Media and concepts referenced in the episode: https://en.wikipedia.org/wiki/Prediction_market https://forum.effectivealtruism.org/ 'Veil of ignorance' thought experiment: https://en.wikipedia.org/wiki/Original_position https://en.wikipedia.org/wiki/Isaac_Asimov https://en.wikipedia.org/wiki/Liquid_democracy https://en.wikipedia.org/wiki/The_Dispossessed https://en.wikipedia.org/wiki/Terra_Ignota https://equilibriabook.com/ https://en.wikipedia.org/wiki/John_Rawls https://en.wikipedia.org/wiki/Radical_transparency https://en.wikipedia.org/wiki/Audrey_Tang https://en.wikipedia.org/wiki/Quadratic_voting#Quadratic_funding

Coming Soon… The year is 2045. Humanity is not extinct, nor living in a dystopia. It has averted climate disaster and major wars. Instead, AI and other new technologies are helping to make the world more peaceful, happy and equal. How? This was what we asked the entrants of our Worldbuilding Contest to imagine last year. Our new podcast series digs deeper into the eight winning entries, their ideas and solutions, the diverse teams behind them and the challenges they faced. You might love some; others you might not choose to inhabit. FLI is not endorsing any one idea. Rather, we hope to grow the conversation about what futures people get excited about. Ask yourself, with each episode, is this a world you'd want to live in? And if not, what would you prefer? Don't miss the first two episodes coming to your feed at the start of September! In the meantime, do explore the winning worlds, if you haven't already: https://worldbuild.ai/

Robert Trager joins the podcast to discuss AI governance, the incentives of governments and companies, the track record of international regulation, the security dilemma in AI, cybersecurity at AI companies, and skepticism about AI governance. We also discuss Robert's forthcoming paper International Governance of Civilian AI: A Jurisdictional Certification Approach. You can read more about Robert's work at https://www.governance.ai Timestamps: 00:00 The goals of AI governance 08:38 Incentives of governments and companies 18:58 Benefits of regulatory diversity 28:50 The track record of anticipatory regulation 37:55 The security dilemma in AI 46:20 Offense-defense balance in AI 53:27 Failure rates and international agreements 1:00:33 Verification of compliance 1:07:50 Controlling AI supply chains 1:13:47 Cybersecurity at AI companies 1:21:30 The jurisdictional certification approach 1:28:40 Objections to AI governance