All of the following are strategies for avoiding presentation software catastrophes EXCEPT

Table of Contents

• 1 1. Many AI experts think there’s a non-negligible chance AI will lead to outcomes as bad as extinction
• 2 2. We’re making advances in AI extremely quickly
  • 2.1 Current trends show rapid progress in the capabilities of ML systems
  • 2.2 When can we expect transformative AI?
• 3 3. Power-seeking AI could pose an existential threat to humanity
  • 3.1 It’s likely we’ll build advanced planning systems
  • 3.2 Advanced planning systems could easily be dangerously ‘misaligned’
  • 3.3 Disempowerment by AI systems would be an existential catastrophe
  • 3.4 People might deploy misaligned AI systems despite the risk
• 4 This all sounds very abstract. What could an existential catastrophe caused by AI actually look like?
• 5 4. Even if we find a way to avoid power-seeking, there are still risks
  • 5.1 AI could worsen war
  • 5.2 AI could be used to develop dangerous new technology
  • 5.3 AI could empower totalitarian governments
  • 5.4 Other risks from AI
• 6 So, how likely is an AI-related catastrophe?
• 7 5. We can tackle these risks
  • 7.1 Technical AI safety research
  • 7.2 AI governance research and implementation
• 8 6. This work is extremely neglected
• 9 What do we think are the best arguments we’re wrong?
• 10 Arguments against working on AI risk to which we think there are strong responses
• 11 What you can do concretely to help
  • 11.1 Technical AI safety
  • 11.2 AI governance and strategy
  • 11.3 Complementary (yet crucial) roles
  • 11.4 Other ways to help
  • 11.5 Want one-on-one advice on pursuing this path?
  • 11.6 Find vacancies on our job board
• 12 Top resources to learn more
• 13 Acknowledgements

Note from the author: At its core, this problem profile tries to predict the future of technology. This is a notoriously difficult thing to do. In addition, there has been much less rigorous research into the risks from AI than into the other risks 80,000 Hours writes about (like pandemics or climate change).5 That said, there is a growing field of research into the topic, which I’ve tried to reflect. For this article I’ve leaned especially on this draft report by Joseph Carlsmith at Open Philanthropy, as it’s the most rigorous overview of the risk that I could find. I’ve also had the article reviewed by over 30 people with different expertise and opinions on the topic. (Almost all are concerned about advanced AI’s potential impact.)

If you have any feedback on this article — whether there’s something technical we’ve got wrong, some wording we could improve, or just that you did or didn’t like reading it — we’d really appreciate it if you could tell us what you think using this form.

Why do we think that reducing risks from AI is one of the most pressing issues of our time? In short, our reasons are:

1. Even before getting into the actual arguments, we can see some cause for concern — as many AI experts think there’s a small but non-negligible chance that AI will lead to outcomes as bad as human extinction.
2. We’re making advances in AI extremely quickly — which suggests that AI systems could have a significant influence on society, soon.
3. There are strong arguments that “power-seeking” AI could pose an existential threat to humanity6 — which we’ll go through below.
4. Even if we find a way to avoid power-seeking, there are still other risks.
5. We think we can tackle these risks.
6. This work is extremely neglected.

We’re going to cover each of these in turn, then consider some of the best counterarguments, explain concrete things you can do to help, and finally outline some of the best resources for learning more about this area.

1. Many AI experts think there’s a non-negligible chance AI will lead to outcomes as bad as extinction

If you think that some new technology is going to be a huge deal (and might even cause human extinction), but everyone who actually works with the technology thinks those concerns are misguided, then you’re probably missing something.

So before we go into the arguments about AI risks, let’s take a look at what the experts think.

We looked at three surveys of AI researchers who published at NeurIPS and ICML (two of the most prestigious machine learning conferences) — one in 2016, one in 2019, and one in 2022.7

It’s important to note that there could be considerable selection bias on surveys like this. For example, you might think researchers who go to the top AI conferences are more likely to be optimistic about AI, because they have been selected to think that AI research is doing good. Alternatively, you might think that researchers who are already concerned about AI are more likely to respond to a survey asking about these concerns.8

All that said, here’s what we found:

In all three surveys, the median researcher thought that the chances that AI would be “extremely good” was reasonably high: 20% in the 2016 survey, 20% in 2019, and 10% in 2022.9

Indeed, AI systems are already having substantial positive effects — for example, in medical care or academic research.

But in all three surveys, the median researcher also estimated small — but certainly not negligible — chances that AI would be “extremely bad (e.g. human extinction)”: a 5% chance of extremely bad outcomes in the 2016 survey, 2% in 2019, and 5% in 2022.10 11

In the 2022 survey, participants were specifically asked about the chances of existential catastrophe caused by future AI advances — and again, over half of researchers thought the chances of an existential catastrophe was greater than 5%.12

So experts disagree on the degree to which AI poses an existential risk — a kind of threat we’ve argued deserves serious moral weight.

This fits with our understanding of the state of the research field. Two of the leading labs developing AI — DeepMind and OpenAI — also have teams dedicated to figuring out how to solve technical safety issues that we believe could, for reasons we discuss at length below, lead to an existential threat to humanity.13

There are also several academic research groups (including at MIT, Oxford, Cambridge, Carnegie Mellon University, and UC Berkeley) focusing on these same technical AI safety problems.14

It’s hard to know exactly what to take from all this, but we think it shows that it’s not a fringe position in the field to think that there is a material risk of outcomes as bad as an existential catastrophe.

Still, why do we side with those who are more concerned? In short, it’s because there are arguments we’ve found persuasive that AI could pose such an existential threat — arguments we will go through step by step below.

It’s important to recognise that the fact that many experts recognise there’s a problem doesn’t mean that everything’s OK, the experts have got it covered. Overall, we think this problem remains highly neglected, with only around 300 people working directly on the issue worldwide (more on this below).

Meanwhile, there are billions of dollars a year going into making AI more advanced.4

2. We’re making advances in AI extremely quickly

“A cat dressed as a computer programmer” as generated by Craiyon (formerly DALL-E mini) (left) and OpenAI’s DALL-E 2. (right). DALL-E mini uses a model 27 times smaller than OpenAI’s DALL-E 1 model, released in January 2021. DALL-E 2 was released in April 2022.15

Before we try to figure out what the future of AI might look like, it’s helpful to take a look at what AI can already do.

Modern AI techniques involve machine learning (ML): models that improve automatically through data input. The most common form of this technique used today is known as deep learning.

ML systems today can only perform a very small portion of tasks that humans can do, and (with a few exceptions) only within narrow specialties (like playing one particular game or generating one particular kind of image).

That said, since the increasingly widespread use of deep learning in the mid-2010s, there has been huge progress in what can be achieved with ML. Here’s a brief timeline of only some of the advances we’ve seen since 2019:

• AlphaStar, which can beat top professional players at StarCraft II (January 2019)
• MuZero, a single system that learned to win games of chess, shogi, and Go — without ever being told the rules (November 2019)
• GPT-3, a natural language model capable of producing high-quality text (May 2020)
• GPT-f, which can solve some Maths Olympiad problems (September 2020)
• AlphaFold 2, a huge step forward in solving the long-perplexing protein-folding problem (July 2021)
• Codex, which can produce code for programs from natural language instructions (August 2021)
• PaLM, a language model which has shown impressive capabilities to reason about things like cause and effect or explaining jokes (April 2022)
• DALL-E 2 (April 2022) and Imagen (May 2022), which are both capable of generating high-quality images from written descriptions
• SayCan, which takes natural language instructions and uses them to operate a robot (April 2022)
• Gato, a single ML model capable of doing a huge number of different things (including playing Atari, captioning images, chatting, and stacking blocks with a real robot arm), deciding based on its context what it should output (May 2022)
• Minerva can solve complex maths problems — fairly well at college level, and even better at high school maths competition level. (Minerva is far more successful than forecasters predicted in 2021.)

If you’re anything like us, you found the complexity and breadth of the tasks these systems can carry out surprising.

And if the technology keeps advancing at this pace, it seems clear there will be major effects on society. At the very least, automating tasks makes carrying out those tasks cheaper. As a result, we may see rapid increases in economic growth (perhaps even to the level we saw during the Industrial Revolution).

If we’re able to partially or fully automate scientific advancement we may see more transformative changes to society and technology.16

That could be just the beginning. We may be able to get computers to eventually automate anything humans can do. This seems like it has to be possible — at least in principle. This is because it seems that, with enough power and complexity, a computer should be able to simulate the human brain. This would itself be a way of automating anything humans can do (if not the most efficient method of doing so).

And as we’ll see in the next section, there are some indications that extensive automation may well be possible through scaling up existing techniques.

Current trends show rapid progress in the capabilities of ML systems

There are three things that are crucial to building AI through machine learning:

1. Good algorithms (e.g. more efficient algorithms are better)
2. Data to train an algorithm
3. Enough computational power (known as compute) to do this training

We spoke to Danny Hernandez, who (at the time) was a research scientist on the Foresight team at OpenAI. Hernandez and his team looked at how two of these inputs (compute and algorithm efficiency) are changing over time.

They found that, since 2012, the amount of compute used for training the largest AI models has been rising exponentially — doubling every 3.4 months.

That is to say, since 2012, the amount of computational power used to train our largest machine learning models has grown by over 1 billion times.

Hernandez and his team also looked at how much compute has been needed to train a neural network to have the same performance as AlexNet (an early image classification algorithm).

They found that the amount of compute required for the same performance has been falling exponentially — halving every 16 months.

So since 2012, the amount of compute required for the same level of performance has fallen by over 100 times. Combined with the increased compute used, that’s a lot of growth.19

It’s hard to say whether these trends will continue, but they speak to incredible gains over the past decade in what it’s possible to do with machine learning.

Indeed, it looks like increasing the size of models (and the amount of compute used to train them) introduces ever more sophisticated behaviour. This is how things like GPT-3 are able to perform tasks they weren’t specifically trained for.

These observations have led to the scaling hypothesis: that we can simply build bigger and bigger neural networks, and as a result we will end up with more and more powerful artificial intelligence, and that this trend of increasing capabilities may increase to human-level AI and beyond.

If this is true, we can attempt to predict how the capabilities of AI technology will increase over time simply by looking at how quickly we are increasing the amount of compute available to train models.

But as we’ll see, it’s not just the scaling hypothesis that suggests we could end up with extremely powerful AI relatively soon — other methods of predicting AI progress come to similar conclusions.

When can we expect transformative AI?

It’s difficult to predict exactly when we will develop AI that we expect to be hugely transformative for society (for better or for worse) — for example, by automating all human work or drastically changing the structure of society.20 But here we’ll go through a few approaches.

One option is to survey experts. Data from the 2019 survey of 300 AI experts implies that there is 20% probability of human-level machine intelligence (which would plausibly be transformative in this sense) by 2036, 50% probability by 2060, and 85% by 2100.21 There are a lot of reasons to be suspicious of these estimates,8 but we take it as one data point.

Ajeya Cotra (a researcher at Open Philanthropy) attempted to forecast transformative AI by comparing modern deep learning to the human brain. Deep learning involves using a huge amount of compute to train a model, before that model is able to perform some task. There’s also a relationship between the amount of compute used to train a model and the amount used by the model when it’s run. And — if the scaling hypothesis is true — we should expect the performance of a model to predictably improve as the computational power used increases. So Cotra used a variety of approaches (including, for example, estimating how much compute the human brain uses on a variety of tasks) to estimate how much compute might be needed to train a model that, when run, could carry out the hardest tasks humans can do. She then estimated when using that much compute would be affordable.

Cotra’s 2022 update on her report’s conclusions estimates that there is a 35% probability of transformative AI by 2036, 50% by 2040, and 60% by 2050 — noting that these guesses are not stable.22

Tom Davidson (also a researcher at Open Philanthropy) wrote a report to complement Cotra’s work. He attempted to figure out when we might expect to see transformative AI based only on looking at various types of research that transformative AI might be like (e.g. developing technology that’s the ultimate goal of a STEM field, or proving difficult mathematical conjectures), and how long it’s taken for each of these kinds of research to be completed in the past, given some quantity of research funding and effort.

Davidson’s report estimates that, solely on this information, you’d think that there was an 8% chance of transformative AI by 2036, 13% by 2060, and 20% by 2100. However, Davidson doesn’t consider the actual ways in which AI has progressed since research started in the 1950s, and notes that it seems likely that the amount of effort we put into AI research will increase as AI becomes increasingly relevant to our economy. As a result, Davidson expects these numbers to be underestimates.

Holden Karnofsky, co-CEO of Open Philanthropy, attempted to sum up the findings of all of the approaches above. He guesses there is more than a 10% chance we’ll see transformative AI by 2036(!), 50% by 2060, and 66% by 2100. And these guesses might be conservative, since they didn’t incorporate what we see as faster-than-expected progress since the estimates were made.

All in all, AI seems to be advancing rapidly. More money and talent is going into the field every year, and models are getting bigger and more efficient.

Even if AI were advancing more slowly, we’d be concerned about it — most of the arguments about the risks from AI (that we’ll get to below) do not depend on this rapid progress.

However, the speed of these recent advances increases the urgency of the issue.

3. Power-seeking AI could pose an existential threat to humanity

We’ve argued so far that we expect AI to be an important — and potentially transformative — new technology.

We’ve also seen reason to think that such transformative AI systems could be built this century.

Now we’ll turn to the core question: why do we think this matters so much?

There could be a lot of reasons. If advanced AI is as transformative as it seems like it’ll be, there will be many important consequences. But here we are going to explain the issue that seems most concerning to us: AI systems could pose risks by seeking and gaining power.

We’ll argue that:

1. It’s likely that we’ll build AI systems that can make and execute plans to achieve goals
2. Advanced planning systems could easily be ‘misaligned’ — in a way that could lead them to make plans that involve disempowering humanity
3. Disempowerment by AI systems would be an existential catastrophe
4. People might deploy AI systems that are misaligned, despite this risk

Thinking through each step, I think there’s something like a 10% chance of an existential catastrophe resulting from power-seeking AI systems this century.23

It’s likely we’ll build advanced planning systems

We’re going to argue that future systems with the following three properties might pose a particularly important threat to humanity:24

1. They have goals and are good at making plans.

   Not all AI systems have goals or make plans to achieve those goals. But some systems (like some chess-playing AI systems) can be thought of in this way. When discussing power-seeking AI, we’re considering planning systems that are relatively advanced, with plans that are in pursuit of some goal(s), and that are capable of carrying out those plans.

2. They have excellent strategic awareness.

   A particularly good planning system would have a good enough understanding of the world to notice obstacles and opportunities that may help or hinder its plans, and respond to these accordingly. Following Carlsmith, we’ll call this strategic awareness, since it allows systems to strategise in a more sophisticated way.

3. They have highly advanced capabilities relative to today’s systems.

   For these systems to actually affect the world, we need them to not just make plans, but also be good at all the specific tasks required to execute those plans.

   Since we’re worried about systems attempting to take power from humanity, we are particularly concerned about AI systems that might be better than humans on one or more tasks that grant people significant power when carried out well in today’s world.

   For example, people who are very good at persuasion and/or manipulation are often able to gain power — so an AI being good at these things might also be able to gain power. Other examples might include hacking into other systems, tasks within scientific and engineering research, as well as business, military, or political strategy.

These systems seem technically possible and we’ll have strong incentives to build them

As we saw above, we’ve already produced systems that are very good at carrying out specific tasks.

We’ve also already produced rudimentary planning systems, like AlphaStar, which skilfully plays the strategy game Starcraft, and MuZero, which plays chess, shogi, and Go.25

We’re not sure whether these systems are producing plans in pursuit of goals per se, because we’re not sure exactly what it means to “have goals.” However, since they consistently plan in ways that achieve goals, it seems like they have goals in some sense.

Moreover, some existing systems seem to actually represent goals as part of their neural networks.26

That said, planning in the real world (instead of games) is much more complex, and to date we’re not aware of any unambiguous examples of goal-directed planning systems, or systems that exhibit high degrees of strategic awareness.

But as we’ve discussed, we expect to see further advances within this century. And we think these advances are likely to produce systems with all three of the above properties.

That’s because we think that there are particularly strong incentives (like profit) to develop these kinds of systems. In short: because being able to plan to achieve a goal, and execute that plan, seems like a particularly powerful and general way of affecting the world.

Getting things done — whether that’s a company selling products, a person buying a house, or a government developing policy — almost always seems to require these skills. One example would be assigning a powerful system a goal and expecting the system to achieve it — rather than having to guide it every step of the way. So planning systems seem likely to be (economically and politically) extremely useful.27

And if systems are extremely useful, there are likely to be big incentives to build them. For example, an AI that could plan the actions of a company by being given the goal to increase its profits (that is, an AI CEO) would likely provide significant wealth for the people involved — a direct incentive to produce such an AI.

As a result, if we can build systems with these properties (and from what we know, it seems like we will be able to), it seems like we are likely to do so.28

Advanced planning systems could easily be dangerously ‘misaligned’

There are reasons to think that these kinds of advanced planning AI systems will be misaligned. That is, they will aim to do things that we don’t want them to do.29

There are many reasons why systems might not be aiming to do exactly what we want them to do. For one thing, we don’t know how, using modern ML techniques, to give systems the precise goals we want (more here).30

We’re going to focus specifically on some reasons why systems might by default be misaligned in such a way that they develop plans that pose risks to humanity’s ability to influence the world — even when we don’t want that influence to be lost.31

What do we mean by “by default”? Essentially, unless we actively find solutions to some (potentially quite difficult) problems, then it seems like we’ll create dangerously misaligned AI. (There are reasons this might be wrong — which we discuss later.)

Three examples of “misalignment” in a variety of systems

It’s worth noting that misalignment isn’t a purely theoretical possibility (or specific to AI) — we see misaligned goals in humans and institutions all the time, and have also seen examples of misalignment in AI systems.32

The democratic political framework is intended to ensure that politicians make decisions that benefit society. But what political systems actually reward is winning elections, so that’s what many politicians end up aiming for.

This is a decent proxy goal — if you have a plan to improve people’s lives, they’re probably more likely to vote for you — but it isn’t perfect. As a result, politicians do things that aren’t clearly the best way of running a country, like raising taxes at the start of their term and cutting them right before elections.

That is to say, the things the system does are at least a little different from what we would, in a perfect world, want it to do: the system is misaligned.

Companies have profit-making incentives. By producing more, and therefore helping people obtain goods and services at cheaper prices, companies make more money.

This is sometimes a decent proxy for making the world better, but profit isn’t actually the same as the good of all of humanity (bold claim, we know). As a result, there are negative externalities: for example, companies will pollute to make money despite this being worse for society overall.

Again, we have a misaligned system, where the things the system does are at least a little different from what we would want it to do.

DeepMind has documented examples of specification gaming: an AI doing well according to its specified reward function (which encodes our intentions for the system), but not doing what researchers intended.

In one example, a robot arm was asked to grasp a ball. But the reward was specified in terms of whether humans thought the robot had been successful. As a result, the arm learned to hover between the ball and the camera, fooling the humans into thinking that it had grasped the ball.33

Source: Christiano et al., 2017

So we know it’s possible to create a misaligned AI system.

Why these systems could (by default) be dangerously misaligned

Here’s the core argument of this article. We’ll use all three properties from earlier: planning ability, strategic awareness, and advanced capabilities.

To start, we should realise that a planning system that has a goal will also develop ‘instrumental goals’: things that, if they occur, will make it easier to achieve an overall goal.

We use instrumental goals in plans all the time. For example, a high schooler planning their career might think that getting into university will be helpful for their future job prospects. In this case, “getting into university” would be an instrumental goal.

A sufficiently advanced AI planning system would also include instrumental goals in its overall plans.

If a planning AI system also has enough strategic awareness, it will be able to identify facts about the real world (including potential things that would be obstacles to any plans), and plan in light of them. Crucially, these facts would include that access to resources (e.g. money, compute, influence) and greater capabilities — that is, forms of power — open up new, more effective ways of achieving goals.

This means that, by default, advanced planning AI systems would have some worrying instrumental goals:

• Self-preservation — because a system is more likely to achieve its goals if it is still around to pursue them (in Stuart Russell’s memorable phrase, “You can’t fetch the coffee if you’re dead”).
• Preventing any changes to the AI system’s goals — since changing its goals would lead to outcomes that are different from those it would achieve with its current goals.
• Gaining power — for example, by getting more resources and greater capabilities.

Crucially, one clear way in which the AI can ensure that it will continue to exist (and not be turned off), and that its objectives will never be changed, would be to gain power over the humans who might affect it (we talk here about how AI systems might actually be able to do that).

What’s more, the AI systems we’re considering have advanced capabilities — meaning they can do one or more tasks that grant people significant power when carried out well in today’s world. With such advanced capabilities, these instrumental goals will not be out of reach, and as a result, it seems like the AI system would use its advanced capabilities to get power as part of the plan’s execution. If we don’t want the AI systems we create to take power away from us this would be a particularly dangerous form of misalignment.

In the most extreme scenarios, a planning AI system with sufficiently advanced capabilities could successfully disempower us completely.

As a (very non-rigorous) intuitive check on this argument, let’s try to apply it to humans.

Humans have a variety of goals. For many of these goals, some form of power-seeking is advantageous: though not everyone seeks power, many people do (in the form of wealth or social or political status), because it’s useful for getting what they want. This is not catastrophic (usually!) because, as human beings:

• We generally feel bound by human norms and morality (even people who really want wealth usually aren’t willing to kill to get it).
• We aren’t that much more capable or intelligent than one another. So even in cases where people aren’t held back by morality, they’re not able to take over the world.

(We discuss whether humans are truly power-seeking later.)

A sufficiently advanced AI wouldn’t have those limitations.

It might be hard to find ways to prevent this sort of misalignment

The point of all this isn’t to say that any advanced planning AI system will necessarily attempt to seek power. Instead, it’s to point out that, unless we find a way to design systems that don’t have this flaw, we’ll face significant risk.

It seems more than plausible that we could create an AI system that isn’t misaligned in this way, and thereby prevent any disempowerment. Here are some strategies we might take (plus, unfortunately, some reasons why they might be difficult in practice):34

• Control the objectives of the AI system. We may be able to design systems that simply don’t have objectives to which the above argument applies — and thus don’t incentivise power-seeking behaviour. For example, we could find ways to explicitly instruct AI systems not to harm humans, or find ways to reward AI systems (in training environments) for not engaging in specific kinds of power-seeking behaviour (and also find ways to ensure that this behaviour continues outside the training environment).

  Carlsmith gives two reasons why doing this seems particularly hard.

  First, for modern ML systems, we don’t get to explicitly state a system’s objectives — instead we reward (or punish) a system in a training environment so that it learns on its own. This raises a number of difficulties, one of which is goal misgeneralisation. Researchers have uncovered real examples of systems that appear to have learned to pursue a goal in the training environment, but then fail to generalise that goal when they operate in a new environment. This raises the possibility that we could think we’ve successfully trained an AI system not to seek power — but that the system would seek power anyway when deployed in the real world.35

  Second, when we specify a goal to an AI system (or, when we can’t explicitly do that, when we find ways to reward or punish a system during training), we usually do this by giving the system a proxy by which outcomes can be measured (e.g. positive human feedback on a system’s achievement). But often those proxies don’t quite work.36 In general, we might expect that even if a proxy appears to correlate well with successful outcomes, it might not do so when that proxy is optimised for. (The examples above of politicians, companies, and the robot arm failing to grasp a ball are illustrations of this.) We’ll look at a more specific example of how problems with proxies could lead to an existential catastrophe here.

  For more on the specific difficulty of controlling the objectives given to deep neural networks trained using self-supervised learning and reinforcement learning, we recommend OpenAI governance researcher Richard Ngo’s discussion of how realistic training processes lead to the development of misaligned goals.

• Control the inputs into the AI system. AI systems will only develop plans to seek power if they have enough information about the world to realise that seeking power is indeed a way to achieve its goals.

• Control the capabilities of the AI system. AI systems will likely only be able to carry out plans to seek power if they have sufficiently advanced capabilities in skills that grant people significant power in today’s world.

But to make any strategy work, it will need to both:

• Retain the usefulness of the AI systems — and so remain economically competitive with less safe systems. Controlling the inputs and capabilities of AI systems will clearly have costs, so it seems hard to ensure that these controls, even if they’re developed, are actually used. But this is also a problem for controlling a system’s objectives. For example, we may be able to prevent power-seeking behaviour by ensuring that AI systems stop to check in with humans about any decisions they make. But these systems might be significantly slower and less immediately useful to people than systems that don’t stop to carry out these checks. As a result, there might still be incentives to use a faster, more initially effective misaligned system (we’ll look at incentives more in the next section).

• Continue to work as the planning ability and strategic awareness of systems improve over time. Some seemingly simple solutions (for example, trying to give a system a long list of things it isn’t allowed to do, like stealing money or physically harming humans) break down as the planning abilities of the systems increase. This is because, the more capable a system is at developing plans, the more likely it is to identify loopholes or failures in the safety strategy — and as a result, the more likely the system is to develop a plan that involves power-seeking.

Ultimately, by looking at the state of the research on this topic, and speaking to experts in the field, we think that there are currently no known ways of building aligned AI systems that seem likely to fulfil both these criteria.

So: that’s the core argument. There are many variants of this argument. Some have argued that AI systems might gradually shape our future via subtler forms of influence that nonetheless could amount to an existential catastrophe; others argue that the most likely form of disempowerment is in fact just killing everyone. We’re not sure how a catastrophe would be most likely to play out, but have tried to articulate the heart of the argument, as we see it: that AI presents an existential risk.

There are definitely reasons this argument might not be right! We go through some of the reasons that seem strongest to us below. But overall it seems very likely to us that, for at least some kinds of advanced planning AI systems, it will be harder to build systems that don’t seek power in this dangerous way than to build systems that do.

Disempowerment by AI systems would be an existential catastrophe

When we say we’re concerned about existential catastrophes, we’re not just concerned about risks of extinction. This is because the source of our concern is rooted in longtermism: the idea that the lives of all future generations matter, and so it’s extremely important to protect their interests.

This means that any event that could prevent all future generations from living lives full of whatever you think makes life valuable (whether that’s happiness, justice, beauty, or general flourishing) counts as an existential catastrophe.

It seems extremely unlikely that we’d be able to regain power over a system that successfully disempowers humanity. And as a result, the entirety of the future — everything that happens for Earth-originating life, for the rest of time — would be determined by the goals of systems that, although built by us, are not aligned with us. Perhaps those goals will create a long and flourishing future, but we see little reason for confidence.37

This isn’t to say that we don’t think AI also poses a risk of human extinction. Indeed, we think making humans extinct is one highly plausible way in which an AI system could completely and permanently ensure that we are never able to regain power.

People might deploy misaligned AI systems despite the risk

Surely no one would actually build or use a misaligned AI if they knew it could have such terrible consequences, right?

Unfortunately, there are at least two reasons people might create and then deploy misaligned AI — which we’ll go through one at a time:38

1. People might think it’s aligned when it’s not

Imagine there’s a group of researchers trying to tell, in a test environment, whether a system they’ve built is aligned. We’ve argued that an intelligent planning AI will want to improve its abilities to effect changes in pursuit of its objective, and it’s almost always easier to do that if it’s deployed in the real world, where a much wider range of actions are available. As a result, any misaligned AI that’s sophisticated enough will try to understand what the researchers want it to do and at least pretend to be doing that, deceiving the researchers into thinking it’s aligned. (For example, a reinforcement learning system might be rewarded for certain apparent behaviour during training, regardless of what it’s actually doing.)

Hopefully, we’ll be aware of this sort of behaviour and be able to detect it. But catching a sufficiently advanced AI in deception seems potentially harder than catching a human in a lie, which isn’t always easy. For example, a sufficiently intelligent deceptive AI system may be able to deceive us into thinking we’ve solved the problem of AI deception, even if we haven’t.

If AI systems are good at deception, and have sufficiently advanced capabilities, a reasonable strategy for such a system could be to deceive humans completely until the system has a way to guarantee it can overcome any resistance to its goals.

2. There are incentives to deploy systems sooner rather than later

We might also expect some people with the ability to deploy a misaligned AI to charge ahead despite any warning signs of misalignment that do come up, because of race dynamics — where people developing AI want to do so before anyone else.

For example, if you’re developing an AI to improve military or political strategy, it’s much more useful if none of your rivals have a similarly powerful AI.

These incentives apply even to people attempting to build an AI in the hopes of using it to make the world a better place.

For example, say you’ve spent years and years researching and developing a powerful AI system, and all you want is to use it to make the world a better place. Simplifying things a lot, say there are two possibilities:

1. This powerful AI will be aligned with your beneficent aims, and you’ll transform society in a potentially radically positive way.
2. The AI will be sufficiently misaligned that it’ll take power and permanently end humanity’s control over the future.

Let’s say you think there’s a 90% chance that you’ve succeeded in building an aligned AI. But technology often develops at similar speeds across society, so there’s a good chance that someone else will soon also develop a powerful AI. And you think they’re less cautious, or less altruistic, so you think their AI will only have an 80% chance of being aligned with good goals, and pose a 20% chance of existential catastrophe. And only if you get there first can your more beneficial AI be dominant. As a result, you might decide to go ahead with deploying your AI, accepting the 10% risk.

This all sounds very abstract. What could an existential catastrophe caused by AI actually look like?

The argument we’ve given so far is very general, and doesn’t really look at the specifics of how an AI that is attempting to seek power might actually do so.

If you’d like to get a better understanding of what an existential catastrophe caused by AI might actually look like, we’ve written a short separate article on that topic. If you’re happy with the high-level abstract arguments so far, feel free to skip to the next section!

What could an existential AI catastrophe actually look like?

4. Even if we find a way to avoid power-seeking, there are still risks

So far we’ve described what a large proportion of researchers in the field6 think is the major existential risk from potential advances in AI, which depends crucially on an AI seeking power to achieve its goals.

If we can prevent power-seeking behaviour, we will have reduced existential risk enormously.

But even if we succeed, there are still risks that AI could pose.

AI could worsen war

We’re concerned that great power conflict could also pose a substantial threat to our world, and advances in AI seem likely to change the nature of war — through lethal autonomous weapons39 or through automated decision making.40

In some cases, great power war could pose an existential threat — for example, if the conflict is nuclear. It’s possible that AI could exacerbate risks of nuclear escalation, although there are also reasons to think AI could decrease this risk.41

Finally, if a single actor produces particularly powerful AI systems, this could be seen as giving them a decisive strategic advantage. For example, the US may produce a planning AI that’s intelligent enough to ensure that Russia or China could never successfully launch another nuclear weapon. This could incentivise a first strike from the actor’s rivals before these AI-developed plans can ever be put into action.

AI could be used to develop dangerous new technology

We expect that AI systems will help increase the rate of scientific progress.42

While there would be clear benefits to this automation — the rapid development of new medicine, for example — some forms of technological development can pose threats, including existential threats, to humanity. This could be through biotechnology43 (see our article on preventing catastrophic pandemics for more) or through some other form of currently unknown but dangerous technology.44

AI could empower totalitarian governments

An AI-enabled authoritarian government could completely automate the monitoring and repression of its citizens, as well as significantly influence the information people see, perhaps making it impossible to coordinate action against such a regime.45

If this became a form of truly stable totalitarianism, this could make people’s lives far worse for extremely long periods of time, making it a particularly scary possible scenario resulting from AI.

Other risks from AI

We’re also concerned about the following issues, though we know less about them:

• Existential threats that result not from the power-seeking behaviour of AI systems, but as a result of the interaction between AI systems. (In order to pose a risk, these systems would still need to be, to some extent, misaligned.)
• Other ways we haven’t thought of in which AI systems could be misused — especially ones that might significantly affect future generations.
• Other moral mistakes made in the design and use of AI systems, particularly if future AI systems are themselves deserving of moral consideration. For example, perhaps we will (inadvertently) create conscious AI systems, which could then suffer in huge numbers. We think this could be extremely important, so we’ve written about it in a separate problem profile.

So, how likely is an AI-related catastrophe?

This is a really difficult question to answer.

There are no past examples we can use to determine the frequency of AI-related catastrophes.

All we have to go off are arguments (like the ones we’ve given above), and less relevant data like the history of technological advances. And we’re definitely not certain that the arguments we’ve presented are completely correct.

Consider the argument we gave earlier about the dangers of power-seeking AI in particular, based off Carlsmith’s report. At the end of his report, Carlsmith gives some rough guesses of the chances that each stage of his argument is correct (conditional on the previous stage being correct):

1. By 2070 it will be possible and financially feasible to build strategically aware systems that can outperform humans on many power-granting tasks, and that can successfully make and carry out plans: Carlsmith guesses there’s a 65% chance of this being true.
2. Given this feasibility, there will be strong incentives to build such systems: 80%.
3. Given both the feasibility and incentives to build such systems, it will be much harder to develop aligned systems that don’t seek power than to develop misaligned systems that do, but which are at least superficially attractive to deploy: 40%.
4. Given all of this, some deployed systems will seek power in a misaligned way that causes over $1 trillion (in 2021 dollars) of damage: 65%.
5. Given all the previous premises, misaligned power-seeking AI systems will end up disempowering basically all of humanity: 40%.
6. Given all the previous premises, this disempowerment will constitute an existential catastrophe: 95%.

Multiplying these numbers together, Carlsmith estimated that there’s a 5% chance that his argument is right and there will be an existential catastrophe from misaligned power-seeking AI by 2070. When we spoke to Carlsmith, he noted that in the year between the writing of his report and the publication of this article, his overall guess at the chance of an existential catastrophe from power-seeking AI by 2070 had increased to >10%.46

The overall probability of existential catastrophe from AI is likely higher than this, because there are other routes to possible catastrophe — like those discussed in the previous section — although our guess is that these other routes are probably a lot less likely to lead to existential catastrophe.

For another estimate, in The Precipice, philosopher and 80,000 Hours Trustee Toby Ord estimated a 1-in-6 risk of existential catastrophe by 2120 (from any cause), and that 60% of this risk comes from misaligned AI — giving a total of a 10% risk of existential catastrophe from misaligned AI by 2120.

A 2021 survey of 44 researchers working on reducing existential risks from AI found the median risk estimate was 32.5% — the highest answer given was 98%, and the lowest was 2%.47 There’s obviously a lot of selection bias here: people choose to work on reducing risks from AI because they think this is unusually important, so we should expect estimates from this survey to be substantially higher than estimates from other sources. But there’s clearly significant uncertainty about how big this risk is, and huge variation in answers.

All these numbers are shockingly, disturbingly high. We’re far from certain that all the arguments are correct. But these are generally the highest guesses for the level of existential risk of any of the issues we’ve examined (like engineered pandemics, great power conflict, climate change, or nuclear war). And, as we’ve explained, we think the arguments for such high estimates of the existential risk posed by AI are persuasive — making risks from AI our top contender for the most pressing problem facing humanity.

5. We can tackle these risks

We think one of the most important things you can do would be to help reduce the gravest risks that AI poses.

This isn’t just because we think these risks are high — it’s also because we think there are real things we can do to reduce these risks.

We know of two broad approaches:

1. Technical AI safety research
2. AI governance research and implementation

For both of these, there are lots of ways to contribute. We’ll go through them in more detail below, but in this section we want to illustrate the point that there are things we can do to address these risks.

Technical AI safety research

The benefits of transformative AI could be huge, and there are many different actors involved (operating in different countries), which means it will likely be really hard to prevent its development altogether.

(It’s also possible that it wouldn’t even be a good idea if we could — after all, that would mean forgoing the benefits as well as preventing the risks.)

As a result, we think it makes more sense to focus on making sure that this development is safe — meaning that it has a high probability of avoiding all the catastrophic failures listed above.

One way to do this is to try to develop technical solutions to prevent the kind of power-seeking behaviour we discussed earlier — this is generally known as working on technical AI safety, sometimes called just “AI safety” for short.

Read more about technical AI safety research below.

AI governance research and implementation

A second strategy for reducing risks from AI is to shape its development through policy, norms-building, and other governance mechanisms.

Good AI governance can help technical safety work, for example by producing safety agreements between corporations, or helping talented safety researchers from around the world move to where they can be most effective. AI governance could also help with other problems that lead to risks, like race dynamics.

But also, as we’ve discussed, even if we successfully manage to make AI do what we want (i.e. we ‘align’ it), we might still end up choosing something bad for it to do! So we need to worry about the incentives not just of the AI systems, but of the human actors using them.

Read more about AI governance research and implementation below.

6. This work is extremely neglected

We estimate there are around 300 people around the world working directly on reducing the chances of an AI-related existential catastrophe. Of these, about two thirds are working on technical AI safety research, with the rest split between strategy (and other governance) research and advocacy.3

In The Precipice, Ord estimated that there was between $10 million and $50 million spent on reducing AI risk in 2020.

That might sound like a lot of money, but we’re spending something like 1,000 times that amount4 on speeding up the development of transformative AI via commercial capabilities research and engineering at large AI labs.

To compare the $50 million spent on AI safety in 2020 to other well-known risks, we’re currently spending several hundreds of billions per year on tackling climate change.

Because this field is so neglected and has such high stakes, we think your impact working on risks from AI could be much higher than working on many other areas — which is why our top two recommended career paths for making a big positive difference in the world are technical AI safety and AI policy research and implementation.

What do we think are the best arguments we’re wrong?

As we said above, we’re not totally sure the arguments we’ve presented for AI representing an existential threat are right. Though we do still think that the chance of catastrophe from AI is high enough to warrant many more people pursuing careers to try to prevent such an outcome, we also want to be honest about the arguments against doing so, so you can more easily make your own call on the question.

Here we’ll cover the strongest reasons (in our opinion) to think this problem isn’t particularly pressing. In the next section we’ll cover some common objections that (in our opinion) hold up less well, and explain why.

The longer we have before transformative AI is developed, the less pressing it is to work now on ways to ensure that it goes well. This is because the work of others in the future could be much better or more relevant than the work we are able to do now.

Also, if it takes us a long time to create transformative AI, we have more time to figure out how to make it safe. The risk seems much higher if AI developers will create transformative AI in the next few decades.

The primary reason that transformative AI could be far away is that it seems plausible that the first transformative AI won’t be based on current deep learning methods. (AI Impacts have documented arguments that current methods won’t be able to produce AI that has human-level intelligence.) This could mean that some of our current research might not end up being useful (and also — depending on what method ends up being used — could make the arguments for risk less worrying).

But even if current methods aren’t going to produce transformative AI, there are still reasons to think we don’t have that much time to work on this problem. For example, finding technical solutions to prevent power-seeking behaviour might be extremely difficult.

Also, even if we can’t use deep learning to build transformative AI, we may well come up with other methods of doing so. As a result, many AI experts guess we’ll get transformative AI remarkably soon.

We think that transformative AI is sufficiently likely in the next 20–80 years that it is well worth it (in expected value terms) to work on this issue now. Perhaps future generations will take care of it, and all the work we’d do now will be in vain — we hope so! But it might not be prudent to take that risk.

If the best AI we have improves gradually over time (rather than AI capabilities remaining fairly low for a while and then suddenly increasing), we’re likely to end up with ‘warning shots’: we’ll notice forms of misaligned behaviour in fairly weak systems, and be able to correct for it before it’s too late.

In such a gradual scenario, we’ll have a better idea about what form powerful AI might take (e.g. whether it will be built using current deep learning techniques, or something else entirely), which could significantly help with safety research. There will also be more focus on this issue by society as a whole, as the risks of AI become clearer.

So if gradual development of AI seems more likely, the risk seems lower.

But it’s very much not certain that AI development will be gradual, or if it is, gradual enough for the risk to be noticeably lower. And even if AI development is gradual, there could still be significant benefits to having plans and technical solutions in place well in advance. So overall we still think it’s extremely valuable to attempt to reduce the risk now.

If you want to learn more, you can read AI Impacts’ work on arguments for and against discontinuous (i.e. non-gradual) progress in AI development, and Toby Ord and Owen Cotton-Barratt on strategic implications of slower AI development.

Making something have goals aligned with human designers’ ultimate objectives and making something useful seem like very related problems. If so, perhaps the need to make AI useful will drive us to produce only aligned AI — in which case the alignment problem is likely to be solved by default.

Ben Garfinkel gave a few examples of this on our podcast:

• You can think of a thermostat as a very simple AI that attempts to keep a room at a certain temperature. The thermostat has a metal strip in it that expands as the room heats, and cuts off the current once a certain temperature has been reached. This piece of metal makes the thermostat act like it has a goal of keeping the room at a certain temperature, but also makes it capable of achieving this goal (and therefore of being actually useful).
• Imagine you’re building a cleaning robot with reinforcement learning techniques — that is, you provide some specific condition under which you give the robot positive feedback. You might say something like, “The less dust in the house, the more positive the feedback.” But if you do this, the robot will end up doing things you don’t want — like ripping apart a cushion to find dust on the inside. Probably instead you need to use techniques like those being developed by people working on AI safety (things like watching a human clean a house and letting the AI figure things out from there). So people building AIs will be naturally incentivised to also try to make them aligned (and so in some sense safe), so they can do their jobs.

If we need to solve the problem of alignment anyway to make useful AI systems, this significantly reduces the chances we will have misaligned but still superficially useful AI systems. So the incentive to deploy a misaligned AI would be a lot lower, reducing the risk to society.

That said, there are still reasons to be concerned. For example, it seems like we could still be susceptible to problems of AI deception.

And, as we’ve argued, AI alignment is only part of the overall issue. Solving the alignment problem isn’t the same thing as completely eliminating existential risk from AI, since aligned AI could also be used to bad ends — such as by authoritarian governments.

As with many research projects in their early stages, we don’t know how hard the alignment problem — or other AI problems that pose risks — are to solve. Someone could believe there are major risks from machine intelligence, but be pessimistic about what additional research or policy work will accomplish, and so decide not to focus on it.

This is definitely a reason to potentially work on another issue — the solvability of an issue is a key part of how we try to compare global problems. For example, we’re also very concerned about risks from pandemics, and it may be much easier to solve that issue.

That said, we think that given the stakes, it could make sense for many people to work on reducing AI risk, even if you think the chance of success is low. You’d have to think that it was extremely difficult to reduce risks from AI in order to conclude that it’s better just to let the risks materialise and the chance of catastrophe play out.

At least in our own case at 80,000 Hours, we want to keep trying to help with AI safety — for example, by writing profiles like this one — even if the chance of success seems low (though in fact we’re overall pretty optimistic).

There are some reasons to think that the core argument that any advanced, strategically aware planning system will by default seek power (which we gave here) isn’t totally right.48

1. There are simple examples we can create where it seems like the reasoning we gave earlier doesn’t quite hold up. We argued that we can expect AI systems to do things that seem generally useful to their overall goal. But consider, for example, an autonomous toy car that can move around only if its engine is on. For most possible goals, it seems like it would be useful for the car to be able to move around, so we should expect the car to turn its engine on. But it also seems likely to be pretty easy to train the car to keep its engine off: we can just give it some negative feedback to turning the engine on, even if we also had given the car some other goals. Simple examples like this shed some doubt on the idea that, just because a particular action is instrumentally useful, we won’t be able to find ways to prevent it. (For more on this example, see page 25 of Garfinkel’s review of Carlsmith’s report.)

2. Not all misaligned power-seeking behaviour seems like it will actually disempower all of humanity. For example, perhaps an AI system will reason that it would be easier to take power away only from people who might oppose whatever goal it has. That said, the more advanced an AI’s power-seeking capabilities (e.g. those relevant to the seven possible power-seeking dynamics discussed here), the more likely it is that an AI system would be successful at an attempt to disempower all of humanity. Disempowering all of humanity appears to be a particularly robust way to ensure that AI systems will always be able to achieve their goals. So if more capable systems are sufficiently likely to be successful at disempowering all of humanity, a good planning system may decide that this is worth including in its plans.

3. We might expect that, by default, when training some ML system, we will select against any observable power-seeking behaviour, reducing the chances of this behaviour when the AI is deployed. Unfortunately, it’s not clear how likely this is to work, especially given that, for advanced enough systems, it seems plausible we’d see deceptive behaviour — meaning that we might not be able to observe any worrying behaviour before it’s too late.49

4. Humans seem like strategically aware agents capable of planning, but we aren’t always power-seeking. We obviously care about having basics like food and shelter, and many people go to great lengths for more money, status, education, or even formal power. But some humans choose not to pursue these goals, and pursuing them doesn’t seem to correlate with intelligence.

   However, this doesn’t mean that the argument that there will be an incentive to seek power is wrong. Most people do face and act on incentives to gain forms of influence via wealth, status, promotions, and so on. And we can explain the observation that humans don’t usually seek huge amounts of power by observing that we aren’t usually in circumstances that make the effort worth it.

   For example, most people don’t try to start billion-dollar companies — you probably won’t succeed, and it’ll cost you a lot of time and effort.

   But you’d still walk across the street to pick up a billion-dollar cheque.

   That said, the absence of power-seeking in many humans does point to the idea that, in principle, alignment research into preventing power-seeking in AIs could succeed. This is good news! But for the moment, we don’t actually know how to prevent this power-seeking behaviour.

   These sorts of objections are part of the reason why, as we saw earlier, Carlsmith thinks there’s only a 40% chance that — even if building advanced planning AI systems is both feasible and desirable — it will be much harder to develop aligned systems that don’t seek power than to develop misaligned systems that do seek power (and that are, at least on the surface, attractive to deploy).

   And in general, these kinds of points are part of why we’re far from sure that each of the steps of the argument go through.

   But again, even given these objections, we still think the chances of the conclusion overall being right — or at least pointing to something very significant — are high enough to merit significant concern.

Arguments against working on AI risk to which we think there are strong responses

We’ve just discussed the major objections to working on AI risk that we think are most persuasive. In this section, we’ll look at objections that we think are less persuasive, and give some reasons why.

People have been saying since the 1950s that artificial intelligence smarter than humans is just around the corner.

But it hasn’t happened yet.

One reason for this could be that it’ll never happen. Some have argued that producing artificial general intelligence is fundamentally impossible. Others think it’s possible, but unlikely to actually happen, especially not with current deep learning methods.

Overall, we think the existence of human intelligence shows it’s possible in principle to create artificial intelligence. And the speed of current advances isn’t something we think would have been predicted by those who thought that we’ll never develop powerful, general AI.

But most importantly, the idea that you need fully general intelligent AI systems for there to be a substantial existential risk is a common misconception.

The argument we gave earlier relied on AI systems being as good or better than humans in a subset of areas: planning, strategic awareness, and areas related to seeking and keeping power. So as long as you think all these things are possible, the risk remains.

And even if no single AI has all of these properties, there are still ways in which we might end up with systems of ‘narrow’ AI systems that, together, can disempower humanity. For example, we might have a planning AI that develops plans for a company, a separate AI system that measures things about the company, another AI system that attempts to evaluate plans from the first AI by predicting how much profit each will make, and further AI systems that carry out those plans (for example, by automating the building and operation of factories). Considered together, this system as a whole has the capability to form and carry out plans to achieve some goal, and potentially also has advanced capabilities in areas that help it seek power. So the risk remains, even from systems of many interacting AIs.

It might just be really, really hard.

Stopping people and computers from running software is already incredibly difficult.

Think about how hard it would be to shut down Google’s web services. Google’s data centres have millions of servers over 34 different locations, many of which are running the same sets of code. And these data centres are absolutely crucial to Google’s bottom line, so even if Google could decide to shut down their entire business, they probably wouldn’t.

Or think about how hard it is to get rid of computer viruses that autonomously spread between computers across the world.

Ultimately, we think any dangerous power-seeking AI system will be looking for ways to not be turned off, which makes it more likely we’ll be in one of these situations, rather than in a case where we can just unplug a single machine.

That said, we absolutely should try to shape the future of AI such that we can ‘unplug’ powerful AI systems.

There may be ways we can develop systems that let us turn them off. But for the moment, we’re not sure how to do that.

Ensuring that we can turn off potentially dangerous AI systems could be a safety measure developed by technical AI safety research, or it could be the result of careful AI governance, such as planning coordinated efforts to stop autonomous software once it’s running.

We could (and should!) definitely try.

If we could successfully ‘sandbox’ an advanced AI — that is, contain it to a training environment with no access to the real world until we were very confident it wouldn’t do harm — that would help our efforts to mitigate AI risks tremendously.

But there are a few things that might make this difficult.

For a start, we might only need one failure — like one person to remove the sandbox, or one security vulnerability in the sandbox we hadn’t noticed — for the AI system to begin affecting the real world.

Moreover, this solution doesn’t scale with the capabilities of the AI system. This is because:

• More capable systems are more likely to be able to find vulnerabilities or other ways of leaving the sandbox (e.g. threatening or coercing humans).
• Systems that are good at planning might attempt to deceive us into deploying them.

So the more dangerous the AI system, the less likely sandboxing is to be possible. That’s the opposite of what we’d want from a good solution to the risk.

For some definitions of “truly intelligent” — for example, if true intelligence includes a deep understanding of morality and a desire to be moral — this would probably be the case.

But if that’s your definition of truly intelligent, then it’s not truly intelligent systems that pose a risk. As we argued earlier, it’s advanced systems that can plan and have strategic awareness that pose risks to humanity.

With sufficiently advanced strategic awareness, an AI system’s excellent understanding of the world may well encompass an excellent understanding of people’s moral beliefs. But that’s not a strong reason to think that such a system would act morally.

For example, when we learn about other cultures or moral systems, that doesn’t necessarily create a desire to follow their morality. A scholar of the Antebellum South might have a very good understanding of how 19th century slave owners justified themselves as moral, but would be very unlikely to defend slavery.

AI systems with excellent understandings of human morality could be even more dangerous than AIs without such understanding: the AI system could act morally at first as a way to deceive us into thinking that it is safe.

Some of the more cartoon-style illustrations of risks from AI involve giving the AI goals that are clearly not what we really want — e.g. “produce as many paperclips as possible” is no one’s real, overriding goal. So it might seem like we can just easily avoid the problems discussed above by making sure to only give AI systems goals that we actually want achieved.

And yes — if we were able to give systems objectives that really, precisely represented what we want to happen, and we knew that it was only those objectives that the systems would pursue, then the risks posed by AI would seem far, far lower.

But, as we’ve discussed, for many goals an advanced planning system could have, there are other additional instrumental goals the system is likely to develop. In particular:

• Maintaining its own existence.
• Gaining more influence and power so that the system is more able to affect the environment to achieve its goal.

This is especially true for systems with ambitious goals, which will be associated with the greatest promises of AI systems, and therefore those goals which we might have very strong incentives to give to advanced planning systems.

We also gave a few reasons above for why controlling these goals could be hard (in short: there might be problems with measurable proxies used to specify goals, and difficulties arise from modern ML systems’ implicit goals developing through training rather than explicit programming). We also looked at why we’d need any solution to retain the usefulness of the AI system being developed, and why many seemingly simple ways of just not giving a system bad goals won’t continue to work as we improve the planning abilities and strategic awareness of AI systems.

There’s also a concern that if one group does actually succeed in giving an AI only “correct” goals, other groups may not — meaning we still end up with highly capable, power-seeking artificial intelligence. And once we have that, the chances of an existential catastrophe of some kind will be elevated.

So overall, while we’d certainly like to not give AI systems bad goals, the problem of finding a way to do this seems pretty similar to the problem of building safe AI. And at the moment, that problem doesn’t have a solution.

There are definitely dangers from current artificial intelligence.

For example, data used to train neural networks often contains hidden biases. This means that AI systems can learn these biases — and this can lead to racist and sexist behaviour.

There are other dangers too. Our earlier discussion on nuclear war explains a threat which doesn’t require AI systems to have particularly advanced capabilities.

But we don’t think the fact that there are also risks from current systems is a reason not to prioritise reducing existential threats from AI, if they are sufficiently severe.

As we’ve discussed, future systems — not necessarily superintelligence or totally general intelligence, but systems advanced in their planning and power-seeking capabilities — seem like they could pose threats to the existence of the entirety of humanity. And it also seems somewhat likely that we’ll produce such systems this century.

What’s more, lots of technical AI safety research is also relevant to solving problems with existing AI systems. For example, some research focuses on ensuring that ML models do what we want them to, and will still do this as their size and capabilities increase; other research tries to work out how and why existing models are making the decisions and taking the actions that they do.

As a result, at least in the case of technical research, the choice between working on current threats and future risks may look more like a choice between only ensuring that current models are safe, or instead finding ways to ensure that current models are safe that will also continue to work as AI systems become more complex and more intelligent.

Ultimately, we have limited time in our careers, so choosing which problem to work on could be a huge way of increasing your impact. When there are such substantial threats, it seems reasonable for many people to focus on addressing these worst-case possibilities.

You’d think.

But, as we’ve argued, there are incentives to build systems that can plan (and strategically execute their plans), as well as incentives to use these systems — even if we’re not sure they are aligned. People might be wrong about how dangerous a system is, or accept small risks for everyone in return for big upsides.

And there are examples of people building and developing really dangerous systems in other areas.

Nuclear weapons are an obvious example: they were built despite the risk that they could cause human extinction. For example, Enrico Fermi, one of the leading experimental physicists on the Manhattan Project, thought it was possible that the Trinity Test in July 1945 could ignite the atmosphere and destroy all life on Earth — but they went ahead with the test anyway. And while, in general, there aren’t strong incentives to actually use nuclear weapons, there are plenty of times we’ve come extremely close to nuclear war.

We’ve also written about potential global catastrophic biological risks. Gain-of-function research is research that deliberately engineers more dangerous pathogens that could potentially cause pandemics, possibly ones far worse than COVID-19. Scientists do this research in spite of the risk we’d see if such pathogens ever escaped the lab.

AI has the potential to be radically transformative — and as a result, make a huge amount of money for the companies and states that develop it. This incentive seems substantially bigger than the monetary incentives behind nuclear weapons or gain-of-function research.

That is to say, for a short while, an AI that will eventually cause an existential catastrophe could also make its developers unimaginably wealthy.50

So, if anything, we might expect people to be less careful with AI than with other dangerous technologies.

It’s undoubtedly true that some people are drawn to thinking about AI safety because they like computers and science fiction — as with any other issue, there are people working on it not because they think it’s important, but because they think it’s cool.

But, for many people, working on AI safety comes with huge reluctance.

For me, and many of us at 80,000 Hours, spending our limited time and resources working on any cause that affects the long-run future — and therefore not spending that time on the terrible problems in the world today — is an incredibly emotionally difficult thing to do.

But we’ve gradually investigated these arguments (in the course of trying to figure out how we can do the most good), and over time both gained more expertise about AI and became more concerned about the risk.

We think scepticism is healthy, and are far from certain that these arguments completely work. So while this suspicion is definitely a reason to dig a little deeper, we hope that, ultimately, this worry won’t be treated as a reason to deprioritise what may well be the most important problem of our time.

That something sounds like science fiction isn’t a reason in itself to dismiss it outright. There are loads of examples of things first mentioned in sci-fi that then went on to actually happen (this list of inventions in science fiction contains plenty of examples).

There are even a few such cases involving technology that are real existential threats today:

• In his 1914 novel The World Set Free, H. G. Wells predicted atomic energy fueling powerful explosives — 20 years before we realised there could in theory be nuclear fission chain reactions, and 30 years before nuclear weapons were actually produced. In the 1920s and 1930s, Nobel Prize–winning physicists Millikan, Rutherford, and Einstein all predicted that we would never be able to use nuclear power. Nuclear weapons were literal science fiction before they were reality.
• In the 1964 film Dr. Strangelove, the US builds a doomsday machine that would automatically trigger an extinction-level nuclear event in response to a nuclear strike, but keeps it secret. Dr Strangelove points out that keeping it secret rather reduces its deterrence effect. But we now know that in the 1980s the USSR built an extremely similar system… and kept it secret.

Moreover, there are top academics and researchers working on preventing these risks from AI — at MIT, Cambridge, Oxford, UC Berkeley, and elsewhere. Two of the world’s top AI labs (DeepMind and OpenAI) have teams explicitly dedicated to working on technical AI safety. Researchers from these places helped us with this article.

It’s totally possible all these people are wrong to be worried, but the fact that so many people take this threat seriously undermines the idea that this is merely science fiction.

It’s reasonable when you hear something that sounds like science fiction to want to investigate it thoroughly before acting on it. But having investigated it, if the arguments seem solid, then simply sounding like science fiction is not a reason to dismiss them.

We never know for sure what’s going to happen in the future. So, unfortunately for us, if we’re trying to have a positive impact on the world, that means we’re always having to deal with at least some degree of uncertainty.

We also think there’s an important distinction between guaranteeing that you’ve achieved some amount of good and doing the very best you can. To achieve the former, you can’t take any risks at all — and that could mean missing out on the best opportunities to do good.

When you’re dealing with uncertainty, it makes sense to roughly think about the expected value of your actions: the sum of all the good and bad potential consequences of your actions, weighted by their probability.

Given the stakes are so high, and the risks from AI aren’t that low, this makes the expected value of helping with this problem high.

We’re sympathetic to the concern that if you work on AI safety, you might end up doing not much at all when you might have done a tremendous amount of good working on something else — simply because the problem and our current ideas about what to do about it are so uncertain.

But we think the world will be better off if we decide that some of us should work on solving this problem, so that together we have the best chance of successfully navigating the transition to a world with advanced AI rather than risking an existential crisis.

And it seems like an immensely valuable thing to try.

Pascal’s mugging is a thought experiment — a riff on the famous Pascal’s wager — where someone making decisions using expected value calculations can be exploited by claims that they can get something extraordinarily good (or avoid something extraordinarily bad), with an extremely low probability of succeeding.

The story goes like this: a random mugger stops you on the street and says, “Give me your wallet or I’ll cast a spell of torture on you and everyone who has ever lived.” You can’t rule out with 100% probability that he won’t — after all, nothing’s 100% for sure. And torturing everyone who’s ever lived is so bad that surely even avoiding a tiny, tiny probability of that is worth the $40 in your wallet? But intuitively, it seems like you shouldn’t give your wallet to someone just because they threaten you with something completely implausible.

Analogously, you could worry that working on AI safety means giving your valuable time to avoid a tiny, tiny chance of catastrophe. Working on reducing risks from AI isn’t free — the opportunity cost is quite substantial, as it means you forgo working on other extremely important things, like reducing risks from pandemics or ending factory farming.

Here’s the thing though: while there’s lots of value at stake — perhaps the lives of everybody alive today, and the entirety of the future of humanity — it’s not the case that the probability that you can make a difference by working on reducing risks from AI is small enough for this argument to apply.

We wish the chance of an AI catastrophe was that vanishingly small.

Instead, we think the probability of such a catastrophe (we think, around 10% this century) is much, much larger than things that people try to prevent all the time — such as fatal plane crashes, which happen in 0.00002% of flights.

What really matters, though, is the extent to which your work can reduce the chance of a catastrophe.

Let’s look at working on reducing risks from AI. For example, if:

1. There’s a 10% chance of an AI-related existential catastrophe by 2100
2. There’s a 30% chance that we can find a way to prevent this by technical research
3. Five people working on technical AI safety raises the chances of solving the problem by 1% of that 30% (so 0.3 percentage points)

Then each person involved has a 0.0006 percentage point share in preventing this catastrophe.

Other ways of acting altruistically involve similarly sized probabilities.

The chances of a volunteer campaigner swinging a US presidential election is somewhere between 0.001% and 0.00001%. But you can still justify working on a campaign because of the large impact you expect you’d have on the world if your preferred candidate won.

You have even lower chances of wild success from things like trying to reform political institutions, or working on some very fundamental science research to build knowledge that might one day help cure cancer.

Overall, as a society, we may be able to reduce the chance of an AI-related catastrophe all the way down from 10% (or higher) to close to zero — that’d be clearly worth it for a group of people, so it has to be worth it for the individuals, too.

We wouldn’t want to just not do fundamental science because each researcher has a low chance of making the next big discovery, or not do any peacekeeping because any one person has a low chance of preventing World War III. As a society, we need some people working on these big issues — and maybe you can be one of them.

What you can do concretely to help

As we mentioned above, we know of two main ways to help reduce existential risks from AI:

1. Technical AI safety research
2. AI strategy/policy research and implementation

The biggest way you could help would be to pursue a career in either one of these areas, or in a supporting area.

The first step is learning a lot more about the technologies, problems, and possible solutions. We’ve collated some lists of our favourite resources here, and our top recommendation is to take a look at the technical alignment curriculum from AGI Safety Fundamentals.

If you decide to pursue a career in this area, we’d generally recommend working at an organisation focused on specifically addressing this problem (though there are other ways to help besides working at existing organisations, as we discuss briefly below).

Technical AI safety

Approaches

There are lots of approaches to technical AI safety, including:

• Working on actual proposals to make future AI systems safe, such as Iterated Distillation and Amplification, AI safety via debate, building AI assistants that are uncertain about our goals and learn them by interacting with us, or finding ways to get AI systems trained with stochastic gradient descent to report truthfully what they know. For more, see Hubinger’s 11 possible proposals for building safe advanced AI.
• Interpretability research — trying to help us actually understand what’s going on inside a neural network. This work has had some success already (see, for example Zoom In: An Introduction to Circuits by Olah et al.). For more, see Hubinger’s A transparency and interpretability tech tree for an overview of this research, and Nanda’s A Longlist of Theories of Impact for Interpretability for an overview of how interpretability research could reduce existential risk from AI.
• Research to increase the robustness of neural networks — making sure that the sorts of behaviour neural networks display when exposed to one set of inputs continues when exposed to inputs they haven’t previously been exposed to, in order to prevent AI systems changing to unsafe behaviour.
• Work to build cooperative AI — finding ways to ensure that even if individual AI systems seem safe, they don’t produce bad outcomes through interacting with other human or AI systems. For more, see Open Problems in Cooperative AI by Dafoe et al. or the Cooperative AI Foundation.

See Neel Nanda’s overview of the AI alignment landscape for more details.

Key organisations

AI labs in industry that have technical safety teams, or are focused entirely on safety:

• Anthropic is an AI safety company working on building interpretable and safe AI systems. They focus on empirical AI safety research. Anthropic cofounders Daniela and Dario Amodei gave an interview about the lab on the Future of Life Institute podcast. On our podcast, we spoke to Chris Olah, who leads Anthropic’s research into interpretability, and Nova DasSarma, who works on information security at Anthropic.
• DeepMind is probably the largest and most well-known research group developing general artificial machine intelligence, and is famous for its work creating AlphaGo, AlphaZero, and AlphaFold. It is not principally focused on safety, but has two teams focused on AI safety. DeepMind is owned by Alphabet (Google’s parent company).
• The Center for AI Safety is a nonprofit that does technical research and promotion of safety in the wider machine learning community.
• OpenAI, founded in 2015, is a lab that is trying to build artificial general intelligence that is safe and benefits all of humanity. OpenAI is well known for its language models like GPT-3, and has a safety team and a governance team. Jan Leike (head of the alignment team) has some blog posts on how he thinks about AI alignment.
• Ought is a machine learning lab building Elicit, an AI research assistant. Their aim is to align open-ended reasoning by learning human reasoning steps, and to direct AI progress towards helping with evaluating evidence and arguments.
• Redwood Research is an AI safety research organisation, whose first big project attempted to make sure language models (like GPT-3) produce output following certain rules with very high probability, in order to address failure modes too rare to show up in standard training.

Conceptual AI safety labs:

• The Alignment Research Center (ARC) is attempting to produce alignment strategies that could be adopted in industry today while also being able to scale to future systems. They focus on conceptual work, developing strategies that could work for alignment and which may be promising directions for empirical work, rather than doing empirical AI work themselves. Their first project was releasing a report on Eliciting Latent Knowledge, the problem of getting advanced AI systems to honestly tell you what they believe (or ‘believe’) about the world. On our podcast, we interviewed ARC founder Paul Christiano about his research (before he founded ARC).
• The Center on Long-Term Risk works to address worst-case risks from advanced AI. They focus on conflict between AI systems.
• The Machine Intelligence Research Institute was one of the first groups to become concerned about the risks from machine intelligence in the early 2000s, and its team has published a number of papers on safety issues and how to resolve them.

AI safety in academia:

• The Algorithmic Alignment Group in the Computer Science and Artificial Intelligence Laboratory at MIT, led by Dylan Hadfield-Menell.
• The Center for Human-Compatible AI at UC Berkeley, led by Stuart Russell, which focuses on academic research to ensure AI is safe and beneficial to humans. (Our podcast with Stuart Russell examines his approach to provably beneficial AI.)
• Jacob Steinhardt’s research group in the Department of Statistics at UC Berkeley.
• Sam Bowman’s research group at the NYU Machine Learning for Language.
• David Krueger’s research group at the Computational and Biological Learning Laboratory at the University of Cambridge.
• The Foundations of Cooperative AI Lab at Carnegie Mellon University.
• The Future of Humanity Institute at the University of Oxford has an AI safety research group.

If you’re interested in learning more about technical AI safety as an area — e.g. the different techniques, schools of thought, and threat models — our top recommendation is to take a look at the technical alignment curriculum from AGI Safety Fundamentals.

We discuss this path in more detail here (though it’s outdated compared to this problem profile):

Career review of technical AI safety research

Alternatively, if you’re looking for something more concrete and step-by-step (with very little in the way of introduction), check out this detailed guide to pursuing a career in AI alignment.

It’s important to note that you don’t have to be an academic or an expert in AI or AI safety to contribute to AI safety research. For example, software engineers are needed at many places conducting technical safety research, and we also highlight more roles below.

AI governance and strategy

Approaches

Quite apart from the technical problems, we face a host of governance issues, which include:

• Coordination problems that are increasing the risks from AI (e.g. there could be incentives to use AI for personal gain in ways that can cause harm, or race dynamics that reduce incentives for careful and safe AI development).
• Risks from accidents or misuse of AI that would be dangerous even if we are able to prevent power-seeking behaviour (as discussed above).
• A lack of clarity on how and when exactly risks from AI (particularly power-seeking AI) might play out.
• A lack of clarity on which intermediate goals we could pursue that, if achieved, would reduce existential risk from AI.

To tackle these, we need a combination of research and policy.51

We are in the early stages of figuring out the shape of this problem and the most effective ways to tackle it. So it’s crucial that we do more research. This includes forecasting research into what we should expect to happen, and strategy and policy research into the best ways of acting to reduce the risks.

But also, as AI begins to impact our society more and more, it’ll be crucial that governments and corporations have the best policies in place to shape its development. For example, governments might be able to enforce agreements not to cut corners on safety, further the work of researchers who are less likely to cause harm, or cause the benefits of AI to be distributed more evenly. So there eventually might be a key role to be played in advocacy and lobbying for appropriate AI policy — though we’re not yet at the point of knowing what policies would be useful to implement.

Key organisations

AI strategy and policy organisations:

• AI Impacts attempts to find answers to all sorts of relevant questions about the future of AI, like “How likely is a sudden jump in AI progress at around human-level performance?”
• The AI Security Initiative at UC Berkeley’s Center for Long-Term Cybersecurity.
• The Centre for the Governance of AI (GovAI) aims to build a global research community, dedicated to helping humanity navigate the transition to a world with advanced AI. On our podcast we’ve spoken to Ben Garfinkel, acting director of GovAI, about some weaknesses of classic AI risk arguments, as well as Allan Dafoe, president of GovAI and leader of DeepMind’s Long-Term Strategy and Governance team, about the destabilising effects of AI.
• The Centre for Long-Term Resilience is a UK think tank focused on existential threats, including those from AI.
• The Center for Security and Emerging Technology at Georgetown researches the foundations of AI (talent, data, and computational power). It focuses on how AI can be used in national security. Listen to our podcast with Helen Toner, their Director of Strategy, for more.
• The Centre for the Study of Existential Risk at the University of Cambridge has a group considering the governance of AI.
• DeepMind and OpenAI both have policy teams (listen to our podcast with members of the OpenAI policy team and our podcast with the head of DeepMind’s governance team, Allan Dafoe).
• The Future of Life Insitute advocates for awareness of AI risk within the academic community and gives out grants for work focused on AI safety.
• The Future of Humanity Institute at the University of Oxford has a macrostrategy research group that considers the future of AI and its contribution to existential risk.
• The Leverhulme Centre for the Future of Intelligence is an interdisciplinary research centre at the University of Cambridge focusing on the impacts of AI on humanity.
• Open Philanthropy provides grants to organisations working on altruistic issues. As a result they have research teams looking at the issues they focus on, including a team looking at potential risks from advanced AI. On our podcast, we spoke to Holden Karnofsky, co-CEO of Open Philanthropy, about his views on risks from AI. (Note: Open Philanthropy is 80,000 Hours’ biggest funder.)
• Rethink Priorities has a team focused on AI governance and strategy.

If you’re interested in learning more about AI governance, our top recommendation is to take a look at the governance curriculum from AGI safety fundamentals.

We discuss this path in more detail here:

Career review of AI strategy and policy careers

Also note: it could be particularly important for people with the right personal fit to work on AI strategy and governance in China.

Complementary (yet crucial) roles

Even in a research organisation, around half of the staff will be doing other tasks essential for the organisation to perform at its best and have an impact. Having high-performing people in these roles is crucial.

We think the importance of these roles is often underrated because the work is less visible. So we’ve written several career reviews on these areas to help more people enter these careers and succeed, including:

• Operations management to help impactful organisations grow and function as effectively as possible.
• Research management at an AI safety research organisation.
• Being an executive assistant to someone who’s doing really important work on safety and governance.
• Other non-technical roles in leading AI labs.

Other ways to help

AI safety is a big problem and it needs help from people doing a lot of different kinds of work.

One major way to help is to work in a role that directs funding or people towards AI risk, rather than working on the problem directly. We’ve reviewed a few career paths along these lines, including:

• Founding new projects — in this case, starting new initiatives aimed at reducing risks from advanced AI.
• Being a grantmaker to fund promising projects focused on reducing catastrophic AI risk.
• Working in communication roles
• Helping to build communities of people working on this problem. The most relevant community is the AI safety community itself, but it could also be impactful to help build the community of people working on the world’s most pressing problems (including risks from AI).

There are ways all of these could go wrong, so the first step is to become well-informed about the issue.

There are also other technical roles besides safety research that could help contribute, like:

• Working in information security to protect AI (or the results of key experiments) from misuse, theft, or tampering.
• Becoming an expert in AI hardware as a way of steering AI progress in safer directions.

You can read about all these careers — why we think they’re helpful, how to enter them, and how you can predict whether they’re a good fit for you — on our career reviews page.

Want one-on-one advice on pursuing this path?

We think that the risks posed by the development of AI may be the most pressing problem the world currently faces. If you think you might be a good fit for any of the above career paths that contribute to solving this problem, we’d be especially excited to advise you on next steps, one-on-one.

We can help you consider your options, make connections with others working on reducing risks from AI, and possibly even help you find jobs or funding opportunities — all for free.

APPLY TO SPEAK WITH OUR TEAM

Find vacancies on our job board

Top resources to learn more

We’ve hit you with a lot of further reading throughout this article — here are a few of our favourites:

• AI could defeat all of us combined and the “most important century” blog post series by Holden Karnofsky, co-CEO of Open Philanthropy, argues that the 21st century could be the most important century ever for humanity as a result of AI.
• Why AI alignment could be hard with modern deep learning by Open Philanthropy researcher Cotra is a gentle introduction to how risks from power-seeking AI could play out with current machine learning methods. Without specific countermeasures, the easiest path to transformative AI likely leads to AI takeover, also by Cotra, provides a much more detailed description of how risks could play out (which we’d recommend for people familiar with ML).
• AGI safety from first principles provides OpenAI governance researcher Richard Ngo’s perspective on how to think about risks from artificial general intelligence.
• Is power-seeking AI an existential risk? by Open Philanthropy researcher Joseph Carlsmith is an in-depth look covering exactly how and why AI could cause the disempowerment of humanity (but watch out — it’s even longer than this article!). For a shorter summary, see Carlsmith’s talk on the same topic.
• Distinguishing AI takeover scenarios by Sam Clarke and Sammy Martin summarises various ways in which AI could go wrong.
• AI governance: Opportunity and theory of impact by DeepMind governance lead Allan Dafoe explores ways in which research into AI governance could effect change.
• A bird’s-eye view of the AI alignment landscape by Neel Nanda summarises the different ways in which technical alignment research could reduce the risk from AI.
• An overview of 11 proposals for building safe advanced AI by Evan Hubinger discusses and evaluates plausible techniques for AI alignment.
• Podcasts: The AI X-risk Research Podcast, particularly episode 12 with Paul Christiano and episode 13 with Richard Ngo — both of which serve as excellent introductions to AI risk.

On The 80,000 Hours Podcast, we have a number of in-depth interviews with people actively working to positively shape the development of artificial intelligence:

• Paul Christiano on his vision of how humanity might progressively hand over decision-making to AI systems.
• Allan Dafoe on trying to prepare the world for the possibility that AI will destabilise global politics.
• Dario Amodei of Anthropic explains how to become an AI researcher.
• Miles Brundage of OpenAI explains how to become an AI strategist.
• Holden Karnofsky, cofounder of GiveWell and Open Philanthropy has been on two of our podcasts: he explains how philanthropy can have maximum impact by taking big risks (including a discussion of his work in positively shaping the development of AI), and he discusses why this might be the most important century.
• PhD or programming? Fast paths into aligning AI as a machine learning engineer, according to ML engineers Catherine Olsson and Daniel Ziegler.
• Jan Leike (now head of the Alignment team at OpenAI) explains how to become a machine learning alignment researcher.

If you want to go into much more depth, the AGI safety fundamentals course is a good starting point. There are two tracks to choose from: technical alignment or AI governance.

And finally, here are a few general sources (rather than specific articles) that you might want to explore:

• The AI Alignment Forum, which is aimed at researchers working in technical AI safety.
• AI Impacts, a project that aims to improve society’s understanding of the likely impacts of human-level artificial intelligence.
• The Alignment Newsletter, a weekly publication with recent content relevant to AI alignment with thousands of subscribers.
• Import AI, a weekly newsletter about artificial intelligence by Jack Clark (cofounder of Anthropic), read by more than 10,000 experts.
• Jeff Ding’s ChinAI Newsletter, weekly translations of writings from Chinese thinkers on China’s AI landscape.

Acknowledgements

Huge thanks to Joel Becker, Tamay Besiroglu, Jungwon Byun, Joseph Carlsmith, Jesse Clifton, Emery Cooper, Ajeya Cotra, Andrew Critch, Anthony DiGiovanni, Noemi Dreksler, Ben Edelman, Lukas Finnveden, Emily Frizell, Ben Garfinkel, Katja Grace, Lewis Hammond, Jacob Hilton, Samuel Hilton, Michelle Hutchinson, Caroline Jeanmaire, Kuhan Jeyapragasan, Arden Koehler, Daniel Kokotajlo, Victoria Krakovna, Alex Lawsen, Howie Lempel, Eli Lifland, Katy Moore, Luke Muehlhauser, Neel Nanda, Linh Chi Nguyen, Luisa Rodriguez, Caspar Oesterheld, Ethan Perez, Charlie Rogers-Smith, Jack Ryan, Rohin Shah, Buck Shlegeris, Marlene Staib, Andreas Stuhlmüller, Luke Stebbing, Nate Thomas, Benjamin Todd, Stefan Torges, Michael Townsend, Chris van Merwijk, Hjalmar Wijk, and Mark Xu for either reviewing this article or their extremely thoughtful and helpful comments and conversations. (This isn’t to say that they would all agree with everything we’ve said here — in fact, we’ve had many spirited disagreements in the comments on this article!)

Which of the following is not advised as a strategy for helping adults learn quizlet?

When giving a presentation to a large audience you should?

Is a meeting organized with the purpose of sharing important information about an organization with the media?

Which organizational pattern is best used for a speech that contains a series of steps?