https://www.alignmentforum.org/posts/KHyBocZncAmtu4Jbc/rl-and-search-is-a-terrifying-way-to-build-agi-an-faq

And here’s the start:

Q1: What are you saying?

A: My claim here is that if you build artificial general intelligence (AGI) via any algorithm that’s choosing actions via reinforcement learning (RL) and/or model-based search and planning—a giant chunk of your AI textbook—then that’s just an utterly terrifying thing that you’re doing. You’re playing around with algorithms that, if they work at all, would tend to create ruthless, callous AGIs, AGIs which would happily exterminate humanity and run the world by themselves, given an opportunity.

Mercifully, large language models (LLMs) today are not in the category of “algorithms that choose actions via RL & search”. At least, not primarily—see LLMs are (still) mostly powered by imitative learning, not RL. So LLMs are outside the scope of this post. However, lots of other researchers and companies around the world are enthusiastically trying to build AGI in the maximally terrifying way, as we speak.

Q2: So you’re saying, don’t build AGI based on RL and/or search & planning?

A: In principle, it’s entirely possible that something is terrifying, but we should do it anyway.

…Like space travel! Space travel is: “Let’s fill a tank with 1000 tons of the most flammable substance imaginable, and then light it on fire, and strap people to the front, to accelerate them until they’re traveling at insane speeds through the extremely lethal vacuum of space.” That’s terrifying! But we do it anyway.

And I’m not being anti-space-travel when I point out how terrifying it is. The space travel enthusiasts and the space travel skeptics can happily work together to spread deep understanding of all the ways that space travel can go lethally wrong. Because you can’t overcome a challenge without understanding it.

…Having said all that, I strongly endorse “Don’t build AGI based on RL & search until and unless we find a much better plan for making it friendly”. So if someone is working on making RL & search systems more powerful right now, then I think that’s bad, and that they should instead be spending their time and effort figuring out how (and indeed whether) such systems may be used safely if they become more powerful in the future. That’s an open problem, and we can work on it today.

Q3: Why do you think it’s terrifying?

A: A big part of the problem is that the reward function (or cost function, or objective function, or whatever we want to call it) will generally be written in Python, not in natural language.1 And RL & search agents will tend to ruthlessly maximize it, including in ways the programmer obviously didn’t intend. That’s the whole point of these algorithms: they ruthlessly maximize a thing. That’s what they were designed to do, and that’s what they actually do, when they work well enough to do anything impressive [nuances in Q3b & Q11 below].

And nobody knows how to write down a Python function relating to the real world, such that ruthlessly maximizing that function would lead to good outcomes.

To illustrate the problem, consider “specification gaming” […]

It’s an FAQ, so I’ll just list the rest of the questions I address in the post:

Q3b: So your concern is the “literal genie” / “monkey’s paw” thing?

[…]

Q4: Won’t this problem go away when the AI is smart enough to understand what we intended when we wrote the reward function code?

[…]

Q5: Can’t we just fix bad behavior when we see it?

[…]

Q5b: Follow-up: I don’t buy that, because even if superintelligent AIs could deceptively hide their bad behavior, won’t earlier AIs be sufficiently incompetent that we’ll see their bad behavior? And if so, again, can’t we just fix the bad behavior when we see it? We do know how to fix bad behavior we see it: the RL & search literature is full of examples where algorithms did useful things as intended.

[…]

Q6: Why don’t we just solve the problem by using an obvious, common-sense reward / cost / objective function, like [FILL IN THE BLANK]?

[…]

Q7: Isn’t this whole thing kinda crazy? After all, LLMs are not ruthless sociopaths all the time, and humans are also not ruthless sociopaths all the time. So where is this idea even coming from? Are you sure you’re not just watching too much sci-fi?

[…]

Q8: Isn’t this problem solved by laws and markets? I.e., if an AGI has sociopathic desires and callous indifference to human welfare, that’s fine! It will still act nice and cooperative and rule-following, because acting nice and cooperative and rule-following is the best way to accomplish goals, in our complex interconnected interdependent world. Right?

[…]

Q8b: Following up on that: Even if you’re right that there’s a local incentive for being open to stabbing your allies in the back, isn’t there a higher-level, group-selection-style, incentive to be genuinely deeply nice? Specifically, won’t the groups of nice cooperative AGIs outcompete the groups of callous transactional AGIs who all keep stabbing each other in the back? And isn’t that related to how humans evolved to be nice?

[…]

Q9: Why would we want to infringe on the AGI’s autonomy by choosing its reward function?

[…]

Q10: Why not just be nice to the AGIs, and then they’ll be nice to us in turn?

[…]

Q11: RL & search algorithms don’t literally optimize the reward / cost / objective function. Doesn’t that invalidate your argument?

[…]

Click the link to read the whole thing!

https://www.lesswrong.com/posts/wYpjXRLqbLbnmjbJP/llms-are-still-mostly-powered-by-imitative-learning-not-rl

And here’s the start:

Reinforcement learning from verifiable rewards (RLVR) is the hot new thing in LLM training. It’s so hot, and people spend so much time talking about it, that they sometimes lose sight of the big picture.

Stepping back, LLMs can do lots of very impressive things. How? Where did those capabilities come from? Fundamentally, they come from a combination of:

• (1) Imitative learning, including pretraining and supervised fine-tuning (SFT)

  • See my earlier discussion: “LLM pretraining magically transmutes observations into behavior, in a way that is profoundly disanalogous to how brains work”.

• (2) Reinforcement learning, including RL from human feedback [RLHF], RL from AI feedback [RLAIF], and especially RLVR.^[1]

If we look at the final trained LLM, we can ask how important each of those two pieces was, in explaining the LLM’s capabilities. And my claim is that it’s way more (1) than (2).

I’ll start in §1 with some relevant evidence, and then in §2 I’ll circle back to operationalizing exactly what I’m claiming, and finally in §3, three reasons why we should care—namely, it affects how we should think about chain-of-thought legibility, about LLM capabilities, and about LLM alignment.

Note that I am not arguing that RLVR does not importantly contribute to LLM capabilities. That would be absurd! Of course it does! Companies use RLVR because it works, and I expect them to continue doing so more and more. Again, the things I’m actually claiming are in §2–§3.

Click the link to read the rest!

https://www.lesswrong.com/posts/BHEcssYwYrzTBf9Qm/will-almost-all-future-companies-eventually-be-founded-and

And here’s an excerpt

My belief is that keeping AI under human control would be an unprecedented global challenge, if it’s even possible at all.

Many other people’s believe that AIs will naturally remain as tools under human control. Either they see this as inevitable—“how could it be otherwise?”—or they see it as not quite inevitable, but still an outcome that can easily happen via ordinary government regulations, civil society, and so on.

Who’s right? What’s the right way to think about this?

This is an issue where people really don’t see eye-to-eye, and tend to get stuck endlessly talking past each other. To pin down the disagreement, I think that the following is a good conversation-starter:

Conversation-starting question: Do you expect almost all companies to eventually be founded and run by AIs rather than humans?

I find that people give a number of different answers:

Possible Answer 1: “No, because the best humans will always be better than large language models (LLMs) at founding and running companies.”

Huh? Whoever said anything about LLMs? I wrote “AIs”!

[…]

Possible Answer 2: “No, because the best humans will always be better than AIs at founding and running companies.”

I think that most people with this view are generally rejecting the idea that actually powerful AI (as in the right side of the table above) is possible at all, or are failing to think about its implications. Again, humans are an existence proof for what is physically possible for AI. Thus, for example:

Humans can acquire real-world first-person experience? Well, an AI could acquire a thousand lifetimes of real-world first-person experience.

[…]

Possible Answer 3: “No, because humans will always be equally good as AIs at founding and running companies.”

I think that most people with this view don’t really believe, in their guts, that skill and competence are a thing. Or at least, they don’t believe that skill and competence are relevant to founding and running successful companies.

When these people imagine what Jeff Bezos was doing day-to-day to build Amazon from scratch, I’m not quite sure what’s in their head. Befriending people? Scaring people? Puppies can also befriend and scare people, but no puppy has ever become a hundred-billionaire by founding and running a successful company.

[…]

Possible Answer 4: “No, because we will pass laws preventing AIs from founding and running companies.”

Even if such laws existed in every country on Earth, and the letter of such laws was enforceable, the spirit wouldn’t be. Rather, the laws would be trivial to work around. For example, you could wind up with companies where AIs are making all the decisions, but there’s a human frontman signing the paperwork.

[…]

Possible Answer 5: “No, because if someone wants to start a business, they would prefer to remain in charge themselves, and ask an AI for advice when needed, rather than ‘pressing go’ on an autonomous entrepreneurial AI.”

This is a really nice vision, and I wish I could believe it. But even if lots of people do in fact take this approach, and they create lots of great businesses, it just takes one person to say “Hmm, why should I create one great business, when I can instead create 100,000 great businesses simultaneously?”

[…]

Click the link to read the rest!

• I mentioned a month ago that I had made some edits to the blog version of Intro to Brain-Like-AGI Safety, but had not yet updated the archival PDF version. Well, now I have! Say hello to PDF version 4. For highlights from the changelog, see the update from a month ago.

• I added yet even more caveats to my old Valence series post 4 (2024), including a note at the top that I’ve disavowed so much of the post by now that it’s hardly worth reading.

New blog post: “What do I mean by ‘Artificial General Intelligence’?”

Here’s the link:

https://www.lesswrong.com/posts/nQH2GhkmSmHoCAN8R/what-do-i-mean-by-artificial-general-intelligence

And here’s the first ~half of it:

In this post, intended for a broad audience, I will paint a brief picture of what I’m talking about when I talk about “AGI”. It will seem obvious to many people, and obviously wrong to many others! So let’s jump in:

---

For example, today, if you want an AI to drive a car, or to control a computer using a mouse, it’s a huge project involving dozens of experts working for years to make a new AI. Whereas if you want a human to do the same, you don’t need to do R&D to breed a new subspecies of human! Instead, you just take an ordinary human—basically the same design from 100,000 years ago—and give them a few hours of practice, and you’re done. Someday we’ll have an AGI design which can do things like that. …

---

In this case, many people’s mental image of AI is already transitioning from “tool” towards “agent”, especially in the past year or two, after they’ve watched LLM agents execute on projects. But even those people are usually not going far enough for what I have in mind. Think of things that took a whole society of humans to do over an extended period of time—like inventing language and science from scratch, and developing them all the way into space travel and microchips and skyscrapers. These AGIs will be able to do those kinds of things too, fully autonomously.

---

Now, don’t be put off by “crazy sci-fi stuff”—indeed, every technology that exists today was “crazy sci-fi stuff” before it was invented!

So the wrong question is: “Is it sci-fi?”. The right question is: “Is it possible?”

And the answer to that question is “Yes”! And we know this because we have an existence proof. Human brains and bodies can do all these things, and they don’t work by magic, but rather follow the principles of physics, math, and engineering, like everything else.

And whatever engineering principles allow humans to do all those right-column things, we should expect future scientists to sooner or later figure out how to exploit those same principles, in order to accomplish the same things. Even if that might seem impossible today! After all, think of how impossible vision must have seemed 1000 years ago: your eyes provide a magical window through which knowledge of your surroundings enters your soul. But now we understand the big-picture principles that explain how eyes work, and we have our own technology (cameras) based on similar principles. Ditto with hearing (microphones), moving (actuators), digestion (industrial catalysts), and so on. …

Click the link to read the rest!

https://www.alignmentforum.org/posts/rKdS7i4StaMmFzYRo/notes-on-technical-alignment-via-human-like-social-drives

And here’s how it begins:

1. Frontmatter

1.1 Backstory for this post

As discussed in Intro to Brain-Like-AGI Safety, I’m working on the technical alignment problem for a hypothetical future “brain-like AGI”, with a particular focus on treating human innate social and moral drives as a possible jumping-off point for our technical alignment approach.

After all, if it’s possible for humans to do stuff that ultimately leads to a good future, then it’s probably also possible for sufficiently human-like AGIs to do stuff that ultimately leads to a good future. Or if it’s not possible for humans to do stuff that ultimately leads to a good future, then we’re screwed no matter what. But assuming it’s possible, the “sufficiently human-like AGIs” would certainly need to have good prosocial motivations. What code do we write that would lead to good prosocial motivations? It’s an unsolved problem (see “We need a field of Reward Function Design” (2025)), but as we search for a solution, we might look for inspiration at how humans (sometimes) wind up with good prosocial motivations.

I’ve been working on this problem for years, but most of that work has involved laying foundations (e.g. trying to understand how human social drives work). Whereas in the past four months, I’ve been thinking very directly about how to apply those ideas to AGI.

I’ve published two little things from this four-month effort—“Act-based approval-directed agents”, for IDA skeptics, and Empowerment, corrigibility, etc. are simple abstractions (of a messed-up ontology)—but most of what I (think I) figured out is not self-contained, but rather part of a big interconnected mess of thoughts and ideas. So for now I’m just dumping that all into this one excessively-long post. Sorry.

This post has lots of new-to-me ideas and opinions, and they are in great need of more scrutiny and especially deconfusion. I’m happy for any feedback, pushback, and riffs on anything herein.

1.2 Table of contents / tl;dr

Sections 2–5 start from human social instincts, and ponder how to usefully employ something-like-that in an AGI. Specifically:

• Section 2 goes over the high-level approach of using human social instincts as a starting point / inspiration for an AGI motivation system. What are the specific instincts in question, how do they work in humans, what roles if any should they be playing in AGI, and what should we be keeping in versus leaving out?

• Section 3 discusses three potential failure modes that I’ve spent a long time thinking about: the possibility that the AGI will wind up with the wrong “moral circle”; the possibility that virtue-ethics-y human motivations (honesty, helpfulness, etc.) rely on a balance-of-power dynamic which wouldn’t apply to the ASI-human relationship; and the possibility that consequentialist desires will squash virtue-ethics-y desires in the long run, when both are present (as I claim they need to be).

• Section 4 is “What controls the set of virtues that the AGI takes pride in?” I discuss two pathways: “person-first” and “desire-first”. For example, someone could wind up taking pride in their encyclopedic knowledge of Disney princesses because that’s what the cool older kids in school are into (“person-first”); or because they really really like Disney princess movies, and that love has gradually wormed its way into their self-image (“desire-first”). I suggest that the “desire-first” pathway is an important way that nerds like me wind up motivated to figure out the truth and share it with others—a key trait that we may want in brain-like AGI. (Hold that thought!)

• Section 5 goes over some implementation details related to transplanting human social drives into the foreign soil of AGI source code.

Then in Section 6 I switch from forward-reasoning (starting with human social instincts) to backwards-reasoning (starting with desiderata), by asking: What AGI motivations do we want anyway?

• Section 6.1 goes over three sets of constraints that I’m trying to satisfy: technical alignment constraints (we need to be able to write the code), strategic constraints (we need to make the world resilient to misaligned ASI), and ethical / societal / buy-in constraints (the plan needs to sound reasonable, such that people will actually follow it).

• Section 6.2 goes over a bunch of possible AGI motivations, and how they seem to stack up against those three sets of constraints. I wind up tentatively advocating for a high-level approach that I’ve been calling “truth-seeking disagreeable nerd AGI”, using the technical alignment idea mentioned in §4 above, and then have that AGI figure out what to do next.

Section 7 has a couple more random things from my notes:

• Section 7.1 discusses two (related) dilemmas that I’ve been struggling with. I call them “the visceral reaction updating dilemma” and “the value drift dilemma”.

• Section 7.2 describes a subtle mistake in how I was thinking about “ruthlessness” until recently.

I close in Section 8 with what I plan to work on next and why.

Click the link to read the rest!

1.1 New version of Neuroscience of human social instincts: a sketch

Neuroscience of human social instincts: a sketch (2024) got another batch of changes (following a different set of important changes just last month), cleaning and streamlining a kinda muddled and partly-incorrect discussion of learning rate modulation, including by introducing new figures and new terminology. Here’s the changelog:

2026-05-19: I rewrote §3–§5.1 to remove unnecessary complication, and clean up some errors and muddled thinking. More details:

In §3.2, I previously had a toy example of learning rate modulation in the thought assessors, where I was daydreaming about Taylor Swift, and then I suddenly orient to a spider jumping at me, and the learning rate modulation (I argued) was necessary to prevent learning that Taylor Swift is a risk factor for spiders jumping out at me. I do think that’s an actual solution to an actual problem … [but] I described this example poorly (and somewhat incorrectly), and more importantly it’s an example that’s not directly related to this post, and I think it was just causing unnecessary confusion (even I was confused when I re-read it). So I switched to a new example that overlaps much more with §4. I also deleted the discussion of learning rate modulation in the Thought Generator, which I decided was somewhat misleading and confusing as written, and off-topic anyway.

That change to §3, in turn, allowed me to shorten and streamline §4, including in ways that hopefully made §5.1 a bit clearer in turn.

The new version introduces and uses a new term I just made up, “interoceptive concept finder”, for a particular type of short-term predictor.

(The archival PDF is now up to version 3.)

1.2 Revisiting an old take on LLMs

I’ll probably annoy all sides with this not-really-an-apology that I just tacked onto the beginning of my LLM-skepticism-related post from Apr 2023:

Update June 2026: I basically stand by this post, except that I regret using the word “plateau”. By analogy, chess engines show no signs of “plateau-ing”: Stockfish 18 (2026) crushes Stockfish 17 (2024), which in turn crushes Stockfish 16 (2023), etc.^[1] But chess engines ain’t gonna take over the world. So anyway, what I should have said was: There’s a school of thought, in which LLMs might (or might not) continue to get ever better at some things, but they will definitely always be bad at other things, and those ‘other things’ are very important, such that their continued absence will prevent LLMs from ever becoming “transformative AI” as discussed below. This post is about how this school of thought relates to AI doom. And separately, if that school of thought is right, should we describe this situation using the word “plateau”? Arguably we could, in the same sense that chess engines would hit a “plateau” of 50% on a test with half chess puzzles and half sports trivia. But all things considered, I think the word “plateau” was a poor choice born of sloppy thinking. Sorry.

1.3 More clarity on what “under-sculpting” means

I added a few sentences to Perils of under- vs over-sculpting AGI desires (2025) to hopefully make it clearer what “under-sculpting” means, especially this part:

Think of it like: the AGI’s desires are a boat, and there’s a radio beacon (the reward function) that we know how to steer towards. If we keep steering towards the beacon (over-sculpting), then we can get very close to the beacon, but alas, the beacon is built on jagged rocks that will kill us (specification gaming, §2). On the other hand, if we stop navigating towards the beacon at some point (under-sculpting), then we’re just adrift on the sea, and our two problems are: (1) we don’t know where we are right now (path dependence, §8.1), and (2) we don’t know where the unpredictable currents will take us in the future (concept extrapolation upon distribution shifts, §8.2).

1.4 A bunch of edits to Intro to Brain-Like-AGI Safety

See changelogs at the bottom of Posts 1, 5, 6, and 15 for details. The biggest changes were thorough rewrites of §5.3 on valence and reinforcement learning, §6.4 on what I mean by “model-based RL”, and §6.6.1 on ego-syntonic desires. (Those changes are not yet in the archival PDF version, sorry, I’ll upload a new PDF when I get a chance.)

The new version of §6.6.1 is fun; here’s most of it:

6.6.1 The distinction between internalized ego-syntonic desires and externalized ego-dystonic urges is unrelated to Learning Subsystem vs. Steering Subsystem

Many people (including me) have a strong intuitive distinction between ego-syntonic drives that are “part of us” or “what we want”, versus ego-dystonic drives that feel like urges which intrude upon us from the outside.

For example, if someone is on a hunger strike for freedom, they might say that their desire to eat comes from their innate drives, whereas their desire to fight for freedom comes from “reason”, or “their best self”, or whatever. What does that mean? What’s going on?

…

I propose that we should not take this intuition at face value. In reality, the hunger-striker’s desire to eat comes ultimately from their innate drives, and their desire to fight for freedom also comes ultimately from their innate drives! These are just two innate drives that are pointing in different directions, and thus they duke it out. One side will win, and then the person will either keep their hunger strike, or break down and eat.

It’s not so different from if you feel simultaneously very sleepy and very hungry; you can’t satisfy both drives, so the drives will duke it out, and one of them will win, and either you’ll nap despite your hunger, or you’ll eat despite your sleepiness.

That said, there do seem to be very important differences between the mundane hunger-vs-sleepiness battle (urge vs urge) and the hunger-vs-fight-for-freedom battle (urge vs ego-syntonic desire). In particular, here are three obvious questions that I need to address:

First, hunger and sleep drives are easy to understand. But it’s much less obvious how innate drives could lead a person to care so much about freedom that they’ll go on a hunger strike. What exactly is the innate drive in question?

Second, if I’m both hungry and sleepy, it’s sorta “a fair fight”. Probably whichever feeling is more immediately powerful will win. By contrast, the internal battle between the hunger-striker’s desire for freedom and their desire for food is not a fair fight. The former desire will punch above its weight by bringing far more intelligence and foresight to bear towards its objective. Thus, if the person is setting up a commitment mechanism, or tying their own hands, or attempting to control themselves and their mood, those actions will almost definitely be in pursuit of freedom, not in pursuit of hunger-satisfaction. Why the asymmetry? If both drives are in the Steering Subsystem, shouldn’t they be equally stupid and myopic? Relatedly, if the person is “applying willpower”, why is it in support of freedom rather than hunger-satisfaction? And by the way, what the heck does “applying willpower” even mean at a nuts-and-bolts level??

Third, if the person’s desire for freedom is not really more a “part of them” than their desire to eat when hungry, then … why does it feel that way to them? In other words, this is an honest introspective report. I’m allowed to claim that the report should be interpreted as a perceptual illusion rather than taken at face value, but you have no reason to believe me unless I can also explain what exactly they were introspecting upon, and what they saw when they did so, and why it left them with the impression that it did.

These are all great questions! And I have answers to all of them! But they’re rather involved.

For the first question, I claim that hunger-striking for freedom is driven by social instincts. You can learn about those at a high level in Post #13 of this series, and then proceed to my follow-up work Neuroscience of human social instincts: a sketch (2024), and Social drives 1: “Sympathy Reward”, from compassion to dehumanization (2025), and Social drives 2: “Approval Reward”, from norm-enforcement to status-seeking (2025), especially §3 of that last post on how Approval Reward leads to pride in one’s self-image, and staying true to your principles even when nobody is watching. (That said, if people are watching, and you thus win the approval of people you admire, so much the better!)

For the second question, my short answer is that people have a strong, salient association between thoughts of themselves, and thoughts of how they would look in someone else’s eyes. The former thoughts are required for strategically tying one’s hands, making precommitments, etc. And the latter thoughts summon social instincts like pride. This is the source of the asymmetry where social instincts “punch above their weight” in the context of intelligent self-reflective plans, such as precommitments, describing one’s life aspirations, and so on. For more detail (along with how “willpower” fits in), see §8.5.5–§8.5.6 of my “Intuitive Self-Models” series (2024), but be warned that it might not make sense without reading the earlier posts in that Intuitive Self-Models series, in which I try to unravel a bunch of misleading intuitions in how we think about our own minds.

The third question (on why and how ego-syntonic desires are “internalized”) is also addressed in that same series, see “Intuitive Self Models” §3.5.4.

…

One way to put it is: why does the hunger-striker care about freedom, and not about, I dunno, ironing the wrinkles out of dollar bills? There has to be some explanation, right? And if you reply “it’s because freedom leads to (blah)”, then I’ll just reply, “OK, and why do they care about (blah), rather than, I dunno, measuring the distance between pebbles on the sidewalk?” We can go back and forth forever. The answer is: at the end of the day, something has to just plain feel intuitively good or bad. And that feeling has to come from an innate drive, one way or another.

Here are three more views on why we should believe that the Steering Subsystem is the ultimate source of not only ego-dystonic urges like hunger, but also ego-syntonic desires like friendship and justice.

• AI perspective: We don’t yet know in full detail how model-based RL and model-based planning works in the human brain—we don’t have brain-like AGI yet. But we do at least vaguely know how these kinds of algorithms work. And we know enough to say for sure that these algorithms don’t develop prosocial motivations out of nowhere. For example, if you set the reward function of MuZero to always return 0, then the algorithm will emit random outputs forever—it won’t start fighting for justice.

• Rodent model perspective: For what it’s worth, researchers have been equally successful in finding little cell groups in the rodent hypothalamus that orchestrate “antisocial” behaviors like aggression, and that orchestrate “prosocial” behaviors like parenting and sociality. I fully expect that the same holds for humans.

• Philosophy perspective: Without the Steering Subsystem, the only thing the cortex can do is build a world-model from predictive learning of sensory inputs (§4.7). That’s “is”, not “ought”. And “Hume’s law” says that you can’t get “ought”-statements from exclusively “is”-statements. Granted, not everyone believes in Hume’s law. But I do—see an elegant and concise argument for it here.

This new section I added to Post 1 is also worth copying here, you wouldn’t believe how often people are confused by this:

1.3.4 So is “Brain-like AGI” a good plan? Or is it a threat model?

Lots of people assume that, if I’m devoting my career to brain-like-AGI safety, I must be very enthusiastic about brain-like AGI.

By analogy, if someone is devoting their career to rocket engine safety, odds are high that they’re a space nerd who thinks that rocket engines are really cool and great.

…But on the other hand, people devote their careers to earthquake safety too! Do those people think earthquakes are really cool and great? Of course not! But they recognize that earthquakes will come, whether we want them or not, so we’d better prepare.

Now as it turns out, I’m much more like the earthquake safety person than the rocket safety person: I think of brain-like AGI as a threat model. Honestly, I expect that brain-like AGI will probably kill us all, in a manner that makes Skynet (from the Terminator movies) look primitive and sentimental. You don’t have to agree! Indeed, both enthusiasts and naysayers have a strong shared interest in understanding potential safety problems and designing mitigations, in a constructive, pedagogical, technical, and detail-oriented way. That’s my aim in this series. But I did want to lay my cards on the table.

1.5 Is valence a linear function on “thoughts”?

In [Valence series] 2. Valence & Normativity (2023), I made a claim in §2.4.1 that “valence is a (roughly) linear function over compositional thought-pieces”, and then immediately in §2.4.1.1 listed a bunch of examples where that claim seems totally wrong—for example, “someone I hate is suffering” or “I’m gonna avoid traffic” (both “someone I hate” and “traffic” are bad, but those thoughts seem overall good). I had (and still have) sound neuroanatomical and algorithmic reasons to posit linearity, but in 2023 I had a pretty hazy understanding of why this hypothesis was not immediately ruled out by those examples. Anyway, I can explain this substantially better now (albeit still a bit hazy), and I rewrote §2.4.1.1 accordingly.

(Almost all of those edits were in response to criticisms from Rif A. Saurous, many thanks to him.)

2. New blog post: “Sympathy for both sides of the egregious misalignment debate”

Here’s the link:

https://www.alignmentforum.org/posts/DZaZ3fqHnvfLCftPu/sympathy-for-both-sides-of-the-egregious-misalignment-debate

And here’s the first half of it:

On one side of this debate is Yudkowsky & Soares, who think that (if AI progress continues) we’re on a direct path to egregiously-misaligned, scheming, out-of-control, rogue superintelligence (ASI), not even slightly nice, in the absence of yet-to-be-invented breakthrough technical alignment ideas.

On the other side of this debate is almost everyone who works on or studies LLMs. Some of them are very concerned about egregious scheming, others much less so, and as a group they’re equally or more concerned about lots of other potential AI problems—AI-assisted bioterrorism, AI-assisted dictatorships, etc. And if they’re concerned about egregious misalignment and scheming, they’ll often say that it would come about through being in too much of a rush, or careless programmers, or bad actors, etc., as opposed to the simpler Yudkowsky & Soares story of “we get egregious misalignment and scheming because nobody has the foggiest idea how to avoid that”.

Here’s my brief idiosyncratic take on this debate. I think BOTH of the following are true:

• (1) If you really think carefully about the properties of ASI, you really do find good reasons to strongly expect it to be egregiously misaligned, scheming, and ruthless, in the absence of yet-to-be-invented breakthrough technical alignment ideas.

• (2) If you really think carefully about the properties of current LLMs, you really do find good reasons to think that existing technical alignment techniques are adequate now, and may well continue to be adequate in the future.

So then here are three (caricatured) positions:

My position:

(1) and (2) are both totally true. And we can reconcile them by saying that LLMs won’t scale to ASI.

Yudkowsky & Soares’s position [caricatured]:

(1) is totally true. We know this with great confidence, having spent decades thinking about it.

So it follows that (2) must be wrong or irrelevant.

Why is (2) wrong or irrelevant? Hard to say! There’s no ASI yet, and nobody knows in detail how it will appear. Sometimes it’s easier to predict what happens eventually than the detailed path. An ice cube in warm water will melt eventually, but don’t ask me to predict how many seconds it will take to melt, etc.

So anyway, one possibility is that (2) is wrong because LLMs will kinda ‘wake up’, or something, when the core pieces of true intelligence finally come together. And then their behavior would change drastically for the worse. And maybe we’re already starting to see glimmers of that in existing LLMs?

Or another possibility [cf. Eliezer tweet] is that LLMs will invent non-LLM ASI. And then (2) will be simply irrelevant!

…Or something else! Again, we don’t know! But we do know that (1) is definitely right.

LLM people’s position [caricatured]:

(2) is totally true. We know this with great confidence, because we are LLM experts and we have thought about these alignment plans in great detail, including matching our theories against real-world data.

So it follows that (1) must be incorrect.

Why is (1) incorrect? I don’t really know! Man, I read Yudkowsky and Soares, and it’s all these words, words, words, and I’m reading along and trying to match those words to my knowledge of LLMs and it just doesn’t make any damn sense. I can and will try to respond to their points in detail, but honestly the core issue is that they’re guilty of head-in-the-clouds armchair theorizing gone off the rails.

Conclusion

…So I think that both sides of the debate are basically coming from a reasonable and sympathetic place, with a big kernel of truth.

…

Click the link to read the rest!

The orienting reflex as exemplifying a broader neuroscience motif

I revised 2 old posts based on deeper appreciation for the orienting reflex as exemplifying a broader neuroscience motif:

(1) In Neuroscience of human social instincts: a sketch (2024):

I changed terminology from “the ‘thinking of a conspecific’ flag” to “the social attention reflex”. I think the new term has better connotations, especially the way it invokes a parallel to “orienting reflex” and “startle reflex”, which likewise are associated with fast, transient, and involuntary changes in both attention and other innate signals like pleasure and arousal.

(I made some other changes to this one too—see changelog—and then also posted an updated version 2 of the archival PDF: https://doi.org/10.5281/zenodo.17953592.)

(2) In Valence series §3.3.5 (2023):

(Update April 2026: I’ve refined my model a bit: I now think of anxiety-related involuntary attention as being more closely analogous to the famous orienting reflex wherein people turn to look at an unexpected loud sound or motion. Traditional orienting reflexes involve involuntary attention towards exteroceptive inputs, coupled with innate motor commands, physiological arousal, etc. By analogy, if you’re anxious, then (I claim) you’ll likewise experience sporadic interoceptive “orienting reflexes” that involve involuntary attention towards the the feeling of anxiety, coupled with a synchronized squirt of negative valence and displeasure (see Appendix A), plus physiological arousal etc. These interoceptive “orienting reflexes” might occur multiple times per second for intense anxiety, or less often for milder anxiety. I’m using anxiety as an example, but the same idea obviously applies as well to fear, hunger, itches, etc.)

Renouncing an old discussion of narcissism

One section of my 2023 post [Valence series] 5. “Valence Disorders” in Mental Health & Personality was a hypothesis about narcissistic personality disorder. Now it’s 2½ years later, and I’ve figured out much more about human social instincts, and when I revisited what I wrote in 2023, I decided it was all wrong, and struck it out. I will hopefully write up “My Model of NPD, Take 2” when I get a chance.

Nuances on the idea of updating-to-convergence

I added a section to Against empathy-by-default (2024) reflecting some things that I’ve learned since then:

3.3 (Added April 2026) Things vaguely analogous to the empathy-by-default argument can happen for certain visceral reactions—just not for the core motivation / reward / RL system

In the above (especially §3.1), I think I conveyed a general vibe that within-lifetime learning will eventually, inexorably, correct all “errors” in learned brain models. But a while after writing this post, I came to better appreciate how certain visceral reactions in the brain can be set up so as to sometimes prevent updates (“corrections”). This is how people can wind up with stable phobias, and stable food-aversions, and stable traumas, and stable autistic “special interests”, and so on, even when there’s no particular innate “ground truth” underlying them. These stable “errors” are the exception not the rule, but it’s interesting that they exist at all, and that they can in some cases last a lifetime.

I discuss the algorithmic trick behind these in a later post: “Perils of under- vs over-sculpting AGI desires” (2025), specifically §6.2: “‘Defer-to-predictor mode’ in visceral reactions, and ‘trapped priors’. In brief, the trick centers around what I call “defer-to-predictor mode”, where e.g. a visceral expectation of imminent disgust can cause an actual disgust reaction. But there’s a loop-y thing, wherein the actual disgust is in turn the ground truth for how we learn that a visceral expectation of disgust is warranted. Thanks to this loop-y thing, we can wind up without any error signal telling our brains that the disgust was never warranted in the first place.…

New blog post: “Empowerment, corrigibility, etc. are simple abstractions (of a messed-up ontology)”

Here’s the link:

https://www.alignmentforum.org/posts/vzHtHHBJoKATi5SeK/empowerment-corrigibility-etc-are-simple-abstractions-of-a

And here’s the start:

1.1 Tl;dr

Alignment is often conceptualized as AIs helping humans achieve their goals: AIs that increase people’s agency and empowerment; AIs that are helpful, corrigible, and/or obedient; AIs that avoid manipulating people. But that last one—manipulation—points to a challenge for all these desiderata: a human’s goals are themselves under-determined and manipulable, and it’s awfully hard to pin down a principled distinction between changing people’s goals in a good way (“providing counsel”, “providing information”, “sharing ideas”) versus a bad way (“manipulating”, “brainwashing”).

The manipulability of human desires is hardly a new observation in the alignment literature, but it remains unsolved (see lit review in §3 below).

In this post I will propose an explanation of how we humans intuitively conceptualize the distinction between guidance (good) vs manipulation (bad), in case it helps us brainstorm how we might put that distinction into AI.

…But (spoiler alert) it turns out not to really help, because I’ll argue that we humans think about it in a deeply incoherent way, intimately tied to our scientifically-inaccurate intuitions around free will.

I jump from there into a broader review of every approach that I can think of for writing a “True Name” for manipulation or things related to it (empowerment, agency, corrigibility, culpability, etc.), or indeed for any other method of robustly getting future AGIs to be able to talk to people without trying to manipulate those people’s desires. I argue that none of them provides much of a path forward on the particular technical alignment problem I’m working on. Indeed, my current guess is that none of these things have a “True Name” at all, or at least not one that’s useful for the technical alignment problem.

Click the link to read the rest!

• I shared a link-post to a podcast, along with excerpts of the part I found most interesting. Title: “Jeremy Howard is bearish on LLMs”.

  • To be clear, he co-invented LLMs and uses them enthusiastically. He is only “bearish” by the standards of people in my AI Alignment professional circles, many of whom have gotten pretty frantic in the past couple months about LLMs imminently rocketing to superintelligence.

• I was invited back for second appearance on the “Doom Debates Podcast”. Our conversation centered around my post Why we should expect ruthless sociopath ASI:

Doom Debates

How Friendly AI Will Become Deadly — Dr. Steven Byrnes (AGI Safety Researcher, Harvard Physics Postdoc) Returns!

Fan favorite Dr. Steven Byrnes returns to discuss recent AI progress and the concerning paradigm shift to "ruthless sociopath AI" he sees on the horizon…

Listen now

5 months ago · 6 likes · Liron Shapira

• Patrick Mineault published a nice two-part discussion of how cell type diversity is connected to the big picture of brain algorithms:

The NeuroAI archive

Cell types: encoding the brain's BIOS

In an interview late last year on the Dwarkesh Podcast, Adam Marblestone—CEO of Convergent Research, former research scientist at Deepmind, and a true polymath in connecting neuroscience and AI—discussed what insights we might extract from complete brain wiring diagrams. His central claim is that the most valuable information in a connectome concerns i…

Read more

5 months ago · 34 likes · 5 comments · Patrick Mineault

The NeuroAI archive

Cell types: from genes to circuits

In the last post, I presented a counterintuitive idea: areas of the brain that contain many cell types encode innate behaviors and rewards. Myriad neurons with strange morphology that are connected idiosyncratically are an indicator of evolutionary pressure: behaviors and rewards that have survival value become genetically hard-coded. What I didn’t get …

Read more

5 months ago · 12 likes · Patrick Mineault

…And now here’s two new actual blog posts since my last newsletter:

Blog post 1/2: “You can’t imitation-learn how to continual-learn”

Here’s the link:

https://www.lesswrong.com/posts/9rCTjbJpZB4KzqhiQ/you-can-t-imitation-learn-how-to-continual-learn

And here’s the start:

In this post, I’m trying to put forward a narrow, pedagogical point, one that comes up mainly when I’m arguing in favor of LLMs having limitations that human learning does not.

See the bottom of the post for a list of subtexts that you should NOT read into this post, including “…therefore LLMs are dumb”, or “…therefore LLMs can’t possibly scale to superintelligence”.

Some intuitions on how to think about “real” continual learning

Consider an algorithm for training a Reinforcement Learning (RL) agent, like the Atari-playing Deep Q network (2013) or AlphaZero (2017), or think of within-lifetime learning in the human brain, which (I claim) is in the general class of “model-based reinforcement learning”, broadly construed. …

When we think of “continual learning”, I suggest that those are good central examples to keep in mind. Here are some aspects to note:

Knowledge vs information: These systems allow for continual acquisition of knowledge, not just information—the “continual learning” can install wholly new ways of conceptualizing and navigating the world, not just keeping track of what’s going on.

Huge capacity for open-ended learning: These examples all have huge capacity for continual learning, indeed enough that they can start from random initialization and “continually learn” all the way to expert-level competence. Likewise, new continual learning can build on previous continual learning, in an ever-growing tower.

Ability to figure things out that aren’t already on display in the environment: For example, an Atari-playing RL agent will get better and better at playing an Atari game, even without having any expert examples to copy. Likewise, billions of humans over thousands of years invented language, math, science, and a whole $100T global economy from scratch, all by ourselves, without angels dropping new training data from the heavens.

Click the link to read the rest!

Blog post 2/2: “‘Act-based approval-directed agents’, for IDA skeptics”

Here’s the link:

https://www.alignmentforum.org/posts/RKtTi82t8X8TQy5FX/act-based-approval-directed-agents-for-ida-skeptics

And here’s the start:

Summary / tl;dr

In the 2010s, Paul Christiano built an extensive body of work on AI alignment—see the “Iterated Amplification” series for a curated overview as of 2018.

One foundation of this program was an intuition that it should be possible to build “act-based approval-directed agents” (“approval-directed agents” for short). These AGIs, for example, would not lie to their human supervisors, because their human supervisors wouldn’t want them to lie, and these AGIs would only do things that their human supervisors would want them to do. (It sounds much simpler than it is!)

Another foundation of this program was a set of algorithmic approaches, Iterated Distillation and Amplification (IDA), that supposedly offers a path to actually building these approval-directed AI agents.

I am (and have always been) a skeptic of IDA: I just don’t think any of those algorithms would work very well.

But I still think there might be something to the “approval-directed agents” intuition. And we should be careful not to throw out the baby with the bathwater.

So my goal in this post is to rescue the “approval-directed agents” idea from its IDA baggage. Here’s the roadmap:

In Section 1, I offer a high-level picture of what we’re hoping to get out of “approval-directed agents”, following a discussion by Abram Demski (2018).

In Section 2, I walk through an example of how this vision can actually manifest in the context of brain-like AGI, a different AI paradigm which (unlike IDA) can definitely scale to superintelligence. I offer an everyday example of having role-models / idols who celebrate honesty, and correspondingly taking pride in one’s self-image as an honest person. In terms of brain algorithms, I relate this phenomenon to (what I call) “Approval Reward”, a hypothesized component of the human brain’s innate reinforcement learning reward function.

Click the link to read the rest!

https://www.alignmentforum.org/posts/ZJZZEuPFKeEdkrRyf/why-we-should-expect-ruthless-sociopath-asi

And here’s how it begins:

The conversation begins

(Fictional) Optimist: So you expect future artificial superintelligence (ASI) “by default”, i.e. in the absence of yet-to-be-invented techniques, to be a ruthless sociopath, happy to lie, cheat, and steal, whenever doing so is selfishly beneficial, and with callous indifference to whether anyone (including its own programmers and users) lives or dies?

Me: Yup! (Alas.)

Optimist: …Despite all the evidence right in front of our eyes from humans and LLMs.

Me: Yup!

Optimist: OK, well, I’m here to tell you: that is a very specific and strange thing to expect, especially in the absence of any concrete evidence whatsoever. There’s no reason to expect it. If you think that ruthless sociopathy is the “true core nature of intelligence” or whatever, then you should really look at yourself in a mirror and ask yourself where your life went horribly wrong.

Me: Hmm, I think the “true core nature of intelligence” is above my pay grade. We should probably just talk about the issue at hand, namely future AI algorithms and their properties.

…But I actually agree with you that ruthless sociopathy is a very specific and strange thing for me to expect.

Optimist: Wait, you—what??

Click the link to read the rest!

• You may remember that two weeks ago, I shared a post “The nature of LLM algorithmic progress”. Well, four days later, I rewrote it as a heavily-revised “version 2”. Thanks everyone for feedback and pushback! If you read “version 1”, consider revisiting it—there’s a changelog at the bottom. Same link.

• There’s a fascinating new preprint, Cellular Scaling Laws in the Mammalian Brain by Fei Chen & Evan Macosko, which dovetails with many of the neuroscience ideas that I’m always talking about, but coming from the angle of transcriptomics and psychology. I’m listed in the acknowledgements—it was a real treat to chat with Fei and Evan.

• Adam Marblestone has been generously discussing and contextualizing my perspective on neuroscience and AI, which he understands quite well:

  Dwarkesh Podcast

  Adam Marblestone — AI is missing something fundamental about the brain

  Listen now

  7 months ago · 86 likes · 7 comments · Dwarkesh Patel

Asterisk Magazine

The Sweet Lesson of Neuroscience

In the early years of modern deep learning, the brain was a North Star. Ideas like hippocampal replay — the brain’s way of rehearsing past experience — offered templates for how an agent might learn from memories. Meanwhile, work on temporal-difference learning…

Read more

5 months ago · 10 likes

New blog post: “The brain is a machine that runs an algorithm”

This one’s just a quick little rant. Here’s the link:

https://www.lesswrong.com/posts/eKGjwRSQD3BLxmBcu/the-brain-is-a-machine-that-runs-an-algorithm

And here’s the start:

Some people say “the brain is a computer”. Other people say “well, the brain is not really a computer, because, like, what’s the hardware versus the software?” I agree: “the brain is a computer” is kinda missing the mark. I prefer: “the brain is a machine that runs an algorithm”.

---

Here’s a mechanical adder:

What’s “hardware versus software” for a mechanical adder? The question is nonsense.

A mechanical adder is not “a computer”, analogous to a MacBook. Rather, it’s a machine that runs an algorithm. (Namely, the binary addition algorithm.)

And the brain is likewise a machine that runs a (much more complicated) algorithm.

---

“A machine??”, you say. Yeah, you heard me. A machine. An extraordinarily complex machine, but a machine all the same. If you could zoom in enough to really see it, it would just be obvious! You should pause here to marvel at some molecular simulations of cell biology in action: DNA replication, more crazy DNA stuff, kinesin, and so on. …

Click the link to read the rest!

• “The nature of LLM algorithmic progress”: https://www.lesswrong.com/posts/sGNFtWbXiLJg2hLzK/the-nature-of-llm-algorithmic-progress

• “In (highly contingent!) defense of interpretability-in-the-loop ML training”: https://www.alignmentforum.org/posts/ArXAyzHkidxwoeZsL/in-highly-contingent-defense-of-interpretability-in-the-loop

The first one has a bit of Cunningham’s Law energy: it’s some spicy hot takes far outside my area of expertise. Here’s how it starts:

The nature of LLM algorithmic progress

There’s a lot of talk about “algorithmic progress” in LLMs, especially in the context of exponentially-improving algorithmic efficiency. For example:

• Epoch AI: “[training] compute required to reach a set performance threshold has halved approximately every 8 months”.

• Dario Amodei 2025: “I’d guess the number today is maybe ~4x/year”.

• Gundlach et al. 2025a “Price of Progress”: “Isolating out open models to control for competition effects and dividing by hardware price declines, we estimate that algorithmic efficiency progress is around 3× per year”.

It’s nice to see three independent sources reach almost exactly the same conclusion—halving times of 8 months, 6 months, and 7½ months respectively. Surely a sign that the conclusion is solid!

…Haha, just kidding! I’ll argue that these three bullet points are hiding three totally different stories. The first two bullets are about training efficiency, and I’ll argue that both are deeply misleading (each for a different reason!). The third is about inference efficiency, which I think is right, and mostly explained by distillation of ever-better frontier models into their “mini” cousins.

Tl;dr / outline

• §1 is my attempted big-picture take on what “algorithmic progress” has looked like in LLMs. I split it into four categories:

  • §1.1 is stereotypical “algorithmic efficiency improvements” related to the core learning algorithm itself (architectures, tokenizers, optimizers, etc.). I’ll argue that the idea of using a Transformer with optimized hyperparameters was an important idea, but apart from that, the field has produced maybe 3-5× worth of stereotypical “algorithmic efficiency improvements” in the entire period from 2018 to today (≈20%/year). That’s something, but it’s very much less than the exponentials suggested at the top.

  • §1.2 is “optimizations”. I figure there might be up to 20× improvement by optimizing the CUDA code, parallelization strategies, precision, and so on, especially if we take the baseline to be a sufficiently early slapdash implementation. But the thing about “optimizations” is that they have a ceiling—it’s not an exponential that can keep growing and growing.

  • §1.3 is “data-related improvements”, including proprietary human expert data, and various types of model distillation, both of which have important effects.

  • §1.4 is “algorithmic changes that are not really quantifiable as ‘efficiency’”, including RLHF, RLVR, multimodality, and so on. No question that these are important, and we shouldn’t forget that they exist, but they’re not directly related to the exponential-improvement claims at the top.

• §2 is why I don’t believe either Epoch AI or Dario, in their claims of exponential training-efficiency improvements (see top).

• §3 is a quick sanity-check, studying “nanochat”, which matches GPT-2 performance but costs 600× less to train.

• §4 is an optional bonus section on why I care about this topic in the first place. (Not what you expect! Unlike everyone else reading this, I don’t particularly care about forecasting future LLM progress.)

Click the link to read the rest!

The second one is much more in my comfort zone. Here’s a teaser:

In (highly contingent!) defense of interpretability-in-the-loop ML training

Let’s call “interpretability-in-the-loop training” the idea of running a learning algorithm that involves an inscrutable trained model, and there’s some kind of interpretability system feeding into the loss function / reward function.

Interpretability-in-the-loop training has a very bad rap (and rightly so). Here’s Yudkowsky 2022:

When you explicitly optimize against a detector of unaligned thoughts, you’re partially optimizing for more aligned thoughts, and partially optimizing for unaligned thoughts that are harder to detect. Optimizing against an interpreted thought optimizes against interpretability.

Or Zvi 2025:

The Most Forbidden Technique is training an AI using interpretability techniques.

An AI produces a final output [X] via some method [M]. You can analyze [M] using technique [T], to learn what the AI is up to. You could train on that. Never do that.

You train on [X]. Only [X]. Never [M], never [T].

Why? Because [T] is how you figure out when the model is misbehaving.

If you train on [T], you are training the AI to obfuscate its thinking, and defeat [T]. You will rapidly lose your ability to know what is going on, in exactly the ways you most need to know what is going on.

Those bits of optimization pressure from [T] are precious. Use them wisely.

This is a simple argument, and I think it’s 100% right.

But…

Consider compassion in the human brain. I claim that we have an innate reward function that triggers not just when I see that my friend is happy or suffering, but also when I believe that my friend is happy or suffering, even if the friend is far away. So the human brain reward can evidently get triggered by specific activations inside my inscrutable learned world-model.

Thus, I claim that the human brain incorporates a form of interpretability-in-the-loop RL training.

Inspired by that example, I have long been an advocate for studying whether and how one might use interpretability-in-the-loop training for aligned AGI. See for example Reward Function Design: a starter pack sections 1, 4, and 5.

My goal in the post is to briefly summarize how I reconcile the arguments at the top with my endorsement of this kind of research program.

Click the link to read the rest!

https://www.alignmentforum.org/posts/hiiguxJ2EtfSzAevj/are-there-lessons-from-high-reliability-engineering-for-agi

And here’s the intro:

This post is partly a belated response to Joshua Achiam, currently OpenAI’s Head of Mission Alignment:

If we adopt safety best practices that are common in other professional engineering fields, we’ll get there … I consider myself one of the x-risk people, though I agree that most of them would reject my view on how to prevent it. I think the wholesale rejection of safety best practices from other fields is one of the dumbest mistakes that a group of otherwise very smart people has ever made. —Joshua Achiam on Twitter, 2021

“We just have to sit down and actually write a damn specification, even if it’s like pulling teeth. It’s the most important thing we could possibly do,” said almost no one in the field of AGI alignment, sadly. … I’m picturing hundreds of pages of documentation describing, for various application areas, specific behaviors and acceptable error tolerances … —Joshua Achiam on Twitter (partly talking to me), 2022

As a proud member of the group of “otherwise very smart people” making “one of the dumbest mistakes”, I will explain why I don’t think it’s a mistake. (Indeed, since 2022, some “x-risk people” have started working towards these kinds of specs, and I think they’re the ones making a mistake and wasting their time!)

At the same time, I’ll describe what I see as the kernel of truth in Joshua’s perspective, and why it should be seen as an indictment not of “x-risk people” but rather of OpenAI itself, along with all the other groups racing to develop AGI.

Click the link to read the rest!

The thing itself is at the same links as before:

• As a series of 15 blog posts on LessWrong / Alignment Forum: https://www.alignmentforum.org/s/HzcM2dkCq7fwXBej8

• As a 225-page PDF (now up to version 3): https://osf.io/preprints/osf/fe36n

• Summary video & transcript: This link

And the abstract is also the same as before:

Abstract: Suppose we someday build an Artificial General Intelligence algorithm using similar principles of learning and cognition as the human brain. How would we use such an algorithm safely?

I will argue that this is an open technical problem, and my goal in this post series is to bring readers with no prior knowledge all the way up to the front-line of unsolved problems as I see them.

Post #1 contains definitions, background, and motivation. Then Posts #2–#7 are the neuroscience, arguing for a picture of he brain that combines large-scale learning algorithms (e.g. in the cortex) and specific evolved reflexes (e.g. in the hypothalamus and brainstem). Posts #8–#15 apply those neuroscience ideas directly to AGI safety, ending with a list of open questions and advice for getting involved in the field.

A major theme will be that the human brain runs a yet-to-be-invented variation on Model-Based Reinforcement Learning. The reward function of this system (a.k.a. “innate drives” or “primary rewards”) says that pain is bad, and eating-when-hungry is good, etc. I will argue that this reward function is centered around the hypothalamus and brainstem, and that all human desires—even “higher” desires for things like compassion and justice—come directly or indirectly from that reward function. If future programmers build brain-like AGI, they will likewise have a reward function slot in their source code, in which they can put whatever they want. If they put the wrong code in the reward function slot, the resulting AGI will wind up callously indifferent to human welfare. How might they avoid that? What code should they put in—along with training environment and other design choices—such that the AGI won’t feel callous indifference to whether its programmers, and other people, live or die? No one knows—it’s an open problem, but I will review some ideas and research directions.

So what’s new?

I went through the whole thing and made a bunch of edits and additions, based on what I’ve (hopefully) learned since my last big round of edits 18 months ago.

See the official announcement blog post for fifteen highlights from the changelog—including newly-added spicy hot takes on brain plasticity, AGI timelines, the definitions of “AGI” and “alignment”, collective intelligence, interpretability, instrumental convergence, and more.

https://www.alignmentforum.org/posts/CF4Z9mQSfvi99A3BR/my-agi-safety-research-2025-review-26-plans

And here’s the outline:

1. Background & threat model

2. The theme of 2025: trying to solve the technical alignment problem

3. Two sketchy plans for technical AGI alignment

4. On to what I’ve actually been doing all year!

   1. Thrust A: Fitting technical alignment into the bigger strategic picture

   2. Thrust B: Better understanding how RL reward functions can be compatible with non-ruthless-optimizers

   3. Thrust C: Continuing to develop my thinking on the neuroscience of human social instincts

   4. Thrust D: Alignment implications of continuous learning and concept extrapolation

   5. Thrust E: Neuroscience odds and ends

   6. Thrust F: Economics of superintelligence

   7. Thrust G: AGI safety miscellany

   8. Thrust H: Outreach

5. Other stuff

   1. Non-work-related blog posts

   2. Personal productivity and workflow

6. Plan for 2026

7. Acknowledgements

Click the link to read the whole thing!

https://www.alignmentforum.org/posts/oxvnREntu82tffkYW/we-need-a-field-of-reward-function-design

https://www.alignmentforum.org/posts/xw8P8H4TRaTQHJnoP/reward-function-design-a-starter-pack

And here’s a teaser for the first one:

We need a field of Reward Function Design

(Brief pitch for a general audience, based on a 5-minute talk I gave.)

Let’s talk about Reinforcement Learning (RL) agents as a possible path to Artificial General Intelligence (AGI)

My research focuses on “RL agents”, broadly construed. These were big in the 2010s—they made the news for learning to play Atari games, and Go, at superhuman level. Then LLMs came along in the 2020s, and everyone kinda forgot that RL agents existed. But I’m part of a small group of researchers who still thinks that the field will pivot back to RL agents, one of these days. (Others in this category include Yann LeCun and Rich Sutton & David Silver.)

Why do I think that? Well, LLMs are very impressive, but we don’t have AGI (artificial general intelligence) yet—not as I use the term. Humans can found and run companies, LLMs can’t. If you want a human to drive a car, you take an off-the-shelf human brain, the same human brain that was designed 100,000 years before cars existed, and give it minimal instructions and a week to mess around, and now they’re driving the car. If you want an AI to drive a car, it’s … not that.

Anyway, human brains are the only known example of “general intelligence”, and they are “RL agents” in the relevant sense (more on which below). Additionally, as mentioned above, people are working in this direction as we speak. So, seems like there’s plenty of reason to take RL agents seriously.

So the upshot is: we should contingency-plan for real RL agent AGIs—for better or worse.

Reward functions in RL

If we’re talking about RL agents, then we need to talk about reward functions. Reward functions are a tiny part of the source code, with a massive influence on what the AI winds up doing.

… [snip] …

We need a (far more robust) field of “reward function design”

So here’s the upshot: let’s learn from biology, let’s innovate in AI, let’s focus on AI Alignment, and maybe we can get into this Venn diagram intersection, where we can make headway on the question of what kind of reward function would lead to an AGI that intrinsically cares about our welfare. As opposed to callous sociopath AGI. (Or if no such reward function exists, then that would also be good to know!)

Oh man, are we dropping this ball

You might hope that the people working most furiously to make RL agent AGI—and claiming that they’ll get there in as little as 10 or 20 years—are thinking very hard about this reward function question.

Nope!

…

Click the link to read the rest!

And here’s a teaser for the second one:

Reward Function Design: a starter pack

In the companion post We need a field of Reward Function Design, I implore researchers to think about what RL reward functions (if any) will lead to RL agents that are not ruthless power-seeking consequentialists. And I further suggested that human social instincts constitutes an intriguing example we should study, since they seem to be an existence proof that such reward functions exist. So what is the general principle of Reward Function Design that underlies the non-ruthless (“ruthful”??) properties of human social instincts? And whatever that general principle is, can we apply it to future RL agent AGIs?

I don’t have all the answers, but I think I’ve made some progress, and the goal of this post is to make it easier for others to get up to speed with my current thinking.

What I do have, thanks mostly to work from the past 12 months, is five frames / mental images for thinking about this aspect of reward function design. These frames are not widely used in the RL reward function literature, but I now find them indispensable thinking tools. These five frames are complementary but related—I think they’re kinda poking at different parts of the same elephant.

I’m not yet sure how to weave a beautiful grand narrative around these five frames, sorry. So as a stop-gap, I’m gonna just copy-and-paste them all into the same post, which will serve as a kind of glossary and introduction to my current ways of thinking. Then at the end, I’ll list some of the ways that these different concepts interrelate and interconnect. The concepts are:

• Section 1: “Behaviorist vs non-behaviorist reward functions” (terms I made up)

• Section 2: “Inner alignment”, “outer alignment”, “specification gaming”, “goal misgeneralization” (alignment jargon terms that in some cases have multiple conflicting definitions but which I use in a specific way)

• Section 3: “Consequentialist vs non-consequentialist desires” (alignment jargon terms)

• Section 4: “Upstream vs downstream generalization” (terms I made up)

• Section 5: “Under-sculpting vs over-sculpting” (terms I made up).

Click the link to read the rest!

https://www.alignmentforum.org/posts/d4HNRdw6z7Xqbnu5E/6-reasons-why-alignment-is-hard-discourse-seems-alien-to

And here’s the summary:

tl;dr

AI alignment has a culture clash. On one side, the “technical-alignment-is-hard” / “rational agents” school-of-thought argues that we should expect future powerful AIs to be power-seeking ruthless consequentialists. On the other side, people observe that both humans and LLMs are obviously capable of behaving like, well, not that. The latter group accuses the former of head-in-the-clouds abstract theorizing gone off the rails, while the former accuses the latter of mindlessly assuming that the future will always be the same as the present, rather than trying to understand things. “Alas, the power-seeking ruthless consequentialist AIs are still coming,” sigh the former. “Just you wait.”

As it happens, I’m basically in that “alas, just you wait” camp, expecting ruthless future AIs. But my camp faces a real question: what exactly is it about human brains that allows them to not always act like power-seeking ruthless consequentialists? I find that existing explanations in the discourse—e.g. “ah but humans just aren’t smart and reflective enough”, or evolved modularity, or shard theory, etc.—to be wrong, handwavy, or otherwise unsatisfying.

So in this post, I offer my own explanation of why “agent foundations” toy models fail to describe humans, centering around a particular non-“behaviorist” RL reward function in human brains that I call Approval Reward, which plays an outsized role in human sociality, morality, and self-image. And then the alignment culture clash above amounts to the two camps having opposite predictions about whether future powerful AIs will have something like Approval Reward (like humans, and today’s LLMs), or not (like utility-maximizers).

(You can read this post as pushing back against pessimists, by offering a hopeful exploration of a possible future path around technical blockers to alignment. Or you can read this post as pushing back against optimists, by “explaining away” the otherwise-reassuring observation that humans and LLMs don’t act like psychos 100% of the time.)

Finally, with that background, I’ll go through six more specific areas where “alignment-is-hard” researchers (like me) make claims about what’s “natural” for future AI, that seem quite bizarre from the perspective of human intuitions, and conversely where human intuitions are quite bizarre from the perspective of agent foundations toy models. All these examples, I argue, revolve around Approval Reward. They are:

• 1. The human intuition that it’s normal and good for one’s goals & values to change over the years

• 2. The human intuition that ego-syntonic “desires” come from a fundamentally different place than “urges”

• 3. The human intuition that kindness, deference, and corrigibility are natural

• 4. The human intuition that unorthodox consequentialist planning is rare and sus

• 5. The human intuition that societal norms and institutions are mostly stably self-enforcing

• 6. The human intuition that treating other humans as a resource to be callously manipulated and exploited, just like a car engine or any other complex mechanism in their environment, is a weird anomaly rather than the obvious default

Click the link to read the rest!

https://www.lesswrong.com/posts/fPxgFHfs5yHzYqgG7/social-drives-2-approval-reward-from-norm-enforcement

And here’s the intro and summary:

(Follow-up to Social drives 1: “Sympathy Reward”, from compassion to dehumanization, but this post is self-contained.)

1. Intro & summary

1.1 Background

In Intro to Brain-Like-AGI Safety (2022), I argued: (1) We should view the brain as having a reinforcement learning (RL) reward function, which says that pain is bad, eating-when-hungry is good, and dozens of other things (sometimes called “innate drives” or “primary rewards”); and (2) Reverse-engineering human social innate drives in particular would be a great idea—not only would it help explain human personality, mental health, morality, and more, but it might also yield useful tools and insights for the technical alignment problem for Artificial General Intelligence.

Then in Neuroscience of human social instincts: a sketch (2024), I worked towards that goal of reverse-engineering human social drives, by proposing what I called the “compassion / spite circuit”, centered around a handful of (hypothesized) interconnected neuron groups in the hypothalamus and brainstem (but also interacting with other brain regions; see that link for gory details). I suggested that this circuit is central to our social instincts, underlying not only compassion and spite, but also (surprisingly) much of status-seeking and norm-following.

Finally, in the previous post, I drew a 2×2 table for when the “compassion / spite circuit” fires:

1.2 This post: “Approval Reward”

The previous post elaborated on “Sympathy Reward”, and now this post will cover “Approval Reward”. (I’m much less interested in the other two.) Approval Reward leads to:

• Pleasure (positive reward) when my friends and idols seem to have positive feelings about me, or about something related to me, or about what I’m doing;

• Displeasure (negative reward, a.k.a. punishment) when my friends and idols seem to have negative feelings about me, or about something related to me, or about what I’m doing;

• …which also generalizes to pleasure from merely imagining those kinds of situations.

1.3 Summary of the rest of the post

• In §2, I’ll list three obvious, straightforward effects of Approval Reward:

  • Credit-seeking & blame-avoidance;

  • Wanting to feel liked / admired (a.k.a. wanting high social status);

  • Norm-following & norm-enforcement.

• In §3, I’ll go through the profound effects of Approval Reward on our moment-to-moment thoughts and consciousness:

  • In §3.1, I suggest that Approval Reward deeply impacts our self-image, such that the way we think of ourselves is closely aligned with what would garner social approval. This observation is well-known in evolutionary psychology, but usually explained as a specific evolved trait, or as motivated reasoning. I agree with the observation, but not those proposed mechanisms, and I offer my own alternative account of how this works under the hood.

  • In §3.2, I suggest that almost all of us are in the throes of an intense, almost nonstop addiction to Approval Reward. All it takes is a quick, ever-so-subtle turn of one’s attention to how one might look from the outside right now, and you can get an immediate squirt of Approval Reward. (The resulting nice feeling is sometimes called “pride”.) I’d guess that neurotypical people thus help themselves to Approval Reward maybe 10,000 times a day or more, every day of their lives.

• In §4, I go through various other non-obvious effects of Approval Reward, including “status games” with almost-arbitrary goals, and One Weird Trick for recognizing sociopaths.

• In §5, I argue that Approval Reward interacts with Typical Mind Fallacy to create a drive to share one’s own special interests, especially in socially-inattentive nerds and kids.

• §6 discusses the limitations of norm-following motivation and other consequences of Approval Reward. For example, do people want to follow norms even when nobody will ever find out?

1.4 Teaser for upcoming posts

In the next post, I’ll pivot back to AI alignment, by arguing that human Approval Reward explains numerous areas where everyday human experience and intuitions make us expect one thing, while “rational agents” / “agent foundations” theory make us expect something quite different. For example, humans are happy to have their goals change as they grow older (contra instrumental convergence), and humans are often helpful and deferential (contra “corrigibility is anti-natural”), etc.

After that post, I’ll (hopefully) finally get to the question of whether we can and should put Sympathy Reward and/or Approval Reward into a future brain-like Artificial General Intelligence.

Click the link to read the rest!

https://www.lesswrong.com/posts/KuBiv9cCbZ6ALjHFw/social-drives-1-sympathy-reward-from-compassion-to

And here’s the intro and summary:

1. Intro & summary

1.1 Background

In Intro to Brain-Like-AGI Safety (2022), I argued: (1) We should view the brain as having a reinforcement learning (RL) reward function, which says that pain is bad, eating-when-hungry is good, and dozens of other things (sometimes called “innate drives” or “primary rewards”); and (2) Reverse-engineering human social innate drives in particular would be a great idea—not only would it help explain human personality, mental health, morality, and more, but it might also yield useful tools and insights for the technical alignment problem for Artificial General Intelligence.

Then in Neuroscience of human social instincts: a sketch (2024), I worked towards that goal of reverse-engineering human social drives, by proposing what I called the “compassion / spite circuit”, centered around a handful of (hypothesized) interconnected neuron groups in the hypothalamus and brainstem (but also interacting with other brain regions; see that link for gory details). I suggested that this circuit is central to our social instincts, underlying not only compassion and spite, but also (surprisingly) much of status-seeking and norm-following.

1.2 Summary of this post

The next task is to dive into the “compassion / spite circuit” more systematically, trying to build an ever-better bridge that connects from neuroscience & algorithms on one shore, to the richness of everyday human experience on the other. In particular:

• Section 2 will introduce a framework for thinking about the “compassion / spite circuit”, by splitting it into four reward streams with different downstream effects. I call them “Sympathy Reward”, “Approval Reward”, “Schadenfreude Reward”, and “Provocation Reward”. I also review some theoretical background for how I think about rewards, desires, drives, and so on.

• Sections 3–6 will apply this framework to analyze one of those four reward streams (“Sympathy Reward”), including both its obvious and not-so-obvious consequences.

  • Topics include dehumanization, anthropomorphization, “compassion fatigue”, hedonic utilitarianism, The Copenhagen Interpretation of Ethics, and more.

1.3 Teaser for subsequent posts

After we finish this post, I have a follow-up post which will analyze “Approval Reward”, a second of those four reward streams coming from the “compassion / spite circuit”.

For the post after that—well, there’s also a third and fourth reward stream, but those are less important from an AI alignment perspective, so I’ll skip those for now. Instead, I’ll pivot back to discussing technical AI alignment more directly.

Click the link to read the rest!

…Upshot: Recommended! I ~90% agree with it.

The authors argue that people are trying to build ASI (superintelligent AI), and we should expect them to succeed sooner or later, even if they obviously haven’t succeeded YET. I agree. (I lean “later” more than the authors, but that’s a minor disagreement.) …

(It sounds like sci-fi, but remember that every technology is sci-fi until it’s invented!)

They further argue that we should expect people to accidentally make misaligned ASI, utterly indifferent to whether humans live or die, even its own creators. They have a 3-part disjunctive argument: … [more at the link]

(2) Someone was kindly interested in how they could help accelerate the research I’m working on, so I wrote a blog post Excerpts from my neuroscience to-do list with ten somewhat-self-contained projects.

(3) I wrote a little physics post, Optical rectennas are not a promising clean energy technology:

“Optical rectennas” (or sometimes “nantennas”) are a technology that is sometimes advertised as a path towards converting solar energy to electricity with higher efficiency than normal solar cells. I looked into them extensively as a postdoc a decade ago, wound up concluding that they were extremely unpromising, and moved on to other things. Every year or two since then, I run into someone who is very enthusiastic about the potential of optical rectennas, and I try to talk them out of it. After this happened yet again yesterday, I figured I’d share my spiel publicly!

…

Rectenna is short for “rectifying antenna”, i.e. a combination of an antenna (a thing that can transfer electromagnetic waves from free space into a wire or vice-versa) and a rectifier (a.k.a. diode). … [more at the link]

There’s a funny thing where economics education paradoxically makes people DUMBER at thinking about future AI. Econ textbooks teach concepts & frames that are great for most things, but counterproductive for thinking about AGI. Here are 4 examples. Longpost: … [more at the link]

(2) I wrote “Neuroscience of human sexual attraction triggers (3 hypotheses)”. Read it on LessWrong

tl;dr:

There’s a stereotype that male sexual attraction is triggered mainly by appearance, and female sexual attraction is triggered mainly by status.

…Yes I know, this stereotype is grossly oversimplified, and is only valid on the margin, and really there’s overlapping distributions, etc. etc. (+ even more caveats in §1.2 below). But it does have a kernel of truth.^[1]

Now, this post is not mainly about sex differences per se. What I actually care about is the more basic observation that “appearance-based sexual attraction” is a thing that exists in humans, and so is “status-based sexual attraction”. These stem from innate reflexes in the brain. My goal in this post is to speculate on how those innate reflexes might work, focusing on the neuroscientific “symbol grounding problem”.

For appearance-based sexual attraction (§2), I suggest two possible hypotheses. One involves innate sensory heuristics calculated in the brainstem, including visual processing by the superior colliculus. The other involves (what I call) “transient empathetic simulations” connected to the somatosensory system.

For status-based sexual attraction (§3–§6), I suggest a mechanism involving “phasic physiological arousal”. I further propose that this mechanism would also explain various other everyday observations, such as the stereotypical female attraction to tall men.

[…much more at the link]

(3) I wrote a question-post: Inscrutability was always inevitable, right? In short, it seems to me that AI will inevitably involve open-ended learning algorithms, and open-ended learning algorithms inevitably create tons of inscrutable unlabeled parameters. So why do some people seem to suggest that “AIs powered by tons of inscrutable unlabeled parameters” is a terrifying aspect of our current AI paradigm, rather than a terrifying aspect of every possible AI paradigm?

I’m glad I asked, I got some great answers! See the link.