Blog post: “Notes on technical alignment via human-like social drives”
Here’s the link:
And here’s how it begins:
1. Frontmatter
1.1 Backstory for this post
As discussed in Intro to Brain-Like-AGI Safety, I’m working on the technical alignment problem for a hypothetical future “brain-like AGI”, with a particular focus on treating human innate social and moral drives as a possible jumping-off point for our technical alignment approach.
After all, if it’s possible for humans to do stuff that ultimately leads to a good future, then it’s probably also possible for sufficiently human-like AGIs to do stuff that ultimately leads to a good future. Or if it’s not possible for humans to do stuff that ultimately leads to a good future, then we’re screwed no matter what. But assuming it’s possible, the “sufficiently human-like AGIs” would certainly need to have good prosocial motivations. What code do we write that would lead to good prosocial motivations? It’s an unsolved problem (see “We need a field of Reward Function Design” (2025)), but as we search for a solution, we might look for inspiration at how humans (sometimes) wind up with good prosocial motivations.
I’ve been working on this problem for years, but most of that work has involved laying foundations (e.g. trying to understand how human social drives work). Whereas in the past four months, I’ve been thinking very directly about how to apply those ideas to AGI.
I’ve published two little things from this four-month effort—“Act-based approval-directed agents”, for IDA skeptics, and Empowerment, corrigibility, etc. are simple abstractions (of a messed-up ontology)—but most of what I (think I) figured out is not self-contained, but rather part of a big interconnected mess of thoughts and ideas. So for now I’m just dumping that all into this one excessively-long post. Sorry.
This post has lots of new-to-me ideas and opinions, and they are in great need of more scrutiny and especially deconfusion. I’m happy for any feedback, pushback, and riffs on anything herein.
1.2 Table of contents / tl;dr
Sections 2–5 start from human social instincts, and ponder how to usefully employ something-like-that in an AGI. Specifically:
Section 2 goes over the high-level approach of using human social instincts as a starting point / inspiration for an AGI motivation system. What are the specific instincts in question, how do they work in humans, what roles if any should they be playing in AGI, and what should we be keeping in versus leaving out?
Section 3 discusses three potential failure modes that I’ve spent a long time thinking about: the possibility that the AGI will wind up with the wrong “moral circle”; the possibility that virtue-ethics-y human motivations (honesty, helpfulness, etc.) rely on a balance-of-power dynamic which wouldn’t apply to the ASI-human relationship; and the possibility that consequentialist desires will squash virtue-ethics-y desires in the long run, when both are present (as I claim they need to be).
Section 4 is “What controls the set of virtues that the AGI takes pride in?” I discuss two pathways: “person-first” and “desire-first”. For example, someone could wind up taking pride in their encyclopedic knowledge of Disney princesses because that’s what the cool older kids in school are into (“person-first”); or because they really really like Disney princess movies, and that love has gradually wormed its way into their self-image (“desire-first”). I suggest that the “desire-first” pathway is an important way that nerds like me wind up motivated to figure out the truth and share it with others—a key trait that we may want in brain-like AGI. (Hold that thought!)
Section 5 goes over some implementation details related to transplanting human social drives into the foreign soil of AGI source code.
Then in Section 6 I switch from forward-reasoning (starting with human social instincts) to backwards-reasoning (starting with desiderata), by asking: What AGI motivations do we want anyway?
Section 6.1 goes over three sets of constraints that I’m trying to satisfy: technical alignment constraints (we need to be able to write the code), strategic constraints (we need to make the world resilient to misaligned ASI), and ethical / societal / buy-in constraints (the plan needs to sound reasonable, such that people will actually follow it).
Section 6.2 goes over a bunch of possible AGI motivations, and how they seem to stack up against those three sets of constraints. I wind up tentatively advocating for a high-level approach that I’ve been calling “truth-seeking disagreeable nerd AGI”, using the technical alignment idea mentioned in §4 above, and then have that AGI figure out what to do next.
Section 7 has a couple more random things from my notes:
Section 7.1 discusses two (related) dilemmas that I’ve been struggling with. I call them “the visceral reaction updating dilemma” and “the value drift dilemma”.
Section 7.2 describes a subtle mistake in how I was thinking about “ruthlessness” until recently.
I close in Section 8 with what I plan to work on next and why.
Click the link to read the rest!