My belief is that keeping AI under human control would be an unprecedented global challenge, if it’s even possible at all.
Many other people’s believe that AIs will naturally remain as tools under human control. Either they see this as inevitable—“how could it be otherwise?”—or they see it as not quite inevitable, but still an outcome that can easily happen via ordinary government regulations, civil society, and so on.
Who’s right? What’s the right way to think about this?
This is an issue where people really don’t see eye-to-eye, and tend to get stuck endlessly talking past each other. To pin down the disagreement, I think that the following is a good conversation-starter:
Conversation-starting question:Do you expect almost all companies to eventually be founded and run by AIs rather than humans?
I find that people give a number of different answers:
Possible Answer 1:“No, because the best humans will always be better than large language models (LLMs) at founding and running companies.”
Huh? Whoever said anything about LLMs? I wrote “AIs”!
[…]
Possible Answer 2:“No, because the best humans will always be better than AIs at founding and running companies.”
I think that most people with this view are generally rejecting the idea that actually powerful AI (as in the right side of the table above) is possible at all, or are failing to think about its implications. Again, humans are an existence proof for what is physically possible for AI. Thus, for example:
Humans can acquire real-world first-person experience? Well, an AI could acquire a thousand lifetimes of real-world first-person experience.
[…]
Possible Answer 3:“No, because humans will always be equally good as AIs at founding and running companies.”
I think that most people with this view don’t really believe, in their guts, that skill and competence are a thing. Or at least, they don’t believe that skill and competence are relevant to founding and running successful companies.
When these people imagine what Jeff Bezos was doing day-to-day to build Amazon from scratch, I’m not quite sure what’s in their head. Befriending people? Scaring people? Puppies can also befriend and scare people, but no puppy has ever become a hundred-billionaire by founding and running a successful company.
[…]
Possible Answer 4:“No, because we will pass laws preventing AIs from founding and running companies.”
Even if such laws existed in every country on Earth, and the letter of such laws was enforceable, the spirit wouldn’t be. Rather, the laws would be trivial to work around. For example, you could wind up with companies where AIs are making all the decisions, but there’s a human frontman signing the paperwork.
[…]
Possible Answer 5:“No, because if someone wants to start a business, they would prefer to remain in charge themselves, and ask an AI for advice when needed, rather than ‘pressing go’ on an autonomous entrepreneurial AI.”
This is a really nice vision, and I wish I could believe it. But even if lots of people do in fact take this approach, and they create lots of great businesses, it just takes one person to say “Hmm, why should I create one great business, when I can instead create 100,000 great businesses simultaneously?”
