Another participant

What you are describing sounds like narrow AIs rather than general ones. What I am unclear about is how exactly they are built. Are they still LLMs, just with a very narrow training set?

In the Taiwan example you described, may I understand the exact process? People were contacted, and so on — what happened in more detail?

Listening to how your original process used surveys to set the agenda, I found myself substituting: it seems to me this whole process could also work very well with elected officials — elected legislators setting the agenda for a committee selected by sortition. We should remember that sortition is a good way to bring people face to face in confronting national issues. And even a minority group within the legislature —

Underlying this whole vision, I would say, is an assumption that we are heading towards a world of distributed artificial intelligences: many different ones, fairly narrow in scope, each of which can be aligned to a small group of people. That is a very different technical premise from the world of a so-called singleton, where one AI develops overwhelming capability. I do not have strong views on which world we are heading towards; it turns on how large the returns to recursive self-improvement really are, and on how much difference intelligence makes. If a runaway capability gap opens between the leading frontier AI and all the others, then all of this becomes irrelevant. But perhaps that is not where we end up. Perhaps we get something closer to Iain Banks’s Culture novels — many, many AI agents, different but broadly comparable, with overlapping capabilities — and then this discussion of how to align them is the one we need.

Of the world government, presumably.

I follow that idea. But how does it connect to the process you ran? How would people plug into this — how would the training process change to take account of the human collectivity you have in mind?

That part I completely follow, and the paper is fantastic. But let me ask this. Today the temptation for, say, mathematicians is to seek an internal appointment at one of the big companies: go in for six months, work with their model, and get your name on a paper that people outside will just nod along to. Presumably you are imagining an AI ecosystem that also encourages human mathematicians to keep talking to other human mathematicians.

Let me come in here, because I find this a really fun project. I would genuinely like to work out how to replicate it — we may be setting up a small server rack in a family home, and a family AI there is appealing. I would be curious to see what my own family’s version would look like.

It also seems to mesh very well with ideas about guardian-angel AIs: fiduciary AIs that try to learn your preferences and act as your representative, your interface and, in a sense, your protector. I would be very curious to hear how work along those lines is going.

But I think what you are proposing is stronger. There is a pessimistic view — one I suppose I share — that the world is going to hell in a handbasket, in which case it would be nice to have someone on your side. What you want goes further: for the entire infrastructure to evolve in the right way.

Related to this: a colleague got in touch recently to say that the work they had been doing for one of the frontier labs felt rather pointless — they mostly babysit the model, which does more or less everything — so what are we going to do? My answer was that their scientific community should train a model of its own, as an open-source project, with everyone in the community on the paper. At least for a while, that would bring the satisfaction of building something together, rather than simply handing it all over to one of the two big companies. That was a thought prompted by some of this. Part of your vision, I think, is to steer AI development, if we can, so that instead of producing an ever-larger brain, we —

I like these Kamis you are describing. This would be a bright future, and it is a really important idea. In some ways they do displace people — are you displacing people who do mediation? There is a body of theory showing that a benevolent intermediary who holds information in confidence can get people to communicate and bargain more, and to offer each other more concessions, as long as those concessions are not immediately heard by the other negotiating parties.

Automating mediation is, in a sense, the ideal kind of optimization. Yes, it will put some social workers out of jobs, and that is a good profession. But an AI devoted to the professional standards of helping community groups work effectively — groups of 200 to 500 people, say, or bounded by the 250 or so of Dunbar’s number — would be AI that genuinely complements human intelligence, because it helps people work together in their own groups.

One more thought. When you talk about different groups, and when you work with Bluesky or X to improve their algorithms, you are generating knowledge of communities. There is a reference you should know, if you do not already: chapter 7 of Thomas Hobbes’s Leviathan . Do you know the passage where he discusses the meaning of the word “conscience”?

This is really interesting, and it takes some time to get one’s head around. You have explained what you would like to see in the world, and I think we now have some idea of it. It sounds as though you see it as something to do instead of those other things. It seems to me there are two parts to the project: one is to build this, and the other is to stop the other thing from being built. Is that right? Would the idea be to get the big AI companies to build this as well, or instead — or for them simply to disappear? How do you see it?

A couple of thoughts. The world currently has some kind of balance between more democratic, egalitarian or liberal tendencies on the one hand, and more nationalistic, selfish or adversarial tendencies on the other. Presumably that balance reflects different forces battling it out over hundreds of years; it is the current state of play. So why would we expect it to change? One answer: if a new technology or cluster of ideas makes collaborative efforts differentially easier, it might give that side a boost and shift the balance.

That raises a few questions. First, could some version of the same capability be used by the other side? One can imagine an entrenched power working out a version of this that helps it stay in power — by placating its fiercest opponents, or by coordinating better internally so that it can act as a unified entity. A group or country using it might be strengthened so that it can serve its internal constituencies more efficiently, at the expense of other nations it currently needs to trade with or has limited power over.

Second, and related: how robust is this against manipulation? When the stakes are fairly small and the issue is technical, perhaps with some widely appealing pragmatic solutions, that is one setting. Once you bring in major decisions — large sums of money, geopolitical consequences — and scale up to large nation states, where two parties with party discipline get hijacked by various interest groups and everything becomes fiercer and less idealistic, I wonder whether similar dynamics would start to emerge within these systems, in non-obvious ways, once the stakes are high enough.

To push that question a little further: could you manufacture consent? Suppose Xi Jinping’s Kami is also trained on Xi Jinping Thought, so that in every discussion it gently steers. That slides towards manufacturing an enormous amount of consent, through processes that on their face seem not to be doing so. A propaganda engine, in other words, but operating at scale and very subtly, so that every group of ten people believes it reached its conclusion unprompted. Is that the kind of risk being raised?

One way to put it: imagine two communication protocols or infrastructures. The first, which I think is what you outlined, helps participants work out what they want while also taking some account of the interests of neighbouring communities — people outside the process itself. The second is identical, except that it cares about what outsiders think only instrumentally. Groups that are already inclined to be nice might choose the first, which plays well with others, while groups that simply want to maximize their own returns would choose the second.

That interoperability point is really interesting. Outside the political sphere, moving your data from one cloud provider to another has been a major issue, partly because most providers are happy to make it hard, and partly because moving large amounts of data genuinely is hard.

But I want to return to common knowledge. Someone I know who is doing a PhD in education once remarked, almost in passing, that one of the main purposes of school is to generate common knowledge. The point is not that we remember what we learned, but that everyone is supposed to have learned it. That gives us an understanding of what is expected of us, which we can then use to form expectations of others — and that is enormously effective in making a society actually work.

One of our biggest challenges may be building tools that give us common knowledge within a more distributed structure. If we each live in AI-mediated bubbles, perhaps the AIs can maintain some common knowledge among themselves, but we probably want common knowledge too; it is essential to functioning as a society. Can we find ways of boosting it? This connects to Bruce Schneier ’s fine essay on common knowledge and attacks on democracy, which argues that authoritarian states like to undermine common knowledge in democracies because doing so seriously weakens them. So methods of strengthening common knowledge matter — and, of course, they are also a good way of making trouble for authoritarianism.

When we talk about national cohesion — about our largest social structures, whether that is a nation or, for Europeans today, the European Union — I think common knowledge has to be created at something like a national assembly. That could involve sortition or election, but it is not about majority rule; minority groups should —

Here is one mechanism for creating democratic common knowledge: a public medium that reaches everyone, that is salient and accessible to all, and that people are encouraged to watch, featuring regular debates and discussions among panels. Any group holding a large enough share of the assembly — 25 percent, say, or 20; you choose the group quota — can put someone on the panel. With a quota of 1/(K+1), where K is the number of seats on the panel, a minority group gets one seat for each quota it holds. For a five-member panel, anything more than one-sixth of the assembly is enough to endorse one panellist.

Do you have a better mechanism for creating public discussion about truths — for isolating areas of disagreement, defining propositions that are widely accepted while registering minority views, and creating common knowledge of the minority views as well as the majority consensus?

Would the bridging bonus give people an incentive to misrepresent their views, or give undue influence to an extreme outlier — a lone person with an outlandish view — so that every proposal gets a big bonus for somehow accommodating them? Or might it even encourage people to profess more extreme views, so as to pull the consensus proposal towards them?

We are at the 90-minute mark, and it has been a fantastic discussion. Audrey, thank you so much for agreeing to do this at short notice. The question of how long it will be before LLMs grow impatient with us — what their intrinsic sense of time is — will have to wait for a future discussion. We will have to be more patient.