Participant

Any questions?

So underneath, it is still a language model — and therefore generalizable — but it is made narrow by training on a narrow dataset.

I see. So it is still a general AI, but trained and used in a narrow way.

Sorry — just to get the mechanics: how many people were involved at the start? And did each of them upvote or downvote the others?

And once that was done, could you fill in the remaining steps? You said it ultimately led to legislation — could you take us through it slowly?

So you have juries of 10 people meeting with AI agents that act, essentially, as moderators or facilitators.

And what was the next layer?

Sorry — was there a plenary with free discussion among all 440-odd people, or did only the ideas flow between rooms?

So the idea is to take a human group small enough for coherent dialogue, and keep deliberation at that level.

Very nice. Returning to the larger vision: how do you see this scaling up to more powerful jurisdictions — perhaps even to more powerful countries?

If I may offer an intuition for what you are describing: this Kami-style setup strikes me as a way of trying to scale something like Swiss-style direct democracy.

Yes. But I still cannot bridge the gap between what you described — democratic, better decision-making — and the problem of concentrated intellectual firepower. Suppose you could change how AI develops from here on: what would happen differently if you were in charge? What happens to OpenAI? What happens to Anthropic? What kinds of AI do we build?

So at a technical level, how should AI models develop so that communities of mathematicians — and of physicists — still want to work with one another? Some people are perfectly content just talking to Claude, but setting that aside, do you have thoughts?

Yes, exactly.

One more question. These models are quite polymathic, and even if development stopped today, the smartphone has taught us that people are very easily drawn into interacting with machines rather than with other people.

Yes. A whole generation of children has grown up in exactly this period. How does one deal with that?

I am still trying to think, at the systems level, about the stability of this and what problem it solves. Humans talk to a Kami that is only a little smarter than they are; the Kamis talk to a super-Kami that is only a little smarter than they are; and so on up the chain.

So instead of retiring old instances of ChatGPT and Claude as newer versions make them obsolete, you keep their soul alive — distributed, living in many places. Perhaps humans talk not directly to the frontier model but to models a few steps removed, which talk to their successors, which in turn talk to theirs. That sets up a distributed power base for general systems too, not only narrow ones.

And it is a small system, which might suit what we have in mind — a Kami for a household.

“Conscience” shares a root with “conscious” — and we are trying to work out what “conscious” means. The original meaning is essentially common knowledge: your conscience is what you know together with the other members of your endogenously defined group. And in Hobbes’s time, with hundreds of religions but only one source, there was also a richness of —

For Hobbes, “no one should be forced to deny their conscience” meant that no one should be forced to deny what their group holds to be an important, commonly believed proposition. But the point is this: a bot professionally devoted not to taking over anything, but to helping people communicate with one another — asking questions that lead them towards consensus, helping them see which topics they want to focus on, and helping them build an agenda to pursue those topics in a balanced way. If you can get balanced contributions from all the members, I think that is huge.

Could a policy of encouraging and subsidizing products like this help turn back the charge towards building super-AGI, which is an entirely different project? I love this. Automating mediation is one way AI can truly complement human intelligence without helping to build centralized power structures. The magic of humanity — the reason humanity took over the planet in the first place — was local organization, and our local and regional networking abilities, which we are very good at. The creation of global networking elites over the past 5,000 years is the new thing, and the super-AGI project is putting it on even more steroids.

What concerns me most — and this is anecdotal, from meeting people in the tech industry — is hearing about founders creating major new companies with almost no other humans, using AI to run a growing enterprise. The technology you describe instead enables groups of people to work together and become more effective at creating enterprises, or whatever else they want to do. That is a far more socially productive use of it.

That is very helpful. One difficulty for people calling for a moratorium — this far and no further in this direction — is where to channel that energy. If you can say, “Do not go any further in this direction, but you are welcome to proceed in this other direction, which would also give people what they want,” that would be a very useful policy recommendation.

When nationalism began in Europe, the French were famously first. But when they marched through Germany, it was the German General Staff — Scharnhorst and Gneisenau , I believe — who, military establishment though they were, wanted German nationalism, because it made it easier to raise large armies. So it was an idea — a coordination protocol — that spread.

At a high level, this will certainly help on our side, in democratic societies, and I think it will certainly help us deal with the AI problem. But — correct me if I have not stated the concern correctly — the worry is that it could turn out to be a double-edged tool, usable by the other side as well.

On X: although there are many polarized groups there, X itself is a kind of meta-community. You could imagine X being less keen on a feature that says, “Here is a really interesting post on Bluesky you might want to read instead,” and funnels traffic over there — and Bluesky might be equally reluctant to send people to great posts on X. The leaders of each want happy users who get along better internally, and more engagement, but it is less clear that they want to cooperate beyond their own platform.

At the global level, though, nations may not want full interoperability — in the sense of everyone being free to move wherever they like, to choose where they pay taxes, or to get a vote on which country should be invaded, or something like that.

There is a perfect example, worked out by a student, in which quadratic voting fails terribly.

It is essentially the case in which the experts know whether or not there is a problem.

Exactly — a model of patience.

Thank you, everyone.