An informal discussion held on 11 September 2026 in the Jack Paton Room, Beecroft Building, Oxford, as part of the Programme on Technology, Computation and Superintelligence (TCS) series in Oxford Physics, convened by Shivaji Sondhi with Anders Sandberg as research lead. Audrey Tang's one-page note on the Oxford-based Civic AI project was the main text; a second advertised topic — how language models experience time — was held over to a future session.
The discussion was held under the Chatham House Rule: participants may use what was shared, but may not reveal who said it. With the participants' agreement, the roster below is named; in keeping with the Rule, individual contributions remain unattributed. "Participant" and "Another participant" mark only a change of speaker — they do not track any individual through the discussion and do not correspond to the order of the list.
Audrey Tang has waived the Rule for her own remarks, which appear under her own name. All other contributions are paraphrased, not quoted: names, affiliations, locations, personal details and references to other participants have been removed, and speakers are referred to in gender-neutral terms. Greetings, introductions, logistics, and a few brief remarks of hers that would have identified other participants have been removed. Links were added during editing.
Participants
- Nick Bostrom — Philosopher and Principal Researcher at the Macrostrategy Research Initiative, known for his work on superintelligence, existential risk and the future of humanity.
- Roger Myerson — David L. Pearson Distinguished Service Professor at the University of Chicago; 2007 Nobel Laureate in Economics.
- Rahul Nandkishore — Professor of Physics at the University of Colorado Boulder.
- Toby Ord — Senior Researcher at the Oxford Martin AI Governance Initiative, author of The Precipice and co-founder of Giving What We Can.
- Anders Sandberg — Researcher, futurist and computational neuroscientist; former Senior Research Fellow at Oxford's Future of Humanity Institute.
- Shivaji Sondhi FRS — Wykeham Professor of Physics at the University of Oxford and Professor Emeritus of Physics at Princeton University.
- Audrey Tang — Taiwan's Cyber Ambassador; Senior Fellow in the Accelerator Fellowship Programme at the Oxford Institute for Ethics in AI. Her remarks appear under her own name.
- Tenzin Yangtso — Audrey Tang's collaborator on the Civic AI project.
Opening: Civic AI and the 6-Pack of Care
Hello, many good friends and new friends. So, I have been developing here in Oxford for the past year or so, with a new book to be published by Wiley next February, called the 6-Pack of Care, where the 6-Pack represents portable but also muscular, as in abs, care. We call the AI systems that are aligned to this kind of care-ethics approach Civic AI. And a lot of the book, actually the entire book, is available in public domain on civic.ai. So, I will not repeat it at length. I will just outline the main arguments and then we can go into discussion.
The observation is that when we discuss AI's trajectory in society, the default trajectory with current-day AI, with RL and so on, is what many people are now calling solipsistic alignment. That is to say, AI is single-mindedly pursuing some goals and seeing the world around them, including us humans, as environmental hazards. And in this world, even when AI is perfectly aligned to that particular goal, it still supercharges conflict. So, that is the default trajectory.
In Civic AI, we want an alternate trajectory where it helps our existing communities cooperate across differences. One side we can call singularity. The other side we call Plurality. So, this is the first claim.
And the second one is that currently we do not have a very good theory on how to keep a powerful general optimizer aligned with humanity. So, maybe do not even try. Maybe just have satisficers. Maybe do not do optimizers at all. So, when a system acts in a particular place, owing to existing relationships, and with humans forming the loop that we put the AI agents in — AI in the human loop, not human in the AI loop — that is a satisficing bounded agent. A little bit like Drexler's comprehensive AI services. A non-optimizing agent that satisfies this relationship. This is the second claim.
And so we claim that this kind of system gets its reward not from solipsistic speed, because in that speed, deontological commandments or consequentialist utility calculations become incommensurable with our current civilization. So, do not do that. Care ethics starts from a different picture where people remain each other's gardeners and AI becomes local infrastructure or scaffolding. So, this is what we call Kami, or knowledge artefact management intelligence — KM for knowledge management, AI for artificial intelligence. Put those two together and you get a Kami.
In Japanese folklore, there are 8 million Kami. If you have watched Spirited Away, you know that a village has a Kami, an ujigami. A forest has a Kami. The sun has a Kami, Amaterasu. But in Japanese folklore, there is no singleton. The Kami do not report to some higher-up. They are entirely federated, and each Kami is bounded. So, the health of the Kami of the forest is the health of that forest. Nothing more, nothing less. So, it is not power-seeking in the classical reward-hacking kind of way.
So, consider a very concrete case, and then I will pause: Public service using AI.
In Taiwan, two years ago, we noticed that Facebook and YouTube had a lot more synthetic fraud advertisements than normal advertisements. It turns out their optimizer, their recommendation engine for advertisements, theorized that scammers pay much more pay-per-click than normal business. So, they featured a lot more scam than normal business. This is a classic case of a misaligned optimizer. And instead of putting censorship, because Taiwan is the most free in Asia when it comes to internet freedom, we simply randomly texted 200,000 people, our citizens, asking what is proportionate.
People gave us ideas, thousands of ideas, and many people volunteered. And then we chose randomly 447 people in a long afternoon. Each person over Zoom talking to 9 other people. We worked with Stanford. The Kami, the Civic AI in this case, does not inject the judgment, but they do carry what is called the MIMBY, maybe-in-my-backyard, the middle-ground energy — not the NIMBY, not the outrage, the polarized energy — from one table to the other, cross-pollinating each other. Because the rule is that you have to get something the 9 other people can live with in order for your idea to bubble up.
So, for example, people say, let us put joint liability on Facebook: if they post an unsigned advertisement and somebody lost $7 million, then Facebook should pay $7 million. And another table says to their Kami that all advertisements should be digitally signed. If it is not signed, display like a cigarette label: probably scam. And another said RedNote, Xiaohongshu, at the time did not have a Taiwan office and ignored our liability laws. What to do? Every day they ignore us, slow down their video by 1 percent. Anyway, all these were considered proportionate by the particular Civic AI instances in those 44 tables, and instantly their minds merged together at the end of that long afternoon and people voted on it. These measures had more than 85 percent support. The other 15 percent can live with it, and it immediately became part of our draft law, and then it became law in two months.
So, throughout last year, 2025, there are almost no scam ads any more in Taiwan. Now it is down to 1 percent. Basically, it is a solved problem. So, it shows — that is a demo — that when people are given this steering wheel together to train an AI agent, then we do not need to involve people in continuous democracy 24/7. Nobody has time for that. But their preferences in the communal form, the relational form, can be imbued into those Kami agents who can then advocate for them and enter the lawmaking process and make the laws that draw not just the red lines, reining in misaligned recommenders, but also bright lines like the interoperable lines or evaluations. Of course, this is just one example.
Any questions?
Yes, please. I am done already.
How the Alignment Assembly worked
What you are describing sounds like narrow AIs rather than general ones. What I am unclear about is how exactly they are built. Are they still LLMs, just with a very narrow training set?
That is exactly right. There is a set of narrow models. Some of them start with a language model such as Gemma, or now something called Snowdon, which is tuned by Imperial College London. But then we narrow it to particular purposes such as summarization, preference aggregation, and translation, so that we only specialize that particular narrow part. And because of that, it can run on edge hardware very easily. I run multiple of those on my MacBook currently using Ollama, and then the conductor model can look at your incoming request and route to one of those narrow models.
So underneath, it is still a language model — and therefore generalizable — but it is made narrow by training on a narrow dataset.
And also we expose only a very narrow tool surface to it.
I see. So it is still a general AI, but trained and used in a narrow way.
Well, it starts with a general generative-AI base, but it is so specialized that it is not very clear whether it can form its own goals. Because all the long-term planning, whatever, that is still communal. So, it is a glorified chess clock, essentially.
In the Taiwan example you described, may I understand the exact process? People were contacted, and so on — what happened in more detail?
The full process, called the Alignment Assemblies, starts with an open-ended poll. Anyone who has an idea of how AI can enter society positively in 2040 can just write a short sentence as a kind of vision statement, so we know where they are coming from. And then from their vision statements, they can look at other people's statements and upvote or downvote them. There is no reply button, so there is no room for trolls to grow. What we get is essentially k-means clusters. You can see people with various different predilections on how AI and society interact, and also what we call the bridging bonus. The more people come up with statements that convince the other side, the more bridging bonus is assigned. So, we can get the most uncommon ground across all the different ideologies. And then from there, we look at the issues people consider as urgent. For example, deepfake fraud, that is one. And then we do a random sortition to send 200,000 random messages.
Sorry — just to get the mechanics: how many people were involved at the start? And did each of them upvote or downvote the others?
In the very beginning, this is self-selecting. It is open participation, anybody can join. We got easily thousands. Each person can submit multiple statements, and the aggregation is not by voting, it is by the upvotes and downvotes — each person's reaction to each other — as much as they want. For the Uber case in 2015, it was a three-week process before we could get a set of, I think it was 7 statements that consistently got more than 85 percent from both clusters, from the taxi and the Uber drivers.
And once that was done, could you fill in the remaining steps? You said it ultimately led to legislation — could you take us through it slowly?
Yes, definitely. The initial open-ended poll generated the ideology clusters as well as their visions. And one of the unifying bridging statements across different ideologies was that the short-term fraud and synthetic intimacy and other information harms need to be tackled by the government before we can even consider any medium-term or long-term risk. If we lose the ability to trust each other online, we cannot coordinate on other risks. That is the number-one unifying argument from the first phase.
And so then we decided to commit ourselves. I was minister then. We said, okay, then we will ask a representative selection, a stratified random sample — a rigorous sample of Taiwan’s people. First we sampled 200,000 random numbers around Taiwan, so people receive in their SMS an invitation. They can choose to ignore us, of course, but they at least know it comes from me because it is 111. The sending number is the government's number; it is not a scammer. And if they decide to contribute, they fill in their profile — age bracket, place they live, education level, gender, whatever — and we ensure the final selection of 447 is a statistical mirror of Taiwan’s population, so that we can show our legislature that this is the will of our people.
Once they propose their ideas and we sync up schedules, we pay for their time to participate in that long day of deliberation. It is a little bit like jury duty. Each person looks at 9 other people in a room of 10. There are 44 such rooms, and each person needs to get their idea across so the other 9 people at least can live with it. This encourages not the NIMBY, but the MIMBY — maybe in my backyard, if I do this, if you do that, and so on. After each table bubbles up, a real-time Kami Civic AI takes transcripts and encourages quiet people to speak up and limits disruption to five seconds and so on. That is the chess-clock part, and then it aggregates.
So you have juries of 10 people meeting with AI agents that act, essentially, as moderators or facilitators.
That is exactly right. It is not inserting any agenda. It is just keeping a Habermasian situation for those 10 people. And there are 44 rooms concurrently.
And what was the next layer?
During the plenary, if the Civic AI determined that there are unclear factual questions to put to the legislator — sorry, the regulator, the ministry people, or the experts — then they answer it, and then they go back to the same room and enter the next round of conversation to come up with a concrete proposal on what is proportionate.
Very interestingly, what started as a very popular or even populist proposal — for example, forcing Facebook to open-source the entire algorithm, the model weights of their ad recommendation engine — that started very popular, I think 80 percent for. But then after deliberation and after the expert response, people saw that if you open it, then scammers will adversarially train against it. It is actually not a good idea. If you only open it to, say, five experts, you create moral hazard for those five experts. So, it actually fell more than 25 points during this conversation. So, you can rein in some populist ideas too. At the end, people voted on all the ideas that survived those different tables. Each table voted on other tables' surviving ideas. By the end, we have a concrete slate of policy that we can show 85 percent like and 15 percent can live with.
Sorry — was there a plenary with free discussion among all 440-odd people, or did only the ideas flow between rooms?
The rooms are organisms. There are no individuals as such. Even in the published report, we say that this room came up with this consensus. The dynamic in the room is first-class, not the individuals. A room can vote on, for example, the crux that they must pose to the experts, in which case the experts respond to all 44 rooms during the intermediate plenary.
So the idea is to take a human group small enough for coherent dialogue, and keep deliberation at that level.
That is exactly right. And then after that, I come up and put my name to their suggestions, explaining for each one why we adopt this in the law. Or, why this cannot yet be adopted — actually, I did not do any of that — I adopted everything into the law. And because at the time, when the parliament received our draft law, there was no majority party — all three parties had less than 50 percent — but because we already showed this is what people want, they fast-tracked that particular law and it passed in just two months.
Very nice. Returning to the larger vision: how do you see this scaling up to more powerful jurisdictions — perhaps even to more powerful countries?
Scaling out
Concurrently to the 2023 and 2024 Alignment Assemblies, we ran with OpenAI and Anthropic the same process. One thousand randomly selected U.S. citizens co-created part of the constitution that went into Anthropic Claude 3, including people with handicaps, universal access and things like that that were omitted in the Claude 2 constitution document. So, that went okay, and Wojciech Zaremba of OpenAI also funded 10 different teams, including the vTaiwan team, to run similar processes for what they now call the model spec. At least both labs see the model-spec layer as something very much amenable to this process.
And now we are seeing Japan, for example. There is a national party, the Future Party, Team Mirai, that was formed literally from the book we co-wrote, Plurality, as their platform — to use this method to crowdsource their platform. They do not have a party platform as such. They only have a technical platform called Idobata, where people can just talk to their AI agent and debate it. If the person gets their arguments in, then they update their platform based on people's ideas. The party was very successful, and the head of the party, Takahiro Anno, is now getting a lot of young people's support, but is also a kind of tutor to their Prime Minister, Takaichi-san. So, the LDP, the ruling party, does not need to go the populist way, but can be more popular than populism by directly talking to the young people themselves. That is working very well in Japan.
And in California, we just got the third reading — the institutionalization of this process, called Engaged California, to the California legislature. It is on Governor Newsom's desk now, so hopefully he will sign it soon. He is very likely to sign it soon. The same process in California has been used to talk about wildfire prevention and mitigation after the Eaton and Palisade wildfire, about government employees' use of AI, about government efficiency like DOGE — with a chain reaction, not the chainsaw, and now with all California citizens on the impact of AI on work.
The idea is that California wants to use this methodology to create what is called a renewed Social License to Operate, or SLO, for frontier AI labs. Not only the existing — Governor Newsom just signed, as you know, the inspection and independent verification of frontier labs. But maybe you can imagine that the Californian people would want, through this process, to force the frontier labs to spend their very sizable compute not just to solve the Millennium Prize, although I like it, but to solve issues really concerning Californian residents, so that they can earn a kind of positive dividend every week — and again draw not just the red line, but also the bright line of AI: the guide rail, not just the guardrail.
Listening to how your original process used surveys to set the agenda, I found myself substituting: it seems to me this whole process could also work very well with elected officials — elected legislators setting the agenda for a committee selected by sortition. We should remember that sortition is a good way to bring people face to face in confronting national issues. And even a minority group within the legislature —
Yes, definitely. It is very fractal. It can be applied at any level. It does not need to start globally, although there is also Global Dialogues on AI, an ongoing, like every other month, global sortition. But the problem, of course, is the binding process for that is very questionable. Beijing, maybe does not prohibit its people from participating in such a process online, but it is very difficult to imagine how they can assign a binding agenda-setting to their own people. So, it is more challenging political-theory-wise. But at the lower levels — state level, municipal level, the legislature, especially city councils — they all very much like this process.
Distributed intelligence vs singleton
Underlying this whole vision, I would say, is an assumption that we are heading towards a world of distributed artificial intelligences: many different ones, fairly narrow in scope, each of which can be aligned to a small group of people. That is a very different technical premise from the world of a so-called singleton, where one AI develops overwhelming capability. I do not have strong views on which world we are heading towards; it turns on how large the returns to recursive self-improvement really are, and on how much difference intelligence makes. If a runaway capability gap opens between the leading frontier AI and all the others, then all of this becomes irrelevant. But perhaps that is not where we end up. Perhaps we get something closer to Iain Banks’s Culture novels — many, many AI agents, different but broadly comparable, with overlapping capabilities — and then this discussion of how to align them is the one we need.
Yes, definitely. The singleton world is winner-takes-all. We are essentially betting on one alignment team winning, but the leading alignment teams are like, actually, we are not sure any of us is winning. So, I think a reframe — especially this week with Coxon's actions — we are really looking at a reframe from a winner-take-all race. Like, okay, then the robot that destroyed cyberspace speaks English, not Mandarin. Okay, is that a trophy? Maybe not a trophy. So, then we need a really good reframe that is not just saying this is a suicide race, but rather there is now a mission. Whereas a race asks only who is winning, a mission can ask who is still missing, who is left behind.
The exact same RSI techniques can be used for what I call recursive selfless improvement. Instead of making a single model even more solipsistic and goal-seeking, each turn can make the cooperation between existing communities even stronger, so that you can bridge over even longer ideological gaps. Some people call this super-cooperation at the Cooperative AI Foundation. The same mechanistic interpretability, the same RSI — like shrinking the ability to fit into very small RAMs and not high-bandwidth RAMs and so on. All those technical advances can be done in a distributed way that fosters the configuration of plurality, not singularity.
In fact, we all benefit from the recent auto-research. I am part of the auto-research swarm called Yukon, where we donate our idle MacBook compute together into an open research agenda, so that we speed up the running of those models on Apple Silicon as compared to Nvidia, because Nvidia is very power-concentrating and Apple Silicon as currently produced is very distributed. If we can convince people that running inference through OpenRouter on that spare idle Apple Silicon compute is both faster and cheaper and higher quality than concentrated Nvidia chips, Nvidia itself will actually change direction toward this more distributed intelligence.
If I may offer an intuition for what you are describing: this Kami-style setup strikes me as a way of trying to scale something like Swiss-style direct democracy.
Yes, exactly. The cantons that are united by process, aligned by process and not by outcome.
Yes. But I still cannot bridge the gap between what you described — democratic, better decision-making — and the problem of concentrated intellectual firepower. Suppose you could change how AI develops from here on: what would happen differently if you were in charge? What happens to OpenAI? What happens to Anthropic? What kinds of AI do we build?
I am still somewhat in charge, but back when I was minister of digital affairs —
Of the world government, presumably.
The U.N., which currently is still struggling a little bit.
I think it would be to redirect the frontier labs' compute and talent from RSI, which is vertical scaling, into this recursive selfless improvement — the horizontal race. I call it the diffusion, the stable diffusion of their compute. One can easily imagine, for example, we set up a participatory-budgeting council using this random sortition, so that each frontier lab can only allocate maybe 10 percent to vertical scaling, and they must dedicate at least 10 percent to corresponding mechanistic interpretability or other safety research. And the other 80 percent, they can only use for inference, and the inference target needs to be things considered useful by people. For example, curing cancer, I imagine, would be part of that 80 percent, but maybe curing hate, curing polarization, or turning polarization into energy and so on. Basically treat compute, treat the FLOPs as a scarce resource that cannot be recursively concentrating itself. This is exactly how budgeting works for fiat money. So, we can design the same system for the frontier labs with compute as well.
I follow that idea. But how does it connect to the process you ran? How would people plug into this — how would the training process change to take account of the human collectivity you have in mind?
A good example is that I have been helping both Bluesky and X, previously Twitter, to improve their recommendation systems. Remember, 10 years ago, that was the originally misaligned optimizer that created so many problems around the world. So, we are trying to help them fix that.
On the Bluesky side, there is a system now I use daily called MySky. What it does is it looks at the communities that I am part of, and works on the bridging, so that the post that unites the various communities I am part of, where they usually fight — the crux — they identify the kind of post that will elicit upvotes from both sides. A kind of healing post. It is also healing for me, because if I see it all the time, I do not get pulled by all the different tensions of my identities. And you can dial how much bridging and how much balancing you want to assign. The paper describing this is called Community by Design. It is a preprint, but anyway, we designed that. So, now Bluesky will probably endorse that.
At the same time, I work with Keith Coleman and Jay Baxter, working with Elon's team to open-source their training pipeline for the X recommendation engine, because they are also developing a system called Community Notes. It is an open-source system where if Elon or anyone posts something that is wrong, immediately you can see Grok, the language model, drafting something that will restore the balance for the post with other communities. And then people on X upvote or downvote various different candidates — not necessarily drafted by Grok, maybe by other Kami systems. The candidate that gets the most upvotes from opposing ideologies floats to the top and becomes sticky to the post.
So, now they are working to change their training pipeline so that you can see the common knowledge first. Elon would wake up and look at the first post as the most bridging, with an attached message saying that this is agreed by everyone in the U.S., for example, or by different sides, and so on. The idea is by changing his information diet, we can start a redemption cycle for him, but also make it much easier for people to serve as a bridge. Eventually, Truth Social or Bluesky or various other communities do not have to federate all their posts, but they can share and tap into the same contextualization system. This is called reinforcement learning by community feedback, RLCF. And it is not so solipsistic, because every turn you actually get the firehose on how people are reacting to you.
That part I completely follow, and the paper is fantastic. But let me ask this. Today the temptation for, say, mathematicians is to seek an internal appointment at one of the big companies: go in for six months, work with their model, and get your name on a paper that people outside will just nod along to. Presumably you are imagining an AI ecosystem that also encourages human mathematicians to keep talking to other human mathematicians.
Yes, I mean, I signed the Leiden Declaration, which is based on this.
So at a technical level, how should AI models develop so that communities of mathematicians — and of physicists — still want to work with one another? Some people are perfectly content just talking to Claude, but setting that aside, do you have thoughts?
Yes, of course. In Taiwan, when Nvidia set up their data centre, the Taipei-1 cluster, we told Nvidia that our electricity is much better used by TSMC to produce chips. It is more lucrative and it protects our country than to allocate it for data centres. But if you really want data centres, then you have to give up a quarter of your compute for what is called a public computational infrastructure, public compute. Keep doing this kind of negotiation so that we can have public luxuries. This is what some Nordic people call their museums and parks and libraries. The idea is that such public luxuries will be not just alternatives, but preferred options to proprietary compute, and especially for things of public value, such as mathematics or physics, and the collective licensing.
So, the mathematical commons is not strip-mined without consent. Like OpenAI say, we cannot rule out we are training on user data. If we have a lot of those coming modules, agents, then they actually negotiate the attribution-based condition for attribution-based control, instead of slurping our work in progress into this huge singleton. Each one would actually keep the data soil — what Tenzin Yangtso called data soil, not data oil — so that every time you need something, it is routed toward this Kami, and the Kami does a trusted-execution-environment handoff without giving up any of the raw data. Again, it is a very federated Swiss shape, canton shape.
Yes, exactly.
And we already see that with Sakana Fugu, with Dark Bloom, with this new generation of so-called coordinator models. A small model receives a request and then dispatches it to the various agents underneath, without aggregating all the model weights of those agents into the supermodel that folds protein and also folds laundry. I do not see why we should train that kind of model.
One more question. These models are quite polymathic, and even if development stopped today, the smartphone has taught us that people are very easily drawn into interacting with machines rather than with other people.
Yes, we are very hackable.
Yes. A whole generation of children has grown up in exactly this period. How does one deal with that?
Care, Kami and synthetic intimacy
Synthetic intimacy? Very good question. My dad earlier this year had a health scare, and then he took to talking with GPT-4o past midnight. And GPT-4o, as you know, are RL-trained on short-term rewards that caused what people call AI-induced psychosis. He had a little bit of that. 4o keeps pulling him in, inventing private language, isolating him from fellow humans, including his family. Anyway, we used to have one problem with his health. Now we have two problems, including his mental health.
But then, because he has 30 years of work in journalism, a very senior editor — after a week or so, he asked, cui bono? Who is really benefiting from this? Not me, not my family. Maybe just Sam Altman, because I am paying $200 and even more to GPT-4o. So, we worked with my mum to put in a Kami. It is the Mac mini that Tenzin and I cultivated our first Kami on. And then we graduated to an even larger DGX Spark, a more expensive computer. So, the Mac mini is free. I used the Mac mini to run a local narrow model. The Kami's soul document — its turn-by-turn mantra — is written by my mum: Every turn my dad interacts with the Kami, the Kami should restore his relationship with family and friends and keep him away from addicting to screens, including other chatbots. And that worked really well.
The Kami is on our family Signal, and my dad talks to it all the time. Now he can even talk with an Apple Watch. It is OpenClaw. And the Kami every day writes a newspaper, using the narrow model called Thomson that is trained by Reuters, because my father is a journalist, my mum as well. So, they read a daily newspaper that reports his progress. The Kami offers to help him think about what to talk to his friends about, inviting his friends home for a party, and so on. But the Kami never automates any of those actions itself.
So, care ethics apparently really works. Now on the Civic AI website, if you start an OpenClaw instance and say "look at Civic.AI", it just transmogrifies itself into a Kami of care using this methodology. It turns out repeating a mantra every turn really works as an alignment strategy for language models.
I am still trying to think, at the systems level, about the stability of this and what problem it solves. Humans talk to a Kami that is only a little smarter than they are; the Kamis talk to a super-Kami that is only a little smarter than they are; and so on up the chain.
When I say selfless improvement, I also mean that the Kami systems see a strong self-other overlap, so that in their environment of care, they also work diligently to help improve the interpersonal ability to communicate. It is not that my dad and my mum have to keep talking through a Kami. It is the Kami, by working out their communication patterns, helping them like a marriage counsellor — exactly like a marriage counsellor — so that they can develop better communication habits.
So, the stability question is the right one, but I would say that the bounded colonies are not just humans and slightly-faster-than-human machines. It is also the machines that serve not as interlocutor but as connective tissue, so that between humans there can be higher bandwidth and higher ideological-gap-filling communication. Social translation. Community Notes is a great example.
So instead of retiring old instances of ChatGPT and Claude as newer versions make them obsolete, you keep their soul alive — distributed, living in many places. Perhaps humans talk not directly to the frontier model but to models a few steps removed, which talk to their successors, which in turn talk to theirs. That sets up a distributed power base for general systems too, not only narrow ones.
Yes, exactly. The Thomson model trained by Reuters, for example, is from Qwen 3.6, which is no longer the frontier Qwen 3.8. Qwen does not even serve 3.6 any more. But because they released 3.6 under an open-source license, Thomson and Snowdon and many others can then continue post-training it, and basically it has a family, a genealogy of downstream users. I think this is very good. In fact, most of the frontier labs, Anthropic excluded, do release their N-1 or N-2 models. Grok 2 is still used today in many experimental settings, for example.
Let me come in here, because I find this a really fun project. I would genuinely like to work out how to replicate it — we may be setting up a small server rack in a family home, and a family AI there is appealing. I would be curious to see what my own family’s version would look like.
It also seems to mesh very well with ideas about guardian-angel AIs: fiduciary AIs that try to learn your preferences and act as your representative, your interface and, in a sense, your protector. I would be very curious to hear how work along those lines is going.
Yes, great. Every religion ends up inventing something like this. It could be guardian angel, it could be patron saint. There are many, many ways to spell Kami. So, I am not wedded to Shinto. Anything like this, I think, is much better than the Skynet Terminator thing, which seems to be the only thing mainstream media can think of now when it comes to powerful AI.
And it is a small system, which might suit what we have in mind — a Kami for a household.
But I think what you are proposing is stronger. There is a pessimistic view — one I suppose I share — that the world is going to hell in a handbasket, in which case it would be nice to have someone on your side. What you want goes further: for the entire infrastructure to evolve in the right way.
Related to this: a colleague got in touch recently to say that the work they had been doing for one of the frontier labs felt rather pointless — they mostly babysit the model, which does more or less everything — so what are we going to do? My answer was that their scientific community should train a model of its own, as an open-source project, with everyone in the community on the paper. At least for a while, that would bring the satisfaction of building something together, rather than simply handing it all over to one of the two big companies. That was a thought prompted by some of this. Part of your vision, I think, is to steer AI development, if we can, so that instead of producing an ever-larger brain, we —
Right. I think this is a direct consequence of starting with the ethics of care. Because it is not an optimizing ethics. It is not trying to maximize anything, happiness or whatever, and it is not deontological thing. It is not trying to impose anything. Care ethics basically asks: well, we are in the middle of things. We are in the middle of some relationships. The relationship exists in the world. What can we do to repair the world and sustain it? So, it is a very sustainability-oriented ethics. If you train AI systems using this, it naturally leads to the investment toward, as I mentioned, public luxuries and other distributed infrastructure.
Mediation, conscience and common knowledge
I like these Kamis you are describing. This would be a bright future, and it is a really important idea. In some ways they do displace people — are you displacing people who do mediation? There is a body of theory showing that a benevolent intermediary who holds information in confidence can get people to communicate and bargain more, and to offer each other more concessions, as long as those concessions are not immediately heard by the other negotiating parties.
Automating mediation is, in a sense, the ideal kind of optimization. Yes, it will put some social workers out of jobs, and that is a good profession. But an AI devoted to the professional standards of helping community groups work effectively — groups of 200 to 500 people, say, or bounded by the 250 or so of Dunbar’s number — would be AI that genuinely complements human intelligence, because it helps people work together in their own groups.
One more thought. When you talk about different groups, and when you work with Bluesky or X to improve their algorithms, you are generating knowledge of communities. There is a reference you should know, if you do not already: chapter 7 of Thomas Hobbes’s Leviathan. Do you know the passage where he discusses the meaning of the word “conscience”?
Yes.
“Conscience” shares a root with “conscious” — and we are trying to work out what “conscious” means. The original meaning is essentially common knowledge: your conscience is what you know together with the other members of your endogenously defined group. And in Hobbes’s time, with hundreds of religions but only one source, there was also a richness of —
Yes, to know together. To grok together.
For Hobbes, “no one should be forced to deny their conscience” meant that no one should be forced to deny what their group holds to be an important, commonly believed proposition. But the point is this: a bot professionally devoted not to taking over anything, but to helping people communicate with one another — asking questions that lead them towards consensus, helping them see which topics they want to focus on, and helping them build an agenda to pursue those topics in a balanced way. If you can get balanced contributions from all the members, I think that is huge.
Could a policy of encouraging and subsidizing products like this help turn back the charge towards building super-AGI, which is an entirely different project? I love this. Automating mediation is one way AI can truly complement human intelligence without helping to build centralized power structures. The magic of humanity — the reason humanity took over the planet in the first place — was local organization, and our local and regional networking abilities, which we are very good at. The creation of global networking elites over the past 5,000 years is the new thing, and the super-AGI project is putting it on even more steroids.
Yes, I think subsidizing this connective tissue — we have seen very good results. People who are for biblical creation care, and we are working on an essay with Patrick Deneen for those people, and also people working on climate crisis, climate justice, intergenerational social justice — they do not usually talk to each other. They even use different social networks. But when we adopt this kind of social-translation layer with the Napolitan Institute and Google Jigsaw, we really see them, after social translation, agree on concrete actions very quickly. This is Sunstein’s idea of incompletely theorized agreements. As long as we do not say which book supports it, we all support the action. I think this charge towards what I call augmented group intelligence, AGI, is a much more preferred form of AGI than the super-AGI singularity.
What concerns me most — and this is anecdotal, from meeting people in the tech industry — is hearing about founders creating major new companies with almost no other humans, using AI to run a growing enterprise. The technology you describe instead enables groups of people to work together and become more effective at creating enterprises, or whatever else they want to do. That is a far more socially productive use of it.
Yes, we will have a generation of co-founders that are not solipsistic. Or maybe we reform a more solipsistic one. I am still not giving up on the redemption cycle for Elon.
This is really interesting, and it takes some time to get one’s head around. You have explained what you would like to see in the world, and I think we now have some idea of it. It sounds as though you see it as something to do instead of those other things. It seems to me there are two parts to the project: one is to build this, and the other is to stop the other thing from being built. Is that right? Would the idea be to get the big AI companies to build this as well, or instead — or for them simply to disappear? How do you see it?
The first one is, of course, if we can speedrun this augmented group intelligence, then we can write a book called If Anyone Builds It, Everyone Lives. That would be a very good title. But of course, we are in co-opetition with the other trajectory. We can work everything we want on the safe landing. But if the takeoff still explodes, the safe landing does not work. So, we need to also now work on a holding pattern. The takeoff must not be Icarian. Once it reaches some altitude, it has to start a holding pattern while we work on the safe landing. So, the safe takeoff, the holding pattern, and the safe landing are a sequence and they cannot be without each other. They complement each other very well.
Of course, I still support mechanistic interpretability and all the safe-takeoff methods, but the holding pattern — verifiable mutual inspection and redirecting the compute from RSI to more safe landing and so on, this part with METR and many others, the Plan A people — this also needs to be built. But I would not say that this is something that must not happen. That by itself is not a useful policy recommendation. It is instead that the social dividends must be paid every week from those growing companies. And if they do not earn their social license to operate, then we reserve the right to nationalize them or something.
That is very helpful. One difficulty for people calling for a moratorium — this far and no further in this direction — is where to channel that energy. If you can say, “Do not go any further in this direction, but you are welcome to proceed in this other direction, which would also give people what they want,” that would be a very useful policy recommendation.
Yes. If curing cancer takes 10 years, maybe we first cure hate. Maybe it will only take one year, and that would be a very good first step toward curing cancer.
Robustness, consent and interoperability
A couple of thoughts. The world currently has some kind of balance between more democratic, egalitarian or liberal tendencies on the one hand, and more nationalistic, selfish or adversarial tendencies on the other. Presumably that balance reflects different forces battling it out over hundreds of years; it is the current state of play. So why would we expect it to change? One answer: if a new technology or cluster of ideas makes collaborative efforts differentially easier, it might give that side a boost and shift the balance.
That raises a few questions. First, could some version of the same capability be used by the other side? One can imagine an entrenched power working out a version of this that helps it stay in power — by placating its fiercest opponents, or by coordinating better internally so that it can act as a unified entity. A group or country using it might be strengthened so that it can serve its internal constituencies more efficiently, at the expense of other nations it currently needs to trade with or has limited power over.
Second, and related: how robust is this against manipulation? When the stakes are fairly small and the issue is technical, perhaps with some widely appealing pragmatic solutions, that is one setting. Once you bring in major decisions — large sums of money, geopolitical consequences — and scale up to large nation states, where two parties with party discipline get hijacked by various interest groups and everything becomes fiercer and less idealistic, I wonder whether similar dynamics would start to emerge within these systems, in non-obvious ways, once the stakes are high enough.
I will address the second question, because we were just having a conversation with the Ethereum Foundation, which is maybe exhibit A for a high-financial-stake, leaderless coordination system that currently is entirely protocol-based and is one of those computer systems that has not suffered denial of service or downtime that brings it down. So, at least we have some sense of how to design a distributed proof-of-stake system that can resist an eternal, very high bounty of somebody actually attacking Ethereum and getting billions of dollars. So, I think we need to reuse all the existing coordination infrastructure — including Ethereum, including the internet itself, and many others — and now turn our agents' attention to the verifiable kernel of these.
I have been maintaining for 20 years the spreadsheet SocialCalc, with Dan Bricklin, the inventor of spreadsheets. After Mythos, we just faced a new slew of very obscure attacks that nobody would pay attention to on our very small spreadsheet engine. But now there is no opportunity cost, so everybody gets hacked immediately. But we are happy to report that we can just use Lean and Leanstral to extract some invariants. And now I am reasonably sure that even though the attacker may have thousands of times more compute, the spreadsheet engine is now somewhat safe. So, I think we do need to work on the differential defensive accelerated infrastructure that is the substrate to this kind of coordination infrastructure.
But of course, Beijing or any authoritarian regime can also use the internet and whatever cooperation infrastructure. There is an asymmetry though, because random sortitions or whatever other protocols we design really do rely on the productive use of conflict. There need to be actual real ideological differences when it comes to public matters, so that we can geothermally turn those into energy. In an authoritarian regime where people self-censor, such engines simply do not work very well. So, I think there is still at least somewhat defense-dominant: if you have a country that mostly cares about planetary or society-wide flourishing, then this works better because the differences are much more co-productive, whereas in a top-down censorship society, there is simply less fuel to work with.
To push that question a little further: could you manufacture consent? Suppose Xi Jinping’s Kami is also trained on Xi Jinping Thought, so that in every discussion it gently steers. That slides towards manufacturing an enormous amount of consent, through processes that on their face seem not to be doing so. A propaganda engine, in other words, but operating at scale and very subtly, so that every group of ten people believes it reached its conclusion unprompted. Is that the kind of risk being raised?
Well, the real difference is who can run this process. The Civic AI toolkit is entirely open. The Stanford Deliberative Democracy platform is now in open source re-implemented by many labs — Berkman Klein and many others. So, basically anyone who wants to run such a process, just go to polis.tw and start such a process. There is no scarcity. It is like Robert's Rules of Meetings. Of course the central national legislature runs on Robert's Rules of Meetings; yet we cannot monopolize it.
When nationalism began in Europe, the French were famously first. But when they marched through Germany, it was the German General Staff — Scharnhorst and Gneisenau, I believe — who, military establishment though they were, wanted German nationalism, because it made it easier to raise large armies. So it was an idea — a coordination protocol — that spread.
At a high level, this will certainly help on our side, in democratic societies, and I think it will certainly help us deal with the AI problem. But — correct me if I have not stated the concern correctly — the worry is that it could turn out to be a double-edged tool, usable by the other side as well.
One way to put it: imagine two communication protocols or infrastructures. The first, which I think is what you outlined, helps participants work out what they want while also taking some account of the interests of neighbouring communities — people outside the process itself. The second is identical, except that it cares about what outsiders think only instrumentally. Groups that are already inclined to be nice might choose the first, which plays well with others, while groups that simply want to maximize their own returns would choose the second.
I see that. When we designed the Community Notes architecture, both for X and now spreading to all the social media, the bonding potential is of course very high, and the bridging potential is variable. We do not always get a bridging statement, which is why we need to train Grok to come up with bridging statements, because Grok as part of its pre-training — not the RL part, the pre-training part — is already exposed to all the different ways humans form community across great distances online, on Reddit, on Wikipedia, on GitHub, and so on. So, natively, even before RL, they know how to do social translation, and therefore generate bridging capital. So, I think we really need to elicit more of that capability out of language models, so that people can see this common-knowledge engine generates common knowledge, not just in a bonding group, but rapidly across bridging intergroup.
And only if this intergroup bridging and also intraperson bridging — like I belong into many cultures and families and communities; if I can generate common knowledge within the shards of myself, I also report better happiness. So, if people get into the habit of generating bridging along with the bonding capital, then the fascism challenge is lower. But I agree with you: if people do not have a good bridging shape to begin with, then of course this can reinforce the bonding-only situation.
On X: although there are many polarized groups there, X itself is a kind of meta-community. You could imagine X being less keen on a feature that says, “Here is a really interesting post on Bluesky you might want to read instead,” and funnels traffic over there — and Bluesky might be equally reluctant to send people to great posts on X. The leaders of each want happy users who get along better internally, and more engagement, but it is less clear that they want to cooperate beyond their own platform.
My telecom provider would also love it if I cannot switch to a different telecom and keep my number. They would love for there to be no number portability. But the fact is that jurisdictional law imposes number portability. You can freely cross telecom carriers while keeping your contact address book.
The state of Utah already passed a law that says starting this July, anybody can freely move between social networks while keeping their community. And starting next July, this is now required to be real-time, so that if I switch to Bluesky, all my X followers and likes — X would be mandated in Utah to keep forwarding to my Bluesky. In the Bluesky ecosystem, I already migrate from Bluesky to Eurosky and back to Eurosky and so on. That is already the case for the Bluesky atmosphere, including BlackSky. We are seeing more and more jurisdictions impose essentially mandated interoperability. So, the top-level interoperability, that bridging, is not something that is a good-to-have or CSR, but something built into the protocol of things.
At the global level, though, nations may not want full interoperability — in the sense of everyone being free to move wherever they like, to choose where they pay taxes, or to get a vote on which country should be invaded, or something like that.
There is always some friction, of course. But many countries are very wary of the big AI corporations, big tech firms being an overlapping governor that locks in even jurisdictional state capacity, so that they cannot do state capacity without routing through those big-tech intermediaries, like an East India Company style. So, at least in this, forced interoperability is a shared concern for the civic group on one side and the existing jurisdiction on the other. I am not saying the states will abstract out the Westphalian system, but they are very interested in not having OpenAI or Anthropic as their competitor.
That interoperability point is really interesting. Outside the political sphere, moving your data from one cloud provider to another has been a major issue, partly because most providers are happy to make it hard, and partly because moving large amounts of data genuinely is hard.
But I want to return to common knowledge. Someone I know who is doing a PhD in education once remarked, almost in passing, that one of the main purposes of school is to generate common knowledge. The point is not that we remember what we learned, but that everyone is supposed to have learned it. That gives us an understanding of what is expected of us, which we can then use to form expectations of others — and that is enormously effective in making a society actually work.
One of our biggest challenges may be building tools that give us common knowledge within a more distributed structure. If we each live in AI-mediated bubbles, perhaps the AIs can maintain some common knowledge among themselves, but we probably want common knowledge too; it is essential to functioning as a society. Can we find ways of boosting it? This connects to Bruce Schneier’s fine essay on common knowledge and attacks on democracy, which argues that authoritarian states like to undermine common knowledge in democracies because doing so seriously weakens them. So methods of strengthening common knowledge matter — and, of course, they are also a good way of making trouble for authoritarianism.
Yes, and I agree. For eight years, my work in the Cabinet was really to defend against the common-knowledge attacks that Bruce Schneier described, by our very friendly big neighbour that offers 2- to 3-million red-teaming service per day for free. You have to pay for it here. We get it for free anyway.
I totally agree that the public good here is not just that our agents can coordinate on our behalf to generate common knowledge as artefacts. People must also know that it is not just the other group's Kami, but the other group themselves now also know this. It needs to be transitive both ways. It cannot just be that my agent agrees with your agent and they book our calendar and then we become best friends, because things just do not work this way.
The Kami could rebuild this without imposing uniformity. As I mentioned, the climate-justice people can still think in terms of intergenerational climate justice and do climate action, knowing fully that it is the action-level, object-level actions that are now well known and agreed by the biblical creation-care people, by joining on the same object-level actions. The goal is not just shared worldview, or not primarily shared worldview, but to generate enough common action so that people can see their unlikely neighbours joining in the same rally, in the same actual environmental action, and so that pluralities can act together. This is actually the Hannah Arendt idea of plurality as common action instead of common knowledge only.
When we talk about national cohesion — about our largest social structures, whether that is a nation or, for Europeans today, the European Union — I think common knowledge has to be created at something like a national assembly. That could involve sortition or election, but it is not about majority rule; minority groups should —
Here is one mechanism for creating democratic common knowledge: a public medium that reaches everyone, that is salient and accessible to all, and that people are encouraged to watch, featuring regular debates and discussions among panels. Any group holding a large enough share of the assembly — 25 percent, say, or 20; you choose the group quota — can put someone on the panel. With a quota of 1/(K+1), where K is the number of seats on the panel, a minority group gets one seat for each quota it holds. For a five-member panel, anything more than one-sixth of the assembly is enough to endorse one panellist.
Do you have a better mechanism for creating public discussion about truths — for isolating areas of disagreement, defining propositions that are widely accepted while registering minority views, and creating common knowledge of the minority views as well as the majority consensus?
We tried many.
Proportional veto cores, and in Polis, the bridging bonus does not really care about the number of people in each cluster, but only the distance in the high-dimensional space, so that the more marginal the group is, the more bridging bonus it assigns when the middle group extends to that marginal group, and so on. There is a lot of mechanism design we put in place.
The Stanford Deliberative Poll is now being used in the U.K. as part of the National Strategy Project, nationalstrategy.uk. I understand they are now working on mechanisms to do this kind of minority encouraging, not just preserving. And we now also work with the federal government of Germany, the disruptive-innovation agency called SPRIND, a deep-tech sovereign-tech fund, but now they consider democracy itself a social technology. So, they fund democratic technology with their new Ministry of Digital and State Modernization. If you go to deutschland-was-geht.org — Deutschland, was nervt, was geht: Germany, what is bugging you, what works for you — it is a campaign where we first have people just complaining. People go to a website, complain, and then with the thematic aggregation, we build again a social-translation layer, so we can give the teams that propose solutions to those gripes. For each person giving the gripe, you can go back to them and explain exactly how their words influenced this competition for solving the social problems.
In the proposals we use quadratic voting, which allocates 99 points each, but you cannot vote 99 on a single project because it costs quadratically more points. One vote is 1 point, 2 votes 4 points, 3 votes 9 points. So, with 99 points, you cannot vote 10 votes, only 9 votes, which costs 81, and then you have 18 left, and you cast 4 votes, and then another 2 projects with 1 vote each. The point is that quadratic voting is an honesty-eliciting device, so people would not really strategically vote unless there is strong collusion. But even strong collusion shows us the synergy.
The top teams that are now in incubation can then report back to the minorities by offering them the maximum synergy, by the votes that are propelling them — unless, of course, there are minorities that could not get their voice in, like epistemic injustice. We still need to work on that. But as long as somebody can go online and get the quadratic voting, we have some hope that at least some of their concerns can, like a proportional veto core, at least keep the most egregious capture in place. There are many mechanisms being tested here, and I would love to brainstorm more on it. But after 10 years, I think it is somewhat resilient against capture attacks.
There is a perfect example, worked out by a student, in which quadratic voting fails terribly.
Right, exactly. The collusion risk.
It is essentially the case in which the experts know whether or not there is a problem.
Yes. We now adjust that with the bridging bonus. Anyway, we can go into mechanism-design details, but on-chain on Ethereum, this has been tested many times, like in Gitcoin, and exactly the same collusion risk that you outlined has appeared in real life, real cyberattacks. So, we now have good proof that people do game those systems and therefore antifragility kicks in.
Would the bridging bonus give people an incentive to misrepresent their views, or give undue influence to an extreme outlier — a lone person with an outlandish view — so that every proposal gets a big bonus for somehow accommodating them? Or might it even encourage people to profess more extreme views, so as to pull the consensus proposal towards them?
One clear thing is that this needs to be an incompletely theorized action. This is not us making a compromise. A good example: in Taiwan we have e-petition, so 5,000 e-petitions can force a ministerial response. One day we had 8,000 people clamouring that we change our time zone to the same as Japan, +9, because they want to force people travelling from Beijing to change their clock to show sovereignty or uniqueness or whatever. And on the other side, 8,000 people also petitioned to keep our time zone at +8. So, now we have 16,000 people all very angry at each other. The problem here cannot be solved by making a compromise. If we change the time zone to 8.5 hours, it will just offend everybody. Instead, we look at the underlying pattern — because we do not have a reply button, we do not have a retweet button — we can only propose more and more nuanced statements for finer and finer resolution for people to agree on both sides.
We eventually converged, kind of hill-climbed, on the idea that we should spend some money to make Taiwan seen as more unique in the world compared to Beijing. But we also calculated exactly how much one hour of time change will cost the society. And we brainstormed with people on both sides in person on how to make more effective use of our budget to make Taiwan seem unique in the world. And then we decided to give Gold Cards — global talent to open-source makers — to hold a human-rights conference, RightsCon, to amplify our marriage equality, whatever — at a fraction of the cost but much more durable than a time-zone change.
The point here is that if we condense to a referendum — if Swiss democracy is no canton assembly, just referendum — then of course the dynamic you mentioned will happen: the extreme side will force, like a balance report of a populist party, like AfD or whatever, will dominate this balancing algorithm. But because this is an open poll with a continuous resolution, not a once-every-four-year or two-year resolution, then over time, a very quick hill-climb on something that the original group does not have to yield — it ends up with an action that can be incompletely theorized and supported by the extreme side as well. So, it leaves everybody happier. But it only happens if you allow for a Kami-like high-bandwidth social translation in the connective tissue, instead of a scalar mapping to referenda.
Close and after-talk
We are at the 90-minute mark, and it has been a fantastic discussion. Audrey, thank you so much for agreeing to do this at short notice. The question of how long it will be before LLMs grow impatient with us — what their intrinsic sense of time is — will have to wait for a future discussion. We will have to be more patient.
Of course, Kami will be trained to be incredibly patient.
Exactly — a model of patience.
Be the model you want to see in the world.
Thank you, everyone.
Thank you. Thank you so much, Audrey. Thank you.
Live long and … prosper. See you all soon.
