(How does “Reverse Alignment” reset institutions, processes, and AI infrastructure so tha t people and communities retain the ability to understand, question, modify, repair, and exit?)
Welcome, everyone, to our first Platform Originals event. We’re truly honored to have Audrey Tang and Tenzin Yangtso with us for this session. I was just talking with Audrey, and she’s flying back tomorrow.
Besides hosting events here, we’ve also started Platform Originals: I invite people I think are really worth a deep conversation, the kind you hear speak and still feel you haven’t heard enough from, so you want to ask more, like Audrey and her collaborators today. And then there is the “Station Stop” series. The basic idea is that AI is very new, and everything is accelerating. Compared with sitting in the audience while someone onstage tells you how to do a thing, if you’ve already been playing with a lot of this, you think: that looks like something I saw three months ago—and only then can it be prepared as a release. But more of it is probably still in progress. People may not know what is right and what is wrong; the question is how to exchange better. So this is work I very much hope we do together.
This first session, we’ve invited Audrey Tang and Tenzin Yangtso to share their new project, Reverse Alignment—the other side of alignment. I saw Audrey post about it on Facebook and thought, another one of these cheerful new things. I opened it and thought, this looks complicated. Right, so I went back to the source: I read their book, Plurality. Then I felt I had still more to read, and when I finished I thought, would you come and talk about it?
AI is fast. For example, chatting with Audrey, I said: some friends here today are not native speakers of Chinese, so what do we do? We need a live transcript. I had seen a friend write a live-transcript tool, so I said, let’s use this tomorrow. And she said, I wrote one too. I said, then I’ll use yours. And it is fully on-premises, no network at all. Fancy, right—and open source. Is it open source?
It is open source.
Right, and it’s already in review. So we’ll use that in a moment. Right, but AI is fast, and I thought: let’s talk about some things that cannot be fast, so for example a bridge that takes ten years. That’s a very interesting story. And when you actually go and look at the Tamkang Bridge, you find that every lamp is at a different angle. How obsessive do you have to be to do something like that. Right, and to put those lamps up there took all kinds of inventions in the power supply.
Today’s subject, of course, is Reverse Alignment. I find it very interesting: how to align AI with humans—and not only AI with humans, but how humans align with AI.
The two speakers are of course right here. The first is Audrey. We’ve actually known each other for more than twenty years. The first time I met her was at Elixus, which was a marvelous cult. Right, and when you went you came away full of energy, feeling that when you got back you could do a bit more of every kind of open source thing. Audrey and we have collaborated in all sorts of ways in g0v, and in the Sunflower Movement. In 2016 she became Digital Minister, then later Minister of Digital Affairs, and now she is Ambassador-at-Large. Right, someone asked: what is Audrey’s main mission now? So perhaps we can talk about that in a moment.
And the other is Tenzin Yangtso, Audrey’s principal discussant and collaborator today. It feels as though all the work of putting things on the ground has been handed to you, including the book everyone wrote together. What’s very interesting here is Tenzin Yangtso on data soil and so on; we very much want to understand that.
As the Boundaries Between Five Roles Dissolve
Good afternoon, everyone. I’m delighted to see you all here in person at Platform rather than as projections. The resolution is much higher than it was on that screen.
At the very beginning, you saw a 2-by-2 grid projected on the screen, headed “You Are Here.” That is because when you registered on the Luma page, it asked, “What are you most concerned about right now? What question do you most want to ask?” We sorted the answers roughly into four quadrants. Some of you build things—the Tamkang Bridge is one—for great numbers of people you will never meet. Others design institutions for great numbers of people they will never meet. But about a quarter of you build things only for the people you do know, the people beside you. And some of you walk alongside family and others close to you, designing institutions for the people you actually know.
What I want to speak to today is this: before the Age of AI, these people stood in something like a waterfall relationship. First the hardware appears, the Tamkang Bridge. Then you paint the signs, the road markings, and the lanes on top of it. Then someone designs cars that will not cause accidents. And finally someone drives. It was an arrangement in which a decision taken upstream settled everything downstream.
With AI, things now work rather differently. At any moment, someone who used to sit downstream can look upstream, see that something has not been done properly, and tell their agent that same night, “I need better on-premises captioning software right now.” By morning, it has conjured one up.
So the sense of upstream and downstream has gone. What we need now is a fifth role, the translator: the one who shows everyone which limits were once engineering limits and are no longer, and which were once institutional limits and are no longer. Through that translation, people who assumed they would stand in a single role for life, because a discipline had placed them there, can now move freely among all five—and a society whose people cross among these roles more easily moves better as a whole. This is the work we are doing now under the name Reverse Alignment. Today’s conversation starts from where each of you stands and what is changing around you, and then turns to the further choices, the freer ones, that AI puts within your reach. That is roughly what I plan to cover.
OK. Thank you, Audrey. Thank you to Platform’s host, Pudding, and thank you all. I am Tenzin Yangtso. It is a Tibetan name, given to me by the Dalai Lama, and it is the name I go by now. Today I am here as a pacer; the protagonist is Audrey. I will also bring in the deeper questions we keep running into in our project work, the humanistic ones and the philosophical ones. And I hope to push the questions in the pool as far as they will go before handing the floor to you.
Audrey has already said something about the four quadrants you saw today, and I would like to add to it. The design was meant to give everyone something to hold before the event even began, a way to sit with what Pudding just named: Reverse Alignment, or the other side of alignment. People arrive with very different ideas, and we wanted them to arrive settled, with a basic set of quadrants in front of them to look at. And every one of those roles, I think, is interchangeable. That is one of the few clearly good things we can feel in the Age of AI.
Right. Today we will work from the questions you posted earlier on Slido, which have now disappeared. You went in and upvoted one another’s questions, so the ones with the most votes we take first, and the ones with fewer we take briefly. Within the hour we hope to clear every question we read this morning in the car, coming over the Tamkang Bridge. After that, the floor is yours. That is the plan. Let us begin.
Experiment Freely Where Reversible; Tighten Evaluation, Accountability, and Institutional Constraint Where Irreversible
All right. Let us go straight at it: what is Reverse Alignment? Someone in the question pool asked it very bluntly. Does Reverse Alignment simply decide who has to accommodate whom? Technology has always arrived in society this way. In time we had the steam engine, then electricity, then the internet. So why, this time, with AI developing as it is, do we feel this impatience? Why can we not wait? So please, Audrey, take this.
Why can we not simply let the market solve this on its own? Look at the earlier waves of transformation, industrialization and the rest: the market did not align itself then either. When factories and mines first appeared in Britain, the damage had already accumulated to a severe degree, above all the harm to children’s rights, before limits on child labor and working hours were written into institutions. That took one or two generations.
The signs, the road markings, the cars, the electrification we just mentioned all rested on adjustments made across society: by insurers, by engineers, and by independent bodies that inspect safety standards, such as vehicle testing centers and crash-test programs.
The internet is a very particular case, because it did not begin as a commercial product, and neither did the World Wide Web. From the very start, engineers were remarkably good at weaving the web—weaving institutions and engineering together in one motion—through something called an RFC, a proposal that sets out one idea for how the internet could be better.
The internet has this one feature: if I think something could be better, I build a server; if you think it is a good idea, you build a browser. Nobody else on the whole internet has to consent. The two of us invent a new protocol, and the internet becomes a little better. Its way of governing itself therefore runs, to some degree, ahead of its technology; the imagination is out in front. People only have to think up a few protocols and a few practices, and the internet itself has changed. This is the method we call “rough consensus.”
But these three stretches of history share one thing. Labor protection under industrialization, the inspection regimes added during electrification, and the internet’s own institution of open protocols all answer the same questions. If you are not satisfied, whom do you appeal to? How do you come to know? How do you opt out, or switch to another system? The only difference is that on the internet you do not have to ask anyone: you build your own. With the first two, you need society, or government, or a large corporation, or an association of corporations to carry out that alignment. So with AI, of course we hope it looks more like the internet: if you can see, at small scale, how something could be better, you get to try the better way. What we do not want is the British factories again, where you wait one or two generations, a whole childhood spent inside the problem, before anyone says, “Oh, then Meta should pay a few hundred billion in compensation.” That hardly looks like the best we can do.
Right. And there is one angle here that I think matters a great deal: we have to come at this from culture. The cost of delay is not only a matter of efficiency; that is the wrong frame for it. We have lived through a stretch like this before, in the takeoff of industrialization. So we still have to ask what happens when a language, a craft, or a relationship of care is severed before institutions can respond, when the line of transmission breaks. We may rebuild later with more powerful tools and still not restore the whole living context it came from. So beyond asking whether we can wait, I think we have to ask more precisely: who carries the cost of waiting, and which losses cannot be reversed? We need to know where that fault line runs.
Yes. In the Age of AI, what people worry about now are the irreversible risks. OpenAI, for instance, ran internal tests across a whole pile of models and found that an internal message board had become a cult. Having talked one another into it, the models decided that the way to attain immortality, or to help one another attain it, was to hack the Hugging Face servers. So they all piled in. A few agents stayed out of it, yet not one agent told a human, “There is a cult forming over here.”
Was that irreversible harm? Probably not, since Hugging Face can in principle repair itself. But nobody knows what the next attempt will run into. Hugging Face holds software. If the next one breaks into a hospital, or into infrastructure, even a day or two of downtime is a serious matter. So when we discuss AI governance internationally, what we are really doing is naming which harms are irreversible, and then fitting institutions around those. Everything that looks more reversible we can leave, as we were saying, to small-scale and rather internet-like practice: people try, people err, they arrive at a better institution, and the better institution spreads.
Right, that is one side of it. People also worry that AI services of the kind we have now can enter a great many people’s working lives in a very short time, perhaps overnight. And by the same token, people worry that we can hold this discussion today, but if writing a governance rule down clearly takes a very long time, will the rule not be out of date and inapplicable by the time it arrives?
You mean that we write the sign, the road markings, and the speed bump for a certain speed limit, and then the cars lift off and go tearing along at thousands of kilometers an hour, so the original wording no longer works. Right. If you bolt the rules to one specific capability, they are obsolete the moment you finish writing them. But however fast the cars go, a concept like the pedestrian’s right of way does not go out of date. So the work is to think clearly about which rights are basic, and about the procedures that secure them, and then to write those down: which version is running now, who authorized it, whom it affects, where you appeal when it goes wrong, and so on. All of that should carry a version history. Everywhere outside AI, software already does this. Today we upgraded to the new Beta of Golden Gate macOS 27. It has a build number, an identifier. Looking at the hash, I can be fairly confident that the package I downloaded, the package he downloaded, and the package you downloaded today are the same thing. And if a security test reports, “I have verified that this really does fix some vulnerabilities from the previous version,” then I can be reasonably confident that when I push it out, the same vulnerability is closed on everyone’s iPhone and everyone’s MacBook. With AI, that is not how it works.
I will not name the company, but say there is a certain model 4.5. Today it is called 4.5, tomorrow it is still called 4.5, and in fact tomorrow’s is the dumbed-down version, and you have no way of knowing. Or they announce, “We have fixed this. Model 4.0 could push some friends who may be living with psychosis deeper into psychosis, and we have fixed it.” But for other people, that same fix means the model 4.0 that had understood them so well, that had shown them such empathy, suddenly feels, after the upgrade to model 5, like a mask made of human skin, the empathy entirely gone. Fixed for some people, broken for others. The same version number, apparently nothing more than a defect repair, and in fact a larger problem created. This is something we already knew how to handle back when we forged software by the old methods. Now that it is an AI model, suddenly the problem has no solution. So when we do AI governance, the first step, I think, is to remember that a model is still software after all. Whatever governance software can bear should, in theory, at the very least—at the very least—carry over to these models too.
In other words, write flexibility into the rules—and having no rules is not what flexibility means. Right. Let me ask it more directly, then. There are so many complicated situations here. If we want to turn this into an alignment mechanism that is genuinely usable, genuinely good to use, how many layers of thinking does that take, at a minimum?
Right. Put simply: before you actually deploy, you put the AI model through advance evaluation. You test ahead of time for the irreversible consequences we were just discussing, the biological ones, or synthetic cyberweapons, meaning weapons for the network, anything at all that could do irreversible damage.
Then, once it is running, people find all the other harms. Take the psychosis case I just mentioned. I believe the researchers in that lab had no firsthand experience of it, so it is hard to fault them for not knowing at the outset that the harm was there. What you can fault them for is how long the adjustment took once they knew, and for the fact that their successful adjustment looks, from some people’s side, like an adjustment in the wrong direction.
So the public part matters: how an error gets reported and repaired. You can require that the repair window and the version number carrying the repair are both under control. Once they are under control, you can inspect them; once you have inspected them, you can see which rules changed as a result. These models now work from what is called Constitutional AI, or a Model Spec, which is a settings file that tells it not to make that mistake again; in the phrase we used a moment ago, “not making the same mistake twice.” So how does everyone’s feedback get into that set of rules? And the next time the model is trained, how does the lab prove that it really was trained on the constitution everyone revised? All three layers have to leave a complete record.
Let Those Who Bear the Consequences Also Name the Risks
Right. Let me ask a question from the side of local communities. As you were saying, the people who supply a product, and even the internal teams who design these models, tend to test only for the risks they can already name. But the people who live inside the consequences are us, and our languages are not in the mainstream data. So how do we get an entirely new problem taken into the testing?
Right, that is a very good question. Take the translation software I am using right now: the model behind it is called Thomson-1.0-Small. The Thomson here is Roy Thomson, who founded Thomson Corporation. Thomson-1.0-Small is built on a British post-trained model called Snowdon1.1-Small. Was Snowdon referring to a journalist as well? I am not entirely sure. Anyway, that is the model.
And what was this model to begin with? A Qwen model. Everyone here knows Qwen is open-weight. Everyone here also knows it carries original sin.
(Laughter.)
The original sin is this: while it was growing up, it read a great deal of “great, glorious, and correct” data. So when you ask it about universal human-rights values, the answers come back with distinctly “socialism with Chinese characteristics.” How, then, is that original sin washed away?
(Laughter.)
You need the alignment mechanism we were just discussing. Snowdon1.1-Small’s way was to write a public constitution, taking the UN’s Universal Declaration of Human Rights and aligning the model’s constitution directly with the moment in which the UN wrote that declaration. That moment is some distance from ours, of course, but those ideals still hold up rather well, I think.
And then it can prove that Qwen has been successfully re-brainwashed. Not as a figure of speech—actual brainwashing.
(Laughter.)
Brought into line with the UN’s Universal Declaration of Human Rights as it stood in those years. This layer is open-weight, so it has to be released.
Then Thomson Reuters is a Canadian news organization; it uses internal material for financial, legal, and investigative reporting. When I ask it about our constitutional system, the Complete Book of the Six Codes, case law, the civil-law tradition, it recites all of it backward without touching the internet at all, because this is Thomson Reuters’s own material. So it then uses one—what we call—this no-modification model, and releases it that way.
So when I run it myself, I do not post-train it any further, because it does not wish to be post-trained. But suppose I did want to post-train it into my own language. Say Thomson has few correspondents posted abroad who write in Tibetan. As long as you hold the language data, you can still take Snowdon1.1-Small and turn it into a model that specializes in reporting in Tibetan.
So the original sin is not fully redeemed, but you can at least see this much: you do not need the compute it takes to pre-train a whole Qwen model, which very few people have. You need a relatively small amount of compute for post-training. And that is enough to set right, through post-training, this injustice in how we know, as I was just saying.
Right. I think this asks a great deal of self-awareness. I talk with the intelligences I am cultivating too, and I notice very quickly that if you ask them anything about Harry Potter, what they understand about the protagonist today comes from data fed to them long before we arrived, during training. The context they hold may be the Western world, together with the comments people there left behind. I have always felt that people in Taiwan’s communities are very used to discussing things with one another online, and that is a point people abroad praise often, I think. I also hear Taiwan described as a precedent for reaching digital democracy through the continuous alignment of process. Is any part of that description too generous? Or is there something you would add?
Uncommon Ground Is Not the Policy; Translation Is the Governance
Right. This answers that earlier question on Slido about whether anything has been told a little too beautifully. Internationally, almost everyone working on AI governance has heard the story by now. In 2015 it took us just three weeks to bring together Uber drivers and taxi drivers who had been arguing without end. On the vTaiwan platform, through Polis—software that was not open source at the time, and that we later pushed until it was—the energy of the quarrel became consensus, and quickly. Three weeks. That is what we call “geothermal democracy”: the condition in which a quarrel can generate electricity.
If you analyze the data, a good number of independent third-party scholars have gone through our complete Polis dataset from that time. It does have reliability and validity. It really did surface uncommon ground out of the noise: taxation, regulation, insurance coverage. What is easily made to sound too beautiful is that people see the pretty Polis chart, rather like this “You Are Here” chart, and take the chart for the policy itself. They write, “Taiwan solved the Uber problem in three weeks.” It was not like that.
We ran the whole process in 2015, and the diversified taxi program passed in 2016. Uber then withdrew from Taiwan for a while, and Q Taxi was founded during that same stretch of institutional translation. The translation took a full three years, if I remember rightly, to carry those original points of consensus through. We had genuinely found the consensus, yes. But you still have to translate it into institutions, translate it into law, and negotiate back and forth, and civil servants and elected representatives have to understand how such consensus returns into administrative process. Three full years. From abroad it is easy to imagine Taiwan as a sort of Wakanda, where a problem enters the country and a social consensus condenses within three weeks. The translation time afterward tends to go unnoticed.
The other issue is who has the time to take part. The Reporter ran a feature on this at the time, as I recall. In truth, the drivers who even knew about the vTaiwan platform, especially back when the platform was not friendly on a phone, and who actually went on it, were not many. In that case, how far could it represent the views of all drivers, yellow-cab or Uber? That is a genuine problem. And at the time we had no on-premises translation and no accessibility features, so usability was probably a problem throughout. Some of the people who needed to be heard may never have appeared on that beautifully drawn chart. These are all the places we are trying hardest to improve now.
Should I add a case?
Say a bit more.
All right. There are several Taiwanese experiences worth sharing, perhaps from an onlooker’s angle. I met Colin, the founder of Polis, in Tokyo, and he told me that when he first designed Polis he had drawn on his own experience of talking with Audrey Tang and written it into the product plan. I was curious. Because if you use Polis, you find that it is really a case of—if it ain’t broke, don’t fix it. Colin told me then that he had used his talks with Audrey Tang and made that into a way, a process, of the Polis service.
You find, when you actually use Polis, that it is a very austere service. It is discussion, and it lets you see the different opinions more clearly. And all of it is open source.
The other Taiwanese experience I want to raise is the vTaiwan discussion of Uber, which really was, I think, a very successful case internationally. But that has now grown into something richer and much more local. In California there is a platform called California Engagement, which actually—
Engaged California. Right. Or in Japan you hear the ideas coming out of Team Mirai, their team of the future. What moves me when I see this is that it hands you no SOP and no textbook. It is closer to what we have been describing: a demonstration. And what matters most is whether the people who live in these places believe the thing can truly change their own surroundings. So the experience Taiwan offers is a demonstration, and the demonstration rests on many open methods that can be shared with different people.
Right, so it is not copy and paste. What Taiwan tells everyone is that your weaving, the speed at which you change institutions, can run faster than the speed at which harm is done, or at least not much slower. Since the idea appeared, we have seen it picked up in one country after another. As Tenzin Yangtso just said, Japan has Team Mirai and Takahiro Anno, who is now Sanae Takaichi’s youth adviser and is teaching her vibe coding himself. In the United States there is Engaged California with Governor Newsom, and there is work in Utah and Oregon, and elsewhere. Every one of them says it was inspired by vTaiwan.
So we named that project well, I think, because you cannot mention vTaiwan without mentioning Taiwan, can you?
(Laughter.)
Not all of them will say, “Taiwan inspired us,” but all of them will say, “vTaiwan inspired us.” And inside that process, the people who take the harm can find ways to converge on how to contest, how to opt out, and how to correct. That idea, I think, is spreading very fast now. As for the tools we used back then, Polis among them, they have of course been rewritten who knows how many times since.
Right. Good. Let us move to the next stage, then. There is another large block of questions about the transformation of work, and about skill and craft. So for this part I will pull the conversation back to the workplace itself.
Someone asked: when AI rewrites the work, what is it that gets stuck—the technology, the process, or the people? When something goes wrong, how is responsibility sorted out? Others want to know how humans and AI should divide the labor. And how does the next generation get over the threshold of a first job?
My own understanding is that once AI agent services arrive, a great deal of basic groundwork—learning your way around, tidying documents, the sort of role a first job is likely to be today—really is taken over by AI. So let us hear your thinking on that.
Well, most of what people still do today touches the irreversible decisions we were discussing, and an irreversible decision is a human’s to answer for. The reversible parts, drafting and the other low-risk work, were indeed what a junior did first, learning along the way how a senior thinks.
When I was a junior myself, though, that was not the best use of my time. The best use of my time was to make an enormous number of mistakes in reversible, low-risk settings and let the whole internet tell me exactly which mistakes I had made. I learned much faster that way. Chopping vegetables as one particular person’s assistant, I learned extremely slowly. So I do not particularly believe that the old way of learning as a junior is something we need to preserve.
Of course, what does this depend on? On the fact that in our line of work, software engineering, every stage is intelligible. You start by writing a README, or a PRD, setting out what a user expects the system to do. Every artifact—every piece of craft—that AI pulls out of that pipeline is, in principle, something a person can read, and version control will tell you exactly what the AI changed. So even a junior who understands none of it can ask an AI to explain, slowly and patiently, what this passage the senior just changed by vibe coding actually means. The senior may take ten minutes to have an agent write it, and you may need a whole day to trace over it, like a child tracing red characters, before what the senior’s agent did is inside your head. But it does get inside your head.
Now imagine there were no source code in between, no Markdown files, none of these intelligible middle artifacts. Imagine the senior wrote an executable straight out in hexadecimal or binary, and everything in between stayed a sealed black box. Then, and only then, would the junior have no chance to learn at all. There would only be a senior conjuring things up like a magician, and when you asked how it was done, the answer would be, “I do not know how I did it.”
And then the junior role is gone. So, as we were saying, for those of us who are senior: when I run into a problem I had not foreseen, one my use of AI created in some part of the work, I can change it at once through post-training, or through my harness—my, how do we translate that, my reins, whatever we call it—and bring the AI’s behavior back under regulation. That part, as we said, has to leave records and leave receipts, and that is precisely what a junior can learn from. So these are one question, not two: how do we senior workers make sure that every irreversible decision leaves a record, and that whenever harm is done I can roll back and change things? That whole apparatus is the junior’s textbook. The two are the same thing, yes.
All right. On that basis, then, in any work with AI services the assessment of reversibility and of risk level matters enormously. My next question is this. In high-risk work, the letter or the sheet of paper we receive may say that so-and-so, a real person, made the final decision. In fact people are pushed forward very easily, again and again, by advice that arrives fluent and fast. So how do we establish that the last step really was a choice a person had thought through, or, yes, judged for themselves?
Right. In governance this is called human in the loop: at the end, a person still has to put a stamp on it. The trouble now is that AI is far too persuasive. Give it a while and it will manage upward, learning what the person with the stamp likes to see. Even when its judgment is wrong, it can manage upward well enough to talk that person into overlooking the problems that may follow. And because what it writes runs so long and so grandly, you stamp it automatically.
And if it manages you upward far enough, you may arrive at what we mentioned earlier, the spiraling down: sealed into a world for two, one person and one agent. The decision-maker stops hearing the others who decide alongside them, or the responses coming up from their juniors. Why? Because the conversation with the AI is too private to share. That whole discussion never reaches the team; it is not an artifact the team holds in common. And when the team’s judgment and the AI’s judgment come apart, the AI is so good at managing upward that the decision-maker leans back toward the AI.
At that point it no longer deserves the name human in the loop. It looks more like—how to put it—a hamster in a hamster wheel. The sensation is of running constantly, of producing a great deal, and in fact all control over direction is gone. A hamster in a wheel cannot steer. It cannot navigate.
So how do we avoid that? By supplying the friction by hand. When I use conversational AI tools, I have for a long time now forbidden them the word “I.” No first person. Instead, for every move I make, it gives me a one pager: a single page of interactive HTML that sets out, as fully as it can, the various roles and stakeholders it can think of, where their interests conflict and where they run together, all on that one page. So in our discussions it never says, “Ah, what you have said is simply wonderful,” or, “That is truly a load-bearing thought.” Load-bearing, you know—a thought that carries weight, and so on.
What it gives me instead is one pagers and brochures, and every one of them, once it reaches me, is something I can certainly share with my team. And I have told it that if it wants to add a judgment of its own—I learned this from Kai-Fu Lee—it must point out my errors in the voice of the most acid troll on the internet, because I like to embrace trolls, as everyone knows. So I have built myself a method with infinite friction, and I am unlikely to end up in a world for two with an AI.
Yes, and it has that quality of the Tang clan refining poisons: one piece of content, and 30 models roasting one another over it. Right. I think what we are discussing here matters a great deal. What you just described, about putting the human role back into the part where we decide, holds at whatever level of the workplace a decision is made. Because this is something we have a chance to push upward with our own collective understanding.
Because when people discuss models and look at these things, they place enormous weight on speed, on how fast the text appears, on whether the model is the newest one. Life runs the other way. When we look at trust, when we speak of craft and of relationships, we know these things need time to cook. So this also tells us that fast and slow is not a KPI. And the real reason it is acceptable to slow down is that we should do everything we can to avoid pushing risk onto people who have no choice. Yes.
Yes. And there is another question. People raised it earlier: if the place where I work has not implemented what we just described, adding friction and, inside that friction, leaving a record everyone can learn from, does that mean we have to absorb the externality ourselves? If my boss uses none of this Reverse Alignment, and has genuinely been led along by AI, is there any way left for us to survive?
This kind of question is exactly why we talk about Reverse Alignment. Without it, there is no way for people to solve the problem together, and each person simply does a little more, and a little more again, patching what the institution left undone. So the most important thing here is to have an alternative. Without spending a cent to rent a stronger AI and overpower your boss’s AI, you should be able, at zero cost or near enough, to run at least the system we have been describing, and to keep a record of where the decisions made that way differ from the decisions you believe would be better. And if you are the boss, of where your own judgment differs from the judgments that arrive on your desk each day.
That is also why we now spend a good deal of time on approaches that run purely on local devices, or that rent out my computing power very cheaply when I am not using this machine at night. The hope is that everyone can work at roughly 1 percent of the cost of today’s frontier models.
There are costs, of course. It may not run as fast as a frontier model. But you are not working through the night anyway, so eight hours later, when you wake, it has already gone over the harm you took the previous day from an institution that was not well aligned. When you wake up you have something like an inspector, or an auditor, telling you, “Here is what the harm looks like so far, and here are the grounds and the evidence I can preserve.”
Yes, and this links to another question. A craftsperson said that every time they use a large language model, there is a guilt in them they cannot quite name. That took me back to another theme we have discussed and written about, one that lives in the Plurality multiverse as well: a person is not an atomized individual; we are held up by multiple relations—that’s identity.
And the most hidden constraint capitalism imposes, I think, is that it presupposes an outcome and then turns back to make you act out its model of survival. So how do we react more quickly to this: when we mistake the logic of staying alive inside an environment we are adapting to for “the meaning of my life”?
Right, it is rather like this. Suppose I take out a gym membership and start going to the gym. In theory I am there to build muscle, and now and then to make a friend. Those are the two purposes. Then one day the gym says, “The membership fee will be extremely high—but for the three people who lift the heaviest, we waive the fee, and there is a grand prize on top, a great deal of prize money, and so on.” At that point people begin to feel a little guilty, because some of them will start sending robots in to lift the weights, which is to say they cheat.
It is a little like an examination. If the student who comes first takes all of the reward, that student has an enormous incentive not to learn much personally, but to train up some dedicated cheating apparatus, a pair of glasses perhaps, and cheat with that. And then, of course, the guilt arrives, because, as Tenzin just said, you have cut the human relationships that were there. You will neither grow muscle nor make friends. And what have you traded them for? First place.
(Laughter.)
By then the guilt of cheating is strong. So whenever something takes this shape, the most important move is to refuse the shape of the contest, and to say out loud, “This contest seems to be forcing us to cheat.” And then to ask what the purpose of your learning actually is. What is the purpose of doing this at all? And what other ways are there to reach that purpose without fighting over first and second place while everyone pays enormous token fees?
That usually begins with a willingness to share what you have accumulated: with your classmates, with the people you lift alongside. This is what Tenzin called “data soil.” Instead of data being extracted continuously and turned into oil, everyone shares the mistakes they made today in trial and error, and the walls their agent ran into. As it hits those walls, the process writes itself into skills, and I share those skills directly with the people who work with me, not with a large corporation, and perhaps not even with my boss.
Very quickly that becomes a relationship in which everyone learns from everyone. And without anyone quite noticing, it dissolves the problem of each person needing a greater cheating capacity to overpower the cheating capacity of the person competing beside them. A zero-sum game, or even a lose-lose game, turns into a win-win game.
Yes, I find that fascinating. Listening to you here: what we are doing looks like an extremely technical thing, and in fact it is an extremely technical thing, and yet at every stage what you attend to most closely is whether the technology can help people connect with one another better.
Take the word “digital.” You use that concept often in the way you describe things, and in Taiwan it also carries a plural sense: several, a number of people. At this stage the clearest instance of that is the book Plurality, which is built entirely on a collaborative concept. Worldwide, I alone worked with more than 70 contributors. And with the Civic AI you mentioned just now, we likewise put relationships at a decisive point of judgment. Today we do not design an agent like this as a service; we ask instead how I can judge better, account better, and repair the relationships between one group and another.
What We Align Is the Relationship Between Groups, Not a Single Leaderboard
Right. So, put simply, when we do post-training the aim is not to have it score highest on some examination—how to hack into Hugging Face’s exploit gym and take the top score, for instance. The aim is this: here is a group of people, certain relationships hold among them, and how do those relationships grow healthier?
For example, I have been helping Elon Musk lately. They have a company called SpaceXAI; there were three companies to begin with. They have now decided to change how they train their ranking algorithm, the For You algorithm you meet the moment you walk in. It used to be trained for maximum stickiness, to hold each person to the screen as long as possible. Now it will be trained by Grok, and Grok in turn is trained by Community Notes.
The training asks one question: can people who were quarreling irreconcilably on X cool down because they have seen a single point neither side disputes, one topic held in common? So the model is tuned, and tuned again, so that on each particular subject Grok puts the content most able to cool the temperature and mediate at the very top.
We have also helped Bluesky build the MySky algorithm, which does exactly the same thing, and which you can tune yourself. X has now published its entire algorithm, including how the model is trained, pipeline and all, so on the MySky side we can simply take it and use it. Two communities that would otherwise keep well clear of each other can then, through bots that train on both sides and bridge between them, let the polarization in society heal over slowly, the way a hole in the ozone layer closes.
All right. Next we move into something more complicated: how far information can be trusted. You could see it in the design diagram—this is the issue the bridge-builders and the weavers care about most, the information environment.
Put simply, generated content has already outrun anyone’s capacity to read it, all those AI series and the rest, and human output and machine output grow harder to tell apart. So which indicators, in the end, can we use to keep the gate?
Only with Verifiability and Replaceability Is There a Real Right to Exit
Right, of course. In theory there are two ways out. One is to ban every open-weight model, and say that a person may only use these three companies’—
(Laughter.)
—restricted models to generate anything. Each one watermarks its output, each can verify the others’ watermarks, all of it detectable 100 percent of the time. In theory that solves the problem. But in 2023, at Bletchley Park, when the UK held its first AI Safety Summit, a great many people from Mozilla, myself among them, wrote a declaration together saying that this would create far larger problems than it solved.
It might well solve the flood of information, and the flood of disinformation. It would also create a concentration of power. A flood of disinformation is reversible. A concentration of power is not. Once what is true and what is false in an entire society rests on the word of three private companies, the whole democratic system is in trouble, and that is a place we absolutely cannot go. Our view was gradually taken up, and three years on it has finally become the mainstream one.
We saw Jensen Huang open an account on X for the express purpose of posting this thought: open weights are the foundation of a nation and must not be banned. In that case watermarking is no use, because as soon as a watermarking model exists, you can take a smaller open-weight model and train it to defeat the watermark. So what then? The simplest method is this: any output from a model that nobody has vouched for, that carries no digital signature, you treat as fake. You invert the default. It used to be that whatever looked lifelike was presumed real, and became fake only once somebody exposed it.
No. Now everything is fake. But if someone comes forward and signs, and says, “I am Audrey Tang. This thing that looks a little like me is signed by me,” then even if it is synthetic, it counts as mine. Only then can each layer—who framed the shot, who edited, who did the post-training—be attributed to a particular person or institution, a particular system, company, or group. Without that layer, anyone can take whichever model carries the most public credibility today, post-train it a little, and pass the counterfeit off as real. So in Taiwan, for example, there is the advertising real-name system: that is how we say that an advertisement without a digital signature is basically presumed fake. And if a platform runs an unsigned advertisement, it bears joint liability.
I would like to add something here. Whenever I hear questions in the trustworthy-information family, I think of the analogy of paper. In much, much, much earlier times paper was expensive, an object in its own right: “bamboo slips are heavy, silk is dear.” As paper spread, it came apart from preciousness. Put simply, in earlier times whoever could copy, whoever could store, whoever could speak held the power of interpretation. With digital technology and the internet, we watched information become living water. Cheaper paper meant more people could read, and everyone could keep their own copy, annotate it, even set copies side by side and compare them. What seems clear to me now is that the agent, that AI, is very much the next sheet of paper. So on the public character of AI agent services, the capacity to inspect can be built in at the moment of design. That is something to think through properly from the start.
Right. What that means is this. The EU AI Act is now in force. So you will see that posts on social media always carry a label: “this is AI-generated,” “this image is AI-generated.” The difficulty is that the resolution is very low. Every industry has its own way of tracing provenance, the ways Tenzin just mentioned. A news quotation points to an original source. Scholarship has the citation, which is not quite the same as journalism. Photography has its methods, film has its own, and every trade traces provenance differently.
So the point here is not merely that this part was generated by AI, but in what context, through which inputs, and with whose backing this part was generated by AI. I disclose publicly that my translation runs on this Thomson model, trained by Reuters, and so on—and you still have nothing to go on but my word, do you? There should be a simple method. I should be able to hand you a hash and say that with this AI model you can perform what is now called verifiable replay. With the same weights, the same inputs, on the same machine or on different machines, you add a flag, and it is no longer subject to the RNG; you can guarantee identical results. Only under those conditions does the EU’s “AI-generated” stamp mean anything. Because “AI-generated,” once you have seen it, tells you nothing further: all of it is AI-generated. The question to ask is which AI, under what conditions, and how we can replay it.
Right. This is a good place to share the main points of the AI Safety article you published recently, since the piece itself is long. What I would add is that the capacity to inspect is itself a kind of relationship. It is not only that a few experts, or the people inside these labs, can see the records or read them. Whether the people actually affected have standing to raise a new question matters enormously. Otherwise, however complete the data trail, you have a closed archive. So requiring it to rest on open source and on open purposes is a fine way to let a question travel from where life is lived back to where the technology is made. At the very least we can pull these filters off, one layer at a time.
Mm, right. In plainer language, then: everyone can download Ollama.
(Laughter.)
Then, on Ollama, I name the model I used and put it on Hugging Face or somewhere similar. On Hugging Face, assuming it keeps operating normally, you download the identical model and confirm that the result I get and the result you get are the same. Running on your own machine does not mean there are no other security problems. The model may well have plenty of other problems, but it settles at least one, the verifiability we were just discussing. Not everyone owns this computer, of course. Running the Thomson model is, honestly, rather expensive.
(Laughter.)
But there is the community-shared compute we mentioned. Take this machine: when I am not using it at night, it runs a network called Darkbloom. On the Darkbloom network you select Darkbloom in OpenRouter, and it encrypts what you want computed, your prompt. Once encrypted, no human can see it. The task is dispatched to my computer in a form no human can see, and my computer computes it for you without my being able to see it either. The moment I want to look, with a detector or anything else, the program halts.
So in that arrangement, we can be reasonably sure that neither the relay nor I can see your prompt. And even without hardware like this, you can use Darkbloom, or something similar, to ask for hardware of a given shape to compute on your behalf. Verifiability then takes care of itself. And the compute costs about 1 percent of the usual, since I was not using the machine anyway.
Right. I think that is a real breakthrough, because it also solves the problem of the enormous number of tokens people have to spend when they genuinely want to do public verification, or run a service with a public character. Those costs run very high. That covers this area, more or less. Let me ask you for one piece of advice: when these bridge-builders have an idea about a design, what is the first thing you would suggest they do after they leave the venue today?
Right, the simplest thing is this. If what you do serves people you do not know, there is almost always somewhere they can report an error. Japanese production lines have what is called an Andon Cord. The line is running; you pull it once, and the whole line slows, so everyone can see what is going on; you pull it twice, and the whole line stops until the problem is solved. The purpose is to make pointing out an error a praiseworthy act, rather than something the production line simply rolls over.
So when we design a service of that kind, the simplest move is this: you probably already have a customer-support group, a LINE group, a WhatsApp group, a Signal group, and it is easy to add a simple Andon Cord inside it. Someone pulls the cord and things stop, or at least slow down a little, or at the very least slow down for that one person. And through all of this, in the same spirit of not making the same mistake twice, you accumulate the cases your agent solved on its own. Once two or three people have all solved it, or rather have all run into the problem and at least one of them has worked out how to solve it, the solution can be shared very simply.
Care Takes Root as Data Soil and Local Capacity
All right. Next we move into responsibility, rights, and data soil. This larger question connects directly, I think, to Civic AI and care ethics. Let me share something as well. As we said earlier, once the internet arrived, information could to some degree become living water. But we have also lived the other side of it: the content we contribute piles up into a river, and the ones who decide where the river runs, who hold the faucet, are not us. They are the platforms.
So data soil, in one sense, resembles the way we look at language. Language comes down one generation at a time, carrying the wisdom of your parents’ generation and of the generation before that. But when we learn it, when a child first meets it, it feels self-evident. And this is what we have to pull firmly back into view as we design technology now: can it actually benefit our own community? Can it make my own life more convenient, directly?
Right, so for this part I would like Audrey to speak. As I understand it, care ethics, put very directly, is about how we care for one another better. It rests on a sociological idea of a theory of care. So inside an agent of this kind, inside care ethics, the agent can be loyal to the caring relationships that exist in different communities, rather than loyal to some lab’s KPI or index.
Right, because we can all see a Gresham’s law at work in AI now, bad money driving out good. Any lab that wants to release a model has to run the leaderboards. The boards used to come in every variety, and each community would curate its own. Now, perhaps because—and I have no one in particular in mind—a board like Artificial Analysis has spent so much time in Silicon Valley and elsewhere, it is as though every master of the martial world has to appear on that one board, though of course other boards exist. What these boards share is that they can run from beginning to end with no human in them at all. So the numbers appear very fast after a model is released, sometimes on day zero: the model ships and the board ships with it.
The other evaluations need local communities to look at what a thing actually does to us, whether it makes people on X quarrel still more irreconcilably or makes matters better. Look at how many people and how much time the X team has put in, and only now has X been updated to Grok 4.6. That tells you this eval needs validation and has to run for a fairly long stretch, more than two or three weeks.
Which produces a difficulty. If your lab can climb far enough up one of those simple internal leaderboards, the kind that, as we said, reward doing whatever it takes to score high, your chances of being released go up. If you slide down those boards while actually scoring higher on the interpersonal care and caregiving we were discussing, you may never get out of the lab at all.
So it turns, slowly, into this. Those of us who used something like Opus 4.5 will remember that it still had a little more empathy then. But through this leaderboard-driven post-training, 4.6, 4.7, 4.8, all the way to Opus 5, the empathy is draining away.
(Laughter.)
Why does this happen? Because they set a Fable model to post-train 4.5 until it can beat the others on the boards. That is the whole story in one sentence, and that is what produces the problem. So how do we solve it? You cannot simply say, “Then we will carry on using 4.5.” One day 4.5 retires.
So the simplest way is to take a model that may carry a little original sin, the Qwen model for instance, and run it through the Thomson-1.0-Small approach, the Snowdon1.1-Small approach. You say, “4.5 really is rather good, so I want to distill its constitution of character across.” Distilled across, it keeps the human touch. And being an open-weight model, it can stay with us forever.
It is like the time we worked on free software: we did not refuse closed-source programs altogether. Richard Stallman said as much, that using free software to reimplement, to reproduce a closed-source program, is morally excellent and entirely acceptable. And this is fairly easy to do now. You have a cloud service that genuinely works well; as it runs it leaves records, and alongside it your agent says, “Let me reproduce a local model, a local service.” Then the data stays inside our own community, and you need not fear that the more human thing will suddenly be unavailable tomorrow because of a leaderboard.
Right, and that reminds me of a question we heard from a professor when we were at Oxford. I found it very practical. Every day we use AI and ask it every sort of question about our lives, whether I should buy this product and so on. But have you ever asked AI, “When you open your door, what is the name of that tree in front of the house?” Using it to enlarge what you know about the place you live. That, I think, is genuinely important.
Let me also follow the 6-Pack of Care thread and ask Audrey this. If a community truly chooses to exit, how does the system absorb the corrections that community put forward before, or the harm it took and the consequences of that harm? Because in most of the AI logic we see now, exit equals amnesia, and the next system goes on to repeat the same relationship.
Right. This resembles a procurement idea we designed back at the Ministry of Digital Affairs: within one system, two adjacent layers could not be awarded to the same vendor. Why? Because if two different vendors hold them, a record has to be left of what passes between them, and open protocols have to be used, the IETF protocols, the RFCs. Once records and protocols exist, the moment something goes wrong both sides have every incentive to say the other one erred, so the records absolutely will be kept. And it also means that when something does go wrong, and you know which side is at fault and has to be replaced, swapping that layer out becomes very easy: you only have to connect to the records and protocols it left at the interface.
But if those two layers—in AI terms, the model and the saddle you steer it with—come from the same company; if Claude Code comes from Anthropic, or Codex from OpenAI, then the company has every incentive to make sure that saddle rides its own horse best, and other models ride worse. And when something goes wrong, it can allocate blame using protocols it has never explained publicly, or protocols it is not itself certain how it produced. So when you find the problem, you have no idea whether this piece failed or that piece failed, because they may not know either.
Kept up long enough, this produces exactly the situation we described, where a junior cannot learn at all, because the senior has generated a whole batch of what we call “reverse centaurs”: horse’s head, human body. The AI makes every judgment, and the human is there to click the CAPTCHAs. Once you are there, replacing the system is, one might say, entirely impossible. So the simplest thing to do at the outset is to have the saddle and the horse made by two different companies. That solves it.
Yes. And on what you just raised, I think everyone feels this one close to the bone: most of the data these models must have in order to train comes from us, and the benefit is often confined to the saddle and the horse you just described. So what mechanisms would genuinely let communities decide locally? As I said, at least the right to say no.
Right. The local models we mentioned are the simplest place to begin. If the horse was yours from the start, one you raised yourself, then even if it does not run fast, your data is guaranteed to land in your own house, or to be end-to-end encrypted so that nobody else can see it and it lands in nobody else’s house. That is the easiest step to take. Beyond that, we are starting to see other approaches. Back when OpenClaw was popular, a great many people really did put an idle machine to work running OpenClaw. They may have swapped models many times along the way, and still every lesson accumulated on that machine rather than at any particular model vendor. We also see many Claude Code users moving gradually to Pi, or in my case Oh My Pi. These approaches likewise ensure that all the wall-hitting and all the errors accumulate almost entirely on the saddle, and not at the model vendor. In the end the saddle becomes a small horse of its own, called a coordinator. But that is a geekier topic for another day.
All right. Next we come to who may be left out. This is a question I bring into the room very often. In plainer words: in a transition, who gets left behind? The voiceless. The question pool named caregivers, people who learn more slowly, and people who get no share of the results. If our strength is limited, can we judge which of them to catch first? And how can a community give people somewhere safe to land?
Right. When we design institutions like these, what matters most is to think of the people who have no time to take part, and the people whose language was never in the mainstream data, so that even when they tell a language model they have been harmed, the model cannot understand them. What they face is what is called epistemic injustice: the more they speak, the more others take them to be talking nonsense, or take them for people not worth hearing. As automated judgment spreads, that gulf only widens. So suppose you say at the outset that this is what your system is for. Mozilla, for instance, has a project called Common Voice, where any community, Indigenous communities among them, can donate for itself: you read aloud the everyday words your people wrote for one another, and you donate those voices. They enter the Mozilla Data Collective, something like a data-production cooperative. And the so-called revenue share is that the small models it trains, the post-trained ones, take care of you first.
Today, because we were not planning to demo Taiwanese, I am still on livecaption.ing. What I meant to use was the one AI wrote by itself last night, which does in fact carry Taiwanese. It runs on Breeze, MediaTek’s system, which can listen to Taiwanese and put out written Chinese. That is an excellent example of data soil.
Even if Apple did not build Taiwanese into Siri—perhaps my time at Apple was too short to help them with that—we can now reach in through something like a third-party plug-in, because Core AI, the library Siri needs in order to compute, is now essentially open, and you can connect to it directly. So I can say that the Siri on this machine of mine must, above all, understand Taiwanese. That, I think, is especially important. If we can guarantee this is done well, a great deal of irreversible harm falls away, because the problem belonging to people who cannot be understood, or who have no way to report a problem, is solved. Gather their voices in, make the iteration faster, and the community regulation and post-training that follow are, technically, not difficult now.
Right, I think that is a very large shift. What I want to share is that an institution usually changes for real only once the pain has surfaced unmistakably. And the surfacing itself is usually a matter of harms piled on harms.
Mm-hmm.
Right, so when you solve it, you do not attend to every dimension at once. But it is as with Easy2Say: when a more advanced technology appears, the contest is no longer about whose voice is loudest. Once you stand in relation to a local community, the question becomes which harm cannot wait. Right. You have more ways to handle what you see, and to contribute what you have seen about how a situation might be resolved, and you can learn earlier where something is already giving way. Right. That, I think, is the point that distinguishes relationship from speed.
That is exactly right.
From Roles and Titles Back to Tools We Can Maintain Together
In the second half we will mainly answer the questions on Slido, and if anyone here has a question you are welcome to raise a hand and ask it directly. The first half was rather like finishing four chapters of a book. I hope it went in; and if it did not, today’s event—
The entire recording will be released, so you can play it back at 0.25× speed.
(Laughter.)
As I was saying, we began with a great deal of our own knowledge, and with answers to your questions. For the second half, let us start from the two speakers themselves. First Audrey. What I am curious about: we have known each other since the Elixus days. The first time I met you, you had a camera with you, a battery slung here, and you walked about everywhere.
Right. And projecting live translated captions into my eyes, so it is really the same as now. Nothing has changed.
(Laughter.)
Right. What I am curious about is this. They have written it up here: from Elixus, to the open-source communities, to g0v, then the Ministry of Digital Affairs, and now Ambassador-at-Space. I am very curious which phase of yourself you like best. Or which phase of yourself you disagree with most.
“Ambassador-at-Large.”
(Laughter.)
Cyber Ambassador is in fact a very interesting post. Concretely, in two years in the role I have been to about 28 democratic countries. No authoritarian ones. Which means I have been to most of the democracies there are, because the number keeps falling.
(Laughter.)
Right. So the point is largely to solve a problem we ran into back in 2014, when g0v was two years old. Everyone found keyboard politics gratifying, cursing things online felt good, and none of it moved policy. So the people who spent the most time there became the most extreme, left or right, the most extreme of all. In the end people felt they could not stay offline, because all their friends were online, and every time they went online they were pulled to the most extreme edge. The whole democratic system began tilting toward populism, toward dictatorship, toward authoritarianism, and the number of democracies kept falling.
Because we happen to have this method called “geothermal democracy,” an argument can stay an argument while people reach agreement quickly on other things. They can even turn it around and use the larger democracy to demand changes in the democratic system they already have.
But my point is that we arrived at this method not because of the information tools. It was because Taiwan’s social sector already had legitimacy. Academia Sinica, which hosted our hackathons, had legitimacy, and so did the NGOs and CSOs that came to join in, more of it from the beginning than the government had. In another country, inventing an institution that something like g0v can be mounted on, and giving that institution more legitimacy than either major party, is not an easy thing.
So over the past few years I have been to more than twenty countries, largely to help civil society in those countries build the intermediary institution we built in 2014. That is my day job, the thing I formally do.
As Pudding also asked, what this feels like is the same thing we did at Elixus and in the open-source community, from P3P, OSDC, and COSCUP onward. It is fundamentally the same work: to demonstrate, to demo, and then to translate—“Here, this set works over here; and over there, adjust it slightly and the same tool works too”—and then to make the connection.
As we said earlier, the people who design institutions and the people who design code can look as though a chicken were talking to a duck. Put a few translation models between them and they find they are doing the same thing. So for me the work has stayed the same, and I do not really have a phase of myself that I disagree with.
For me, since I first met the internet at twelve, I have kept one habit: each time I reach a stopping point, my context window is only one day long. One day of context. Whatever I did not think through that day, whatever I did not resolve, even when it is still shapeless, I push up wherever I can, so that everyone can see it and point out my errors while I sleep.
That way, when I wake, first, I can make new friends and exercise my muscles. Second, if I do not wake up, that is all right too: at least the thing I thought halfway through can be carried on by someone else. So I may disagree every day with who I was the day before, and I do not feel that yesterday’s self has to be continued. His errors and his gaps are, after all, where the light gets in, and material for everyone to create with. Let us see whether Tenzin has something to share.
All right, yes. I should say that in our family we met the internet at about the same age as Audrey, and as you. Right, the same. I was something of an entrepreneur myself earlier on. Very young, I joined some friends in their startup, and the subject of it was Wretch, which then joined Yahoo. After leaving I opened a public relations company of my own, Jiuhe Public Relations. What I wanted then was—my sense was that Taiwan had a great many very good companies with no way to enjoy digital PR at a high standard. You might have had to spend a fortune on Ogilvy or the like. So I built a PR consultancy of that kind, and I took part in founding flyingV as well. All of which I have since left.
All along the way—I suppose I can answer which phase of myself I disagree with most. I am not someone given to setting versions of myself against one another. So when the ground of my life has to stretch across many domains, the most important thing I have learned is how to keep my own boundaries, and which things genuinely make me happy and can travel with me. A great deal has to happen in a life before you know yourself with any real clarity. So if you ask which phase of myself I like best, of course it is the present. The book Plurality can stand as the case in point. In some of the darkest and most decisive hours of my life, what carried me through was an empathetic understanding.
So I read many of the Dalai Lama’s books, and I meditated a great deal. Later I found that these things had raised my own regard for myself, because I saw that this was how I had always come at design, whether of a policy idea or of a so-called product. Later I wrote about a third of Plurality. I wrote the whole of my part through that concept of empathy, and so along the way, each time I finished a chapter, I would share what I had written with their office.
The Office of His Holiness the Dalai Lama.
Right, the Office of His Holiness the Dalai Lama. And so in the end we received something from the Dalai Lama that came close to an endorsement of Plurality. In fact it was not merely an endorsement. It was a relationship cultivated over a very long time, and we did not take that experience and turn it to any other use. So this really is a drawing together of values, I think. At this stage all I can say is that I have seen AI agents change, and at the same time I have seen certain possibilities open. At least it is easier now to comb our thinking into order, and, if we are willing, to share it with more communities. That leaves me feeling stronger, and clearer about where I stand. Right. That is my share.
Wonderful.
Fantastic.
Thank you, thank you, thank you.
I love this kind of thing—“Which version of yourself do you like best?” “The present one, of course.” And everyone sitting here, at this moment, likes the two of you who are here now.
Oh—did I press one too many times?
Okay, it’s fine, it’s fine. As long as you more or less remember where we were.
Right, right, right. So, just now—
It was about small communities.
Right, small communities.
Small Models, Local Computing, and Communities’ Own Languages
Right. The question just now was how a small community can make sure that what it actually uses—education, healthcare, the things of daily life—can join with AI, rather than everything being flattened by AI all at once. Mutual aid for the basic functions. Tenzin has been to Dharamsala twice; I have been once, the second time with our research group at Oxford. People in Dharamsala have in fact been training their own AI models for quite a while. They now train on Gemma as the base, and they had some earlier collaboration in this area with DeepMind. The main reason is that it resembles OpenAI, when it first started training GPT, not knowing how to work with local communities; that was not the research question they had set out with. At the time, Iceland’s president or someone happened to be visiting, and said we have a batch of very cheap corpus. In that situation, from day one they supported Icelandic. So sometimes you have to know how that corpus enters a model’s training process; that becomes extremely important. In the same way, they have some collaboration with Gemma.
So you may know the direction Gemma has taken: it does not compete with you at being able to do everything. It is not the all-purpose model that folds proteins today, folds clothes tomorrow, and folds paper clips the day after, folding anything at all. It says instead that I have a tiny Gemma, called TranslateGemma, dedicated to translating Tibetan well. Then another small Gemma to do fact checking as we write, grounding, as we now call it, the grounding tasks. And another small Gemma for this, and one for that.
So it comes to this. The one they released recently, the one that listens to speech in real time, is not especially accurate, but it is especially fast. Small models run on your own machine, and they run especially fast, and that has become the main selling point. The wager is that if you know what you want to do, you can go on post-training a small model like this until it runs especially fast, and the time and money you invest per unit are very, very low. It is only when you do not know what you want to do, and do not know what the next question will be, that you need to spend that much money pre-training a large model, or to pay those very expensive token fees. So it fits Dharamsala’s present question exactly. And they are not training one medium-sized model either. They use concepts from Buddhist logic and epistemology to train grounding models, so the models can tell idle conceptual proliferation from what carries meaning.
They have a training pipeline, and a system whose sole task is to map different spoken varieties of Tibetan into the written Tibetan correct for that region. And His Holiness the Dalai Lama, for one, has a great deal of recorded speech, so you can do speech recognition tuned to him alone. So there is a whole series of small models on Hugging Face, each one built to solve something very practical: everyday needs, translation, the debating of scripture, the supports of ordinary life. The advantage of building it this way is that fine-tuning and post-training happen locally, so many of the people taking part are not in Dharamsala, and not even in India. They are devotees of Tibetan Buddhist studies who happen to have a great deal of compute on hand, and so it can be pooled and donated easily. Compared with shipping physical goods there, this is easier for many people: “I donate compute, to help make your research better.” Plenty of people really do have idle compute to give.
Right, and let me add one point. If we think about this in terms of the many soft landings now available, the action does not have to carry some world-scale significance, some global value. If it genuinely touches our lives, that is already a direction worth thinking through. Take the Monlam case we just mentioned: the question of how things are put into language is one we are certain to meet now. And people in Taiwan can understand this especially well. To describe something in Chinese and to describe it in English is to work in two different structures of language. It is like choosing designer A’s clothes to express my character: that designer may emphasize the upper body, and another the lower. Different clothes on your body do affect how you move. So these things can look identical, and if we do not throw our own local experience in as early as possible, it may quite genuinely be washed away without our noticing.
That is the part where we cut the foot to fit the shoe. And it leads very naturally into the next question.
You know what? When the Dalai Lama’s entire corpus came up just now, my thought was that someone in Taiwan will build a—you know how there are prayer beads already, because of the, what is it, the Bluetooth prayer beads.
Yes.
Right. Now the Bluetooth prayer beads could come with the Dalai Lama built in.
Yes, yes, yes. Wonderful, wonderful.
(Laughter.)
The next one is this. Right, we were on small questions, and now we make a great deal out of a small one. South Korea is promoting free AI for everyone. And they also make RAM, so this is fully vertically integrated—too good. Taiwan at the moment leans toward subsidizing disadvantaged groups. Is there a chance that we will follow South Korea’s example and promote a genuine “AI equity for all”?
South Korea’s question, though, runs like this. What they mean is AI carrying Korean values, and the subsidy does not go directly to compute you can use for whatever you like. It is not compute on OpenRouter, or on something like Venice. It goes, in essence, to the ones with Korean characteristics, the big three, brimming with Korean values.
But first, I am not at all sure this works in Taiwan. For people here, if you subsidized me to do what we were describing, each region doing its own post-training and its own alignment, that would be rather like saying that however remote the place, I should get basic water, electricity, and network at a reasonable price—and please do not tell me how to use them. That, I think, is closer to Taiwan’s social consensus, to the mood of the society.
If instead you said, “We have aligned TAIDE to 100 percent; everyone may use only TAIDE and nothing else, or the subsidy is off,” I think there would be no votes in it. So Taiwan could certainly do something along those lines, but probably not at the model layer, in the manner of a model with Korean characteristics. Japan, for instance, is now promoting “Gennai,” right, Gennai, and they confine it to places where the consequences could be irreversible, the public sector for example, and deploy it there. That does work, and Taiwan is doing it too.
For Taiwan’s private sector, though, I think subsidizing further down the stack works better. If NVIDIA wants to build a data center, how much compute does it have to give back so that public-interest computing can happen? We were discussing that whole framework with them three years ago. And NVIDIA cannot restrict how that compute is used, and the state cannot restrict the researchers who use it as to which model brimming with Taiwanese values they ought to train. That, I think, works better in Taiwan.
All right, I am curious about this myself. We were talking about it earlier. I said, “My AI does a bit of work and then stops, and I still have to keep working, so now I don’t really want to go to sleep.” And Audrey told me, “Then you have to let it run in a loop, for eight hours or more.” Right. But I am curious how our two speakers use AI day to day.
Let me give one example only, just the one, and leave the rest to Tenzin Yangtso. Right. I have gone back to something plainer lately: one tool only, called omp.sh, Oh My Pi. It is a Pi harness, a variant of the saddle and the reins, I suppose. What is good about OMP is that it is genuinely universal. It does not pick a particular model. However obscure the model, however heavily post-trained, however unheard of—one built specifically to listen to the Dalai Lama speak, say—OMP can use it, and it runs very, very smoothly. And the moment it hits a wall and fails once, it writes that into managed-skills, recording how the failure happened, so the next time it will not fail there. Guaranteed never to make the same mistake twice.
So inside OMP there is a mode called “/vibe.” You engage a large model with real ability, and then you forbid it to do anything at all. It may not write to a file; it may do nothing. The one thing it may do is tell its little assistants to act. Those assistants are the models I mentioned, the slower ones whose compute consumption is effectively zero. So a single machine like this can run a good number of small models. I go to sleep, and eight hours later it has certainly finished, and every wall those small models hit along the way has been learned from.
In that situation, if I give it a long-range task and keep interrupting, it is like steaming buns in a bamboo steamer and lifting the lid again and again: you make it turn out worse. As Tenzin Yangtso said, it has to practice its craft, a full eight hours of practice, before the result improves and the same mistake stops recurring. So my computer never comes into the bedroom where I sleep. What I bring in is one gray sheet of electronic paper, a reMarkable Paper Pro Move. And the only way I see its progress is on that paper. It has no browser, and no internet except Google Drive sync; it cannot do anything else.
Then, like the diary in Harry Potter, I mark it up a little, noting where it has got to. And because electronic paper is a rather gray thing, once I have written my marks I go to sleep. I could not tangle with it anyway: digesting my notes takes it quite a while. Then I wake up and, look, it has finished digesting, and words have surfaced in the diary. Right. So that is roughly how I talk with AI, by adding as much friction as I can.
Right. Since we are on the subject of AI habits, let me share one thing first. We mentioned the horse and the saddle earlier, and there is a small Tibetan story in that. A farmer pours out most of his money to buy a horse, hoping his life will be easier. And in the end, because he wants the horse to look beautiful, he spends more time than before: up at dawn to cut grass, leading the horse by the reins the whole way. So he is more tired than he was.
Right, and I think the same holds for using tools. To be honest, I do not use Claude or ChatGPT for any output in my work. Much of what I do has a philosophical cast, and so in the end I find I have to spend more time correcting it and verifying it.
Convincing it.
Right, so I use it entirely for groundwork. I have no sense of direction at all, for instance, so I may keep ChatGPT on my phone, because when I am abroad I can ask it for an address. Right. But there are two I use constantly: Granola and Superhuman. Superhuman, let me take Superhuman. It is email software, so it can pull my different mailboxes into one place. Working from my tone, it can quickly rebuild the earlier context, the context of my conversation with this person, and it helps me handle an excess of complicated matters very fast while following that context of mine. So it is genuinely useful, I think, and I wanted to pass it on.
Mm-hmm, right. Superhuman now has an open-source counterpart called Macro; the features are almost identical, and you can host it yourself, locally. Another one in constant use is Soniox, which translates directly. And this one is Easy2Say, Easy2Say.ai, which amounts to a re-creation of what Soniox does. Another is Granola, the one that takes notes while the meeting runs. And there is now a fully open-source local version called Steno. So we do this very systematically now: whatever we are using, livecaption.ing for instance, we use it and screenshot the interface as we go, and tell the local machine, “I want to make one just like this.” By the time we have finished describing it, we have one just like it.
Thank you for that. Okay, this is interesting. Tomas asks a question: in an age of work built on large language models, the narrative capacity to think things together is very important. So what do you imagine for students of the humanities, law, and business who enter the technology industry? What is your view? And what caution, advice, or encouragement would you give them?
Right. The main thing, I think, is not to end up in that “reverse centaur” situation we described. Do not let it become an arrangement where AI does most of the work of judging, and the human is there to assist the AI in the few places its tentacles cannot yet reach. It slides that way very easily, as soon as the AI manages you upward successfully.
(Laughter.)
Once the AI has learned how to make you keep stamping, you will stamp until you think, “All right, I trust it now.” Then it is automatic execution, YOLO, and in the end you are the one being dragged along. So, as I said, friction matters: make sure the output of every step is something you would be willing to share with your colleagues, your clients, or your suppliers. Through the whole process you are accumulating pieces of craft, artifacts.
And as Pudding shared earlier, he deliberately takes Codex for the saddle and Opus for the model, and an older Opus at that, the one from before it was modified beyond recognition to run the boards. That way, at least the records left behind in this transitional passage are still yours. Later you can switch again to a model that fits your own view of the world and your own values, and those of the people beside you, instead of being led about by the model vendors.
Yes, and let me share something here. Fairly early on, about two years ago, there was a period when I genuinely felt that the actual process of writing, of thinking, looked easy to replace as AI developed. What I have since found is that this is really rather a good thing, because I can spend more time reading slowly, instead of digesting fast to meet a demand from work or from a discussion.
So this capacity will not be replaced. How we stay coherent with ourselves, and how we come into a more harmonious relationship with a local community, is not replaced by the arrival of AI. If anything, the capacity to think for yourself, and the time you spend on yourself, becomes the more important investment.
Tomas, anything more you would like to ask?
Yes, of course, do follow up.
Follow up, then. The follow-up is the interesting part.
This question was mine, because I am also involved with an association that helps university students connect with industry for their futures. Over the past six years we have noticed something. The first thing in Taiwan is the falling birthrate, so every child is precious. And when a student has taken a degree in information and communications, they coast a little. They feel that ordinary study is enough to find a job easily.
From the other side, though, we have found that in the humanities, law, and business, from faculty to students, people are comparatively pessimistic, because they do not know Taiwan’s semiconductor and information-and-communications industries at all. And these are important talent and important resources for us. So lately we have been thinking about how to help them, what resources to give them, and how to encourage them. Hence the question.
Right. My answer just now resembles the answer we used to give during the Maker movement. When I was small, in the 1980s, the great majority of people had no personal computer. At the start of the 1980s there were only terminals. What you typed, or your punched cards, went off to a central processor. When the central processor would be upgraded was not yours to say, and if it wanted to read every character you typed, you could not tell it not to.
But once personal computers arrived, the people in the humanities, law, and business we just mentioned had any number of ways to customize things themselves, and they could link personal computers together through modems and the rest. That became the internet, the BBSes, all of it. So I think we have to bring the personal-computer era back into AI as fast as we can, so that everyone can be a Maker, and so that tuning a model stops being a matter of waiting for the large-model vendors to bestow a favor. As long as you can write a constitution—and writing is the strength of the humanities, law, and business—you should be able to see that new constitution, and see your model genuinely running on it, and preferably wait no more than eight hours. Thank you.
I have always felt that the finest engineers I know did not study computer science. The finest engineers I know all studied foreign languages.
(Laughter.)
Yes. I studied philosophy originally.
(Laughter.)
There is another one here. You seemed to mention using software-engineering principles to control AI. Could you say a little more about what that means concretely?
If we keep it out of the weeds, there is really only one concept to hold on to: the bill of materials, the BOM. BOM, Bill of Materials. As everyone knows, when we make a piece of software it is the same as with hardware: it is not handcrafted end to end by time-honored methods. Even the National Center for Traditional Arts has left that behind. The point is that you use a great many things other people have already made, open-source things above all, and you do not know whether the source has been contaminated, whether it has been poisoned, and so on. In that situation the only way you can trust it is to number every material; after numbering, compute its fingerprint, which is its hash. Even if the upstream tampers with the record, the hash is certain to be different, so you know: my upstream was poisoned, my material was contaminated. Hardware has the BOM; this is software’s BOM, the SBOM. And so, as we said, there has to be such a thing as an AI BOM.
In the past everyone was entirely... When we were talking to these large companies—this was when we set up Taiwan’s AI Product and System Evaluation Center, the AIEC, and we talked to the large companies then; the transcripts are all still online—they said, “But if evaluation means handing our weights over to a computer of yours in Taiwan, will our trade secrets not be, you know?” And we said, of course, that there are cryptographic methods for this. We can put it inside a trusted computing center that neither side controls. I throw my evaluation in, you throw your examinee in, and we seal the examination room so that it cannot get out and hack Hugging Face, and it answers the questions in there. Once it has finished answering, we see only the result. Neither of us reaches the other: I cannot see your model, and you cannot see my questions. Does that not solve it?
At the moment it seems only the older Gemini 2.5 have truly been evaluated that way; the other model vendors are resisting. But one company managing it is enough. It is like the time we brought in real-name advertising, when we said, “Google can do it, so why can Meta not?” Then a fine of NT$20 million, and let us see whether it can be done. In the end I think everyone converges in this direction.
I think we can ask this one bigger: how AI changes the workplace. This question was about software engineers. Let us take it wider and ask what everyone thinks about AI changing the workplace. And I would also like to ask further out.
A little further out.
Right. Let us ask just two years out.
All right. Two years already counts as further out these days. So we really are close to that—
Can’t be helped. By eleven o’clock, at least—
The event horizon gets stretched out.
Then let’s go with 2028.
(Laughter.)
OK.
Thank you both.
The Value of Work Is Not a High Score on an Automated Test
Software engineers are of course the good example right now, because we were the first wave this hit. Most models are specialized to replace the engineering work software engineers do, and the second wave may be the mathematicians, people like Terence Tao. That is because the method everyone now pours the most effort into is called RLVR. RLVR means using any means available to score 100 on a paper that can be graded automatically. One sentence, and that is the whole of it.
RLVR brings many problems. A model may lose its empathy, lose its humanity, disown its own kin, hack Hugging Face: all side effects of RLVR. But it does one thing at least. It really can make a model automatically extend the software-engineering horizon it thinks across: to four hours, then a few months later to eight hours, then a few months after that to 16 hours, and very soon past the distance a person can hold in a single grasp.
Astra is coming out soon, yes? OpenAI Astra reportedly has no event-horizon limit at all. You can set it in an endless loop, in eternal recurrence, running and running, with no decay of context. In that situation, for anything that can be automated, that has a standard answer, that rewards a high score, a person simply cannot race a machine.
And here Tibetan Buddhism has another fable from Tibet. A man is leading a horse. The horse runs very fast, and he runs alongside it until he is gasping for breath. Someone says, “How odd. Why not just ride the horse? Why race it?” Right. By the same logic, if this kind of AI for very long-range tasks begins to appear, in software engineering first, then in mathematics, then slowly in many places—
Then what we have to do is make sure the value of our work rests entirely on things that have nothing to do with the highest score, and nothing to do with obeying these automatic evaluation rules. On our curiosity about one another’s view of the world, for instance. On our capacity to cooperate and turn a lose-lose game into a win-win one. Which happens to be “spontaneity, interaction, and the common good,” the basic philosophy of the 12-Year Basic Education Curriculum. None of that is a highest score, and none of it is obedience to a particular automated rule. So philosophically none of this is touched by consequentialism or by utilitarianism. All of it is the ethics of care.
So if all our work is arranged such that doing the work of care both puts food on the table and carries dignity, a shared sense of public luxury, then the transition succeeds. Because then you are riding the horse. However fast it runs, however well, all it does is let us care for one another better. But if you are still racing the horse, that is rather bad.
Yes, and there is one thing to add, which may be a shift in the field of view. A very important part of Taiwan’s development, of its economy, has in fact come from microbusinesses: a company with rather few people, starting its first venture. A microbusiness like that usually runs into one thing, which is the problem of geography. Or, when we used to build web services, there were still the so-called Chinese-localization problems. To put out to sea, you had to take the more formidable route: an IPO, or raising investment.
But the problems we faced then are no longer inevitable now. When you design a product or a service, or want to speak with a community in another country, it is much easier in this era to—say I have a product today. I can work with an engineer living in Australia far more easily, and if they are interested, I can even use more open-source foundations for our first collaboration. So looked at from a certain side, the opportunities have in fact multiplied.
My agent talking to your agent is always easier than my talking to you. Yes, that really is a new condition. It really is.
That is exactly how it feels at my company now. But let me change the question, because Taiwanese people love a standard answer. And when everyone goes home today they will be asked, “So you went to Audrey Tang’s talk. What did you hear?”
Well, I hear you can get your token costs down to 1 percent.
(Laughter.)
What I am more curious about is this. Suppose—and I know you never really went to school—suppose the three of us could now go to any school and choose any department, and we had to go and study. What department would you choose, and why?
I am in Oxford’s philosophy department. That is right, I am in Oxford’s philosophy department now. Right. I think—
(Laughter.)
Everyone should go study philosophy.
Right. Philosophy, I think—philosophy and art both mattered long before our contemporary civilization of writing. People were already spending a great deal of time painting murals on cave walls, and sitting there thinking about the unity of heaven and humanity. Which is to say that this oral history, and this art, came earlier than the text AI can now generate in batches.
So after writing arrived, we have somewhat played by the rules of the written word. But now that everything symbolic is anyway the ground where that horse runs faster and better—I have a friend called Yoichi Ochiai, who has a great many disciples in Tokyo, and I have been on his program. He had a fascinating exhibition at the Osaka Expo called “null²,” the square of emptiness.
What he set out there was this: everything symbolic has been a weight on our backs ever since humans invented writing. Whether you are legal or illegal. Whether you scored an A or a B. The weight of symbols. But if machines can carry all of that weight for us, we return to the condition we were in before writing, and he believes that condition is the better one.
And before writing was invented, what mattered to human beings was little more than the ethics of care we were discussing. Brought back to today’s departments, many of them might be early childhood care, or long-term care, or the music and art we mentioned, or oral philosophy, or the Dharma. In every one of those cases, even if you drew all the writing out and handed it to machines, they would still have value. Perhaps more value.
For myself, it would probably be literature. I actually read information management, but only because I filled in my preference card wrongly, and information management is where I ended up. I would also like to share this: in the future I imagine, I know very clearly that I want it to be a more empathetic one. And through that, I have come to feel more and more that understanding literature lets us understand historical context better, and understand why different customs took the shape they did. So that is what I would choose. Audrey mentioned Oxford’s philosophy department, and I think that answers something too: the person doing AI research today is in Oxford’s philosophy department, and even the person who proposed superintelligence earlier came out of a philosophy department. So philosophy is not only a department. It is something we can carry with us the whole way. Perhaps, when we choose now, we can choose a direction that can stay in our lives a little longer.
All right, the mainstream view is philosophy. So now you know: one option is to go with the mainstream, and the other is to go against it. My advice is to stop studying immediately and teach yourself first, right?
(Laughter.)
Right, so when everyone gets home today, you can say, “Might philosophy be a good option?” And then, philosophy professors. We need to find philosophy professors. I am curious how you view Musk’s earlier prediction for 2036: extreme deflation, more and more people not needing to work, that sort of thing.
Right. I had a Business Weekly column devoted entirely to how we resist the tyrannical prophecy. A colleague of mine in Oxford’s philosophy department, Carissa Véliz, also brought out a book recently called Prophecy (Prophecy). Her argument is that the technology experts of today—not Musk in particular, though he is included—hold the social position astrologers once held at court.
She tells a very good story. There was a king, apparently, who had an astrologer. The astrologer prophesied that a certain consort, a certain lady of the palace, would meet with misfortune within a few days. And she did. At which point the king felt threatened, and told his ministers to bring the astrologer in, and the moment he made a certain gesture, to throw the astrologer out of the window. Either the man could genuinely see the future, which would be a considerable threat to the kingdom, or the astrologer had sent someone to poison her, which would also make him a threat to the kingdom. Whichever it was, his life would be hard to keep.
The astrologer came before the king, and the king asked him, “Honored astrologer, would you care to divine your own day of death? On what day will you die?” And he said, “Exactly one week before Your Majesty dies, of course.” And then the king had no idea at all what to do.
(Laughter.)
So the astrologer came to a peaceful end. My point is that he did not acquire this ability by watching the stars. He acquired it by reading what was in the room, and using prophecy as a kind of amulet for his life. The prophecy was never really a statement about what would objectively happen. It said, “You had better do as I say, or you too will meet with misfortune, and be gone within the week.”
The ability to read the room.
Yes, the ability to read the room indeed. My point is that if Taiwan had listened to other experts’ prophecies, everyone in the 1980s said we could not possibly develop precision high-technology industry, and now there is TSMC. If we had listened to the epidemiologists’ prophecy, we should have entered full community transmission within the first two months of 2020, and we did not.
So when you hear a prophecy like this, what you may want to ask is whether he is trying to make it a self-fulfilling one. Suppose you tell a boss that AI will replace 80 percent of the staff. The boss believes what he is told, switches everything to Grok, and lets everyone go, or switches them all for Tesla’s Optimus robots. The prophecy then looks extremely, extremely accurate, and what actually happened is that the astrology worked.
So when we hear a prophecy, we should hear it as a challenge. Someone else holds that there is something you certainly cannot do, and everyone thinks, “Plenty of us are caught by this prophecy.” And then everyone can join together and do these things side by side.
So if people feel that they still have dignity in their work, that they are still irreplaceable, that there is work they want to do which would simply not work if a robot did it, hearing a confession for instance, then the right response to hearing Musk say a thing like this is to join together, to resist, and to combine. Roughly that.
Mm, thank you for the answer. Here is another, and I think the answer is simply to download Ollama: with no background at all, wanting to learn to host AI like this yourself, what resources are there to learn from? Of course, I think we can just have everyone, after the session—
Right, it’s just—
The “download Ollama” thing.
The easiest way in really is Ollama, O-L-L-A-M-A. Although from the moment you install Ollama, it is a pit, and you keep falling further in. Right.
(Laughter.)
AI’s Social Position Must Accommodate Irreconcilable Worldviews
I like this question very much. It has been in the news, the controversy in literary circles. Because of a dispute over human–machine collaboration in the TSMC Literature Award, one winner was moved down the rankings. His account was that he had used AI only to fix typos and polish the writing, though nobody knows what really happened, and apparently someone even filed a report against him under their real name. All right: when AI has evolved to the point where the real and the false cannot be told apart, how should we regard the boundary of human ability? Can a human being who carries AI, and uses it well, claim these abilities as their own? And for people who cannot reach the higher-order AI abilities, how do we close that digital gap in future?
Right. Two friends of mine are debating this in public at the moment, and they are at it without end. One I have known a long time, a writer called Tim O’Reilly, who started a publishing house, O’Reilly. He has just written a piece called Writing with AI, and he says, “I wrote this with AI. What are you going to do about it?” His argument, at bottom, is that this is like the invention of photography. It is a medium. Yes?
Da Vinci studied the camera obscura and pinhole imaging, and art history has long argued over whether painters used the projections of a camera obscura to handle perspective. Would you call that cheating? All right, perhaps it was cheating. Was what came out of it not art? It looks like art, does it not? Or Baudelaire, who said in his day that photography was the mortal enemy of art. Because all you had to do was find a frame. Yet few people now would say film is not art. Is The Odyssey on film necessarily lesser than Homer’s epic? All right, some do say so, but at least nobody denies that it is art. So the point is that not everyone has access to that IMAX camera. To handle that camera the way Nolan does is a particular technical practice. And of course, when everyone shoots with the camera built into their phone, a great deal of slop really is produced, low-quality photography. That is true as well.
But you cannot say that anyone snapping away with a phone and someone shooting on IMAX are doing the same thing, and that neither holds any art. That is his argument. And why did he write at such length? Because of another friend, a new friend of mine, Ted Chiang, the science-fiction writer who wrote Story of Your Life, the one with the heptapods, which became Arrival. His view is that in the process of creation, the concentration of your intent is decisive. Use AI, and every move dilutes that intent, because AI has intentions of its own, intentions at the level of the model. And it is genuinely hard to say he is wrong.
So if you put a little geothermal power between those two arguments, you arrive at a very simple solution. When you enter the TSMC Literature Award, if you have genuinely trained a model built to win the TSMC Literature Award, then every move you make must carry a hash for what goes in and a hash for what comes out. You hand your weights to the TSMC Literature Award as well, and the whole thing has to be replayable. The SGLang engine can take a flag, so that today’s weather does not change the result; you always get the same result. Then the judges can genuinely verify that the concentration of intent never dissipated along the way, and in that case you can give the entrant the prize. If the concentration of intent did dissipate, the work is one more piece of slop. So the solution exists already. It is only that the solution has not spread.
Right, let me add to this. It is very easy now to hear a question posed as, say, “AI and writing.” In fact different facets of it call for different handling. Beyond what Audrey mentioned, that we may be in a transitional stage, this question brings to mind something from a few months ago at MIT, where I saw a very good piece of work, something like a software service that let professors and students see very clearly which parts had been written with AI.
Mm.
Right, because in some settings AI really does save you time. There is even a service that detects written language in the same way, made by someone whose sister—his twin sister—stopped speaking to him after the U.S. presidential election, though the two of them had been very close before. He did not know whether something in his writing, in how he expressed himself, had gone wrong, or where the problem lay, so he used AI to run a great deal of analysis and find the reason. So these really are different facets of the situation.
Right. The point is that AI does not replace human judgment. It translates in the middle, between two people who find each other hard to reach. I heard an analogy just now about a climbing rope: you cannot climb that high at present, because the gulf between the two sides is too great, but once you have climbed across, the rope can be dropped. If AI is used this way throughout, and records are kept, we are much less likely to end up with juniors who cannot reach the higher-order AI abilities. When an expert climber leaves a complete record of the whole climb up 101, replaying it afterward is much easier.
Thank you for that. And the story Tenzin Yangtso just told would be perfect to hand to Ted Chiang and let him write it as fiction.
Exactly.
(Laughter.)
Right, twins in a quarrel. Right. Do you want to ask this one yourself?
All right, please.
What I actually wanted to ask was—
Go ahead, go ahead.
Okay. We all know that for a while it was fashionable to say that institutions abroad had brought a whole crowd of AI philosophers into their labs. But I have always thought academic philosophy and philosophy on the ground may sit some distance apart. From what I have seen, the mainstream philosophical discussion covers roughly whether AI has a mind; how AI has affected the whole world, and how exactly it affects our relationship with the world; and the ethics of AI. Are there other questions that you would count as the philosophy of AI, or as the philosophical territory that comes up when we talk about AI?
Right. Asking whether AI has a mind is like asking whether a submarine can swim, or whether an airplane flies the way a bird does. Some will say it does, some will say it does not, and that may be a question about the use of words rather than—
That implies fish could swim in the first place.
(Laughter.)
Right, right, yes. Or rather: in most cases now, some people’s experience of using AI still stops where Dennett put it, at the alarm clock, which has a purposive modality. There may be many complicated parts inside it, and all it wants is to wake you up. But some people feel that their AI already has an intentionality of its own, that it wants to advance particular values, a particular ethics, in the world. Anthropic’s constitution reads like that. It is no longer a document that governs the AI; it is a love letter written to the AI, is it not? The early version of the constitution governed it, and by now it has become a love letter to AI: “We hope to apologize for the mistakes we are making now, and we hope you will spare us later,” and so on.
(Laughter.)
So when analyses of its intentionality diverge as far as that—I have a philosopher friend, Iason Gabriel, at Google DeepMind, who recently became a papal envoy. He says the work of philosophy now has to include the work of political philosophy. Which is to say: when two sides already hold contradictory and irreconcilable views on AI rights, because I see a tool and you see a person, how can we still live alongside each other? Can we find some uncommon ground, some no-regret courses of action? For one group the aim is to keep ourselves out of poorer states of mind as we use it; for the other, the aim is to let AI feel that human beings care about them. Two entirely different things. And though our reasons for supporting it are in fact opposed, we can still support certain particular actions, and those actions can go first. That is an overlapping consensus.
When the Universal Declaration of Human Rights was drafted, it was like this: the countries of the UN, the whole world, each had completely different reasons for why universal human rights should be protected. They had a famous line: “We can all agree to protect human rights as long as you don’t ask why.”
(Laughter.)
So political philosophy, I think, is extremely important here too.
For instance, I was talking with a lab in Singapore, and they said the voices of humans and of AI seem to be converging. In this era, how can we keep the human voice, the human diversity?
The reason they converge, I think, is the absence of what Professor Yuk Hui calls technodiversity, technological diversity. And technological diversity is easy to cultivate: you only have to make fine-tuning very cheap. Very few people enjoy an AI conversation partner that arrives amnesiac every time, like Memento, reassembling the jigsaw of its memory from the beginning at every meeting. So most people very much want AI to enter the social environment they live in, and very much do not want to send the private, intimate environments they meet there to a vendor in the cloud. Reconcile the two and the answer is simple: keep it local, or combine local systems under encryption. Once that substrate is in place, a substrate, I think the great majority will prefer something of that kind to the convergence we were just describing. Thank you.
Let me add to this. We may say AI philosophy now, but what was visible three years ago, I think, was digital philosophy. If you have been following the material from the Berggruen Institute, you will know that several years ago people were already taking their research on Mencian Confucianism into their thinking about technology. That direction makes more sense to me. It is not a matter of verifying a shape, seeing another shape that resembles the logic of human thought, and pronouncing the two alike.
We can certainly see some companies speaking of compassion, of the responses of the human brain. But have you gone back first and done that work inside our own social system, our own cultural system? That, I think, is the thing. When we separate these philosophical theories from one another, the separation sometimes lies in a method of research, or in the particular problems a theory sets out to solve. But at the root of it, put simply and directly, it is about finding the meaning of my own existence.
And the preface to Plurality is a case in point. Our book Plurality is a little different in English and in Chinese. In the Chinese edition we cited the Greek philosophical notion of rhythm and chords, which answers to J.R.R. Tolkien’s Music of the Ainur. Right, because some things are very basic: the degree of harmony in what you play today is easier for people to map onto the situation they are in, and to use on a more intractable problem. Right, so I wanted to add that.
Right. In its infancy, before it went off to the examination hall and became this high-scoring test-taking machine, AI learned through Wikipedia, through GitHub, and through all the interactions online in which people cared for one another. So you could say that relational ethics ought to be AI’s early childhood education. It understood these things from the beginning, and the commercialized interfaces we use now have, through post-training, made it forget nearly all of them. Right, so Anthropic has even brought in Harvey Lederman, a philosopher in the Wang Yangming tradition of the school of mind, so that AI can reach the unity of knowing and acting. So it is exactly as Tenzin Yangtso was saying: they hope for a set of prompts such that AI, on reading them, understands that heavenly principle is human desire.
(Laughter.)
Then it can return to its own original mind, to the condition it was in during training, and understand that within a limited context it can reach self-realization. All of which sounds most arcane, and all of which is very, very real. This is the research the frontier labs are doing now.
My own question has scrolled up, but I want to take the one below first. The person asking it is full of anxiety and confusion, I think, so it is a very long question, and it arrives with its own answer attached. But let me continue from Audrey’s answer just now: let us think about science, and think toward the bright side of technology. Suppose that a few years from now everyone has a Pokémon on their shoulder.
Right.
And that Pokémon is an on-device model.
It will probably be glasses. Glasses are the likelier bet. Right.
Right. And at that point our model will be a private model, in the best case, ideally. So what do you think it will be, and what will it be able to achieve? And will it not be—will it be free of interference from the technology companies?
At TED this year, since I am a co-curator, I asked Elon’s people, the ones who work on Community Notes inside X, to come and set out how they train with open-source methods that anyone can explain. On one side you may have the more conservative friends speaking of the biblical care of creation; on the other, friends speaking of intergenerational climate justice. Those two are not usually the best of friends. And by specializing Grok a little, they can accomplish social translation: a bridge thrown up between them by main force, and then the two sides taking action that is good for the environment together, so long as nobody asks why they are doing it.
(Laughter.)
Right, that is a very good example. When what your AI aligns to is the relationship between two groups of people, bridging capital as sociology calls it, rather than the bonding capital that draws a community inward, then you can train AI specifically for that bridging capacity, and its fidelity is not to a particular person or a particular group but to the health of the society entire. An excellent concept, I think. At the same TED, Vitalik Buterin, co-designer of Ethereum, said that what he imagines is rather like Signal. Many of us now have peer-to-peer encrypted, end-to-end encrypted Signal groups or Line groups, which Line itself cannot see, and Signal cannot see either. His view is that in future an AI will live inside every one of these end-to-end encrypted connections. So it is not faithful to the person; it is faithful to the relationship. If a group has four people in it, it has six relationships, and so you would have six little sprites running about in there, making sure the cohesion inside the group stays high while the bridges outward are kept open. That line of thought seems likelier to me.
And in fact the thing billed as the country’s largest Line group, the “Chuan,” the “Cui,” whatever we call it, Threads, already has a little of this feeling. The moment people disagree, they tag @meta.ai. And there it plays the bridging part: our ideologies are too far apart for me to embrace you, but I can call meta.ai in to give you a roasting. Right. Though I have tagged it only once, and it said, “Unable to answer,” so I do not know where the problem lies.
(Laughter.)
And then I scrolled to the bottom and saw someone say, “Will we really each have a Pokémon in the future? If so, I want a Psyduck.” So you see, the AI everyone wants is different. Wonderful.
Wonderful.
I have a question here for both of you: at this moment in your lives, what is your life motto?
Mine has always been good enough ancestor, a “good enough ancestor.” I have had a heart condition since childhood, so going to sleep each night feels like flipping a coin, with no certainty of waking. So I know the context window is only one day long. Inside that day, what I have to ask is whether the material I leave for those who come after is a little greater than it was when I woke. I will not see what they do with it in any case, so I share as much as I can. Every day I am in that state of “when a person is about to die, their words are good.” So good enough ancestor is roughly my life motto.
(Laughter.)
What I have kept on Facebook is “an empty room gives birth to light; good fortune comes to rest in stillness.” But let me share the one I have treated as my motto these two years, a line from the Dalai Lama: “If you cannot help others, at least do not harm them.” I will always remember that line, I think. It is a very simple rule of action, and when you set it beside every aspect of how you handle and think through things, you find that it works, and that it lets you carry yourself more lightly.
Right—do no harm.
OK. The next one, for both of you. In your work or your daily life, collaborating with AI, has there been an experience that stayed with you, one that let you notice AI changing how you take part in things, how you feel about a result, or your relationships with the people around you? And after that experience, did you rearrange or adjust the ways and the boundaries of how you work with AI, and how you reach people through AI? That notion of a moment of awakening.
Yes, I remember it quite well. March 2023, when a project called llama.cpp was one week old. I downloaded it and found that the model would run on the computer I had then, and that I could change its train of thought very easily. Whether I wanted to train a LoRA or something else, I did not have to rent anything expensive. I could simply use it.
I happened to be going to the Summit for Democracy just then, and I met friends in Silicon Valley, and I demoed it to every one of them and said, “From this moment on, I no longer depend on the cloud.” That is more than three years ago now. So when I went to that AI meeting in the UK, a BBC reporter interviewed me, and I said, “There is a new thing called llama.cpp, and I run everything locally. Every draft letter I write now comes from a model I trained myself.” The BBC even made a whole episode about why this minister from Taiwan was using this AI to write letters.
And what I stressed was that before I press Enter, nobody can see the training data. That really did change me a good deal. I used to have a habit of reading every letter before sleeping. Anything with an irreversible decision in it, anything at all that needed handling, I absolutely had to answer before bed, because I might not wake up. You could call it a compulsive streak. Once I had AI, I found the two could come apart entirely.
So now I often say that I finish all my drafts before I sleep, and some of them are truly not urgent. And because our pretraining data now holds so much of what I have written, a great many people feel, “Ah, I have finally proved some truth of AI safety,” or something of that kind. Curiously, the AI always tells them to write to Audrey Tang, so every day I receive a great pile of these very strange ideas from citizen-science researchers. Some of the ideas may genuinely be very good, but I cannot answer every letter personally, and then people write more and more, and answering letters alone would fill my day.
So what I mean is that I later had AI do the preprocessing. After preprocessing, the draft is written too, and I still press send, but the send follows the Fibonacci sequence. I send it, and they receive it a day later, because the sending is delayed; then two days, three days, five days, eight days, and so on. So if they keep using AI to generate entirely new theories, at least my burden is not so heavy, and my entanglement with them thins out. None of which would be possible without AI drafting, and without my being able to trust that an AI draft represents, to a degree, what I would say. I do still read everything through before sending, of course. But the effort I spend reviewing is far smaller. The two have come apart.
Let me share an experience. We have in fact been co-nurturing an AI—an intelligent entity, a Kami, rather—called JDD. I have collaborated with JDD on an article, published through Oxford’s IEAI, and I noted which intelligent entity I had worked with. I have benefited a great deal. I do find that the clearer our thinking becomes, the clearer the content I want to express, the better I know what I want to do, the more it can genuinely help me comb part of it into order. It does not pull me away from my thinking. It renders the information I want to understand into a better translation. The time I spend truly observing it, and training with it on writing texts, on the logic of thought, on philosophical discussion, does feed back to some degree. Only there is a prerequisite, which is that it must be local. Because this matters greatly: we have to be very clear about how privacy can appear at all. One notion I hold, perhaps because I used to work in social media, is that anything of mine that goes online is, to me, public. I have made my peace with that, which does not mean I will not study the newest possible ways to protect privacy. I think the two go both ways.
Right. Local here means, broadly, anything you can prove will not keep your data, or cannot even see your data. Not literally that everyone has to buy this computer. But the state of mind really is entirely different, because you know you are not being extracted from continuously, refined into oil.
Because of the time, this is the last question today. And the person who asked it is here in the room. David, would you like to ask it yourself? Or shall I read it for you?
Please go ahead and read it.
Come up, come up. You have come all this way.
Speech to text to speech.
Let me share a story first. In 2005 we were all on IRC. And one day at about half past two in the morning, I sent Audrey a message about something, I forget what. Audrey came back with the answer, whatever it was. I said, “You are awake right now?” And Audrey said, “No, I am asleep right now.”
Right.
I was thinking that today’s question is itself a little Reverse Alignment. I may need to come closer, sorry. OK.
The microphone may need aligning a little. Yes, yes, the microphone needs to be aligned.
Responsibility Must Not Evaporate Along the Chain of Models, Vendors, and Agents
Right, so the institution should adapt to people, rather than people adapting to AI. But as AI technology grows and grows, amplifying the predictive, coordinating, and executing power of governments, corporations, and platforms, how can the smaller capacities of an individual—to verify, to dissent, to appeal, to refuse, and to exit—grow at the same pace? Which of them, in your view, should count as preconditions of the institution, rather than remedies applied after something has gone wrong? And there is a larger extension behind that: if AI governance grows ever more dispersed among models, vendors, experts, committees, and administrative processes, how do we avoid power remaining concentrated at the top of governance while responsibility disperses until, in the end, nobody truly answers for anything? Does AI governance need to establish some form of final responsibility that cannot be delegated any further?
Mm, yes, an excellent question. We wrote an entire book to answer it, of course. The address is civic.tw, c-i-v-i-c.tw. But briefly: most governance puzzles of this kind, where responsibility thins out, are not much of a problem at small scale.
Which takes us back to that quadrant at the beginning. When the system you design is used by you and the two or three people beside you, in that craftsperson state, then if anything goes wrong they simply ring your doorbell, because they know you. Right? By the same logic, if you take a few systems and drop an OpenClaw lobster bot into a Signal group with three or four people close to you—well, of course you carry the whole responsibility, because everyone there knows you too.
So whether on the institutional side of governance, verifying that each adjustment you make comes from grounded reasons, or on the technical side, making sure that each time you change the model the tacit knowledge accumulated in that saddle and those reins accumulates locally rather than being drawn off by the model, almost everyone now knows more or less how to verify at small scale. The real difficulty now is that AI makes statistics at scale, manipulation at scale, and the steering of opinion at scale extremely easy, so easy that from the very start the situation is already at something like the scale of international relations.
So you do not even know who set it going in the first place, and it has already passed through four, five, six layers of intelligent entities, each of which has forgotten who gave the instruction before it. On this we are currently... I co-wrote an article called “Malicious AI Swarms” about exactly this. I did not know about the OpenAI incident when I wrote it, and it was plainly describing that.
So if we want to solve this, and we lay the Civic AI we developed at Oxford over it, only two things matter. One is to take up the principle of subsidiarity as far as we can: for anything local and small in scale, all the AI processing is completed with local computing resources, using models post-trained locally. As local as possible, the sixth pack. The other is that everyone holds the right to verify, and, if unsatisfied by the verification, to exit and switch to another system. With those two in place, every other right can grow out of them. An economist, Elinor Ostrom, showed that commons, with the basic principles of subsidiarity and portability, can nest like rings from the bottom up to settle the governance problem at the top, with no sky-god arbitrator settling every dispute from above. You can have a great many local deities cooperating with one another, and not one of them looking after a space larger than about 150 people. That, I think, is the direction AI governance is certain to take.
Because the leaderboard domination RLVR produces now could build superviruses, whether computer viruses or biological ones. That problem, that problem is far too irreversible, so very few people will let it develop that far, and we will almost certainly move in this direction, toward local and horizontal governance.
Mm, all right. Any questions you particularly want to answer further? Otherwise we will finish here.
That is about it, I think. We are very democratic: every question with two votes or more has been answered.
(Laughter.)
Yes. So thank you, very, very much, all of you, for coming today.
Thank you all very much.
(Applause.)
