Rob Reich

好的。非常感謝你，唐鳳。很高興你來到這裡。剛才的介紹有點簡短，所以我想確保你得到應有的敬意。

我認為，也有許多人認為，唐鳳是世界上最傑出的公民科技家之一，不只是因為她曾擔任臺灣的數位發展部部長，也因為從這裡的一些提案可以看出，她把前沿技術專家的核心技能，用在建設公民基礎設施與民主韌性，而不是用在某種私人倡議上推進技術本身，儘管那樣也很好。

所以我想在我們僅有的幾分鐘裡，談談你的一些提案。不過，鑑於這是一場題為「Confronting Our Future: Hopes and Fears」（直面我們的未來：希望與恐懼）的大會的開場對話，我想先歡迎你來到舊金山共識。

這裡的氛圍很奇怪，在座來自舊金山的人，多數都能作證。我相信你也知道，這裡的 AI 公司正處於一件堪稱世界歷史時刻的邊緣：讓沙子像人類一樣會思考，甚至可能比人類更會思考。

你知道基本的敘事。能力前沿正快速向前推進。 遞迴自我改進 正在開始。也許我們正處在 智慧爆炸 的邊緣。為前沿配速，是所謂實驗室，也就是模型開發者內部人士最近的努力。

再加上至少一部分恐懼，而我想聽聽你對這個世界歷史時刻的反應：這可能是前所未有、任何形式都沒有過的，智慧規模超越人類智慧，工作被取代近在眼前，也許導致永久的底層階級，災難性風險， Hugging Face 駭侵事件 ，失控、濫用，也許是生存風險。圖靈獎、諾貝爾獎得主 Geoffrey Hinton 說，也許我們有百分之十的機率，會在未來十年內死亡。事情很奇怪。

你傳奇地，我不想說得太廉價，樂觀，或至少你對這項技術抱持著比較開朗的態度。你想對舊金山共識的人說什麼？

再多談一點這個。Yoshua Bengio，另一位圖靈獎得主，我相信在座許多人都對這個名字很熟悉，最近呼籲在 後訓練 中完全放棄 強化學習 ，部分原因是你剛才描述的一些事，也就是 獎勵駭取 ，導致了一些失控行為。你認為這是個好主意嗎？我們應該在後訓練中放棄強化學習嗎？

是的。好的，太好了。那我們來談談其中一些提案，而正如你說的，這取決於「我們」是誰。

因為我和你的共同作者、耶魯的 Hélène Landemore 一樣，也是政治理論家，同樣關心民主價值、制度與理想，所以你不會意外，我在思想上對你的許多提案有很深的同感。但主持人的任務不只是稱讚並附和講者，所以我想對其中一些想法做壓力測試。

尤其是你與 Hélène Landemore 合寫的那項提案：一個隨機選出、由一千人組成的常設公民大會——

來自人類，來自地球各個角落。過去，我們從未在任何前沿科學發現或技術創新中，覺得有必要設立一個從全人類中抽選出來的常設公民大會。為什麼是現在？

是的。好。這正是我接下來想談的。隨機選任甚至稱不上是激進的想法。在民主國家裡，這是很尋常的做法。我們 透過某種抽籤程序選出陪審團 ，在不同層級的政府中採用隨機選任，在這方面是熟悉的做法。

但你說美國有五十個實驗室，五十個州，是經典的民主實驗室，也許可以再往上走。你針對人類整體的提案，我特別想問，尤其考量你為臺灣政府服務的經歷：我們生活在一個民族國家彼此處於某種競爭的世界。我們很容易就能談 AI 的地緣政治。你為什麼會建議從人類中抽籤？或者，你能想像一條政治路徑，讓民族國家真的同意這樣的事嗎？

是的。好的，剩下最後幾分鐘，我想利用你對臺灣的了解，再問一個地緣政治問題。臺灣當然處在 AI 發展堆疊中極其關鍵的位置，有 台積電 ，也就是製造晶片的廠商，那是 AI 革命的動力。許多人認為，競賽動態可能構成另一種地緣政治衝突的威脅，不再圍繞晶片本身，而是真正的冷戰或武力戰爭，如果結果是你來自的這個相對很小的地方，成為地緣政治角力的中心。

告訴我你如何評估實際爆發戰爭的機率，也告訴我你認為臺灣獨立的前景如何。給我你的看法，以及你想就臺灣的地緣政治複雜性傳達什麼。

好的。我們剛好到了時間的盡頭。請和我一起感謝唐鳳的參與。

All right. Thank you so much, Audrey. It’s great to have you here. The introduction was a little bit minimal there, so I want to make sure that you get the proper due.

I regard, I think many people regard Audrey as one of the world’s foremost civic technologists, both in the role she served in as the Minister of Digital Affairs in Taiwan, but also, as you can see from some of the proposals here, bringing a frontier technologist’s core skill set not to the purpose of trying to advance the technology in the form of some private initiative, as wonderful as that also may be, but for the chief purpose of building civic infrastructure and democratic resilience.

So I want to take up in the couple minutes we have here some of the proposals you have. But given the fact that this is our kickoff conversation at a conference entitled Confronting Our Future: Hopes and Fears , I wanted first to welcome you to the San Francisco Consensus.

Vibes are weird here, as most of the people here in the audience, if you’re from San Francisco, can attest to. I’m sure you know as well, the AI companies here are on the brink of something that is nothing short of a world historical moment in making sand think as well as human beings or perhaps better than human beings.

You know the basic narrative. The capability frontier is marching forward very quickly. The recursive self-improvement is beginning. Maybe we’re on the brink of an intelligence explosion . Pacing the frontier is the recent effort from people within the so-called labs, the model developers.

And then to add to at least some of the fear, and here I want to get your reaction to this world historical moment, possibly without any precedent of any kind, intelligence on a scale greater than human intelligence, job displacement on the horizon, maybe leading to a permanent underclass, catastrophic risk, the Hugging Face incident , loss of control, misuse, maybe existential risk. Turing Prize, Nobel Prize winner Geoffrey Hinton says maybe 10% chance we could die within the next 10 years. Things are weird.

You’re legendarily, I don’t want to make it cheap, but optimistic, or at least you have a more sunny disposition about the technology. What do you want to tell people from the San Francisco Consensus?

Just one more beat on this. Yoshua Bengio, another Turing Prize winner, a name I’m sure is familiar to many people here, has recently called for the complete abandonment of reinforcement learning in post-training , in part because of some of the things you just described about the reward hacking it has led to some of this loss of control behavior. Do you think that’s a good idea? We should abandon reinforcement learning in post-training?

Yeah. All right, excellent. So let’s take up some of the proposals, and as you said, it depends on who’s the we.

Since I’m, like your co-author Hélène Landemore at Yale, also a political theorist, also concerned with democratic values, institutions, and ideals, you won’t find it surprising at all that I have a great deal of sympathy intellectually with many of your proposals. But the task of a moderator is not simply to praise and agree with the speaker, so I want to pressure test some of the ideas here.

Especially with the proposal with Hélène Landemore, a randomly selected standing civic assembly of a thousand people—

From humanity, from all parts of the globe. We haven’t had in the past any frontier scientific discovery or technological innovation in which we found it somehow important to have a standing civic assembly drawn from all of humanity. Why now?

Yeah. Good. That’s where I wanted to go to next. Random selection is not even a kind of radical idea. It’s an ordinary one within democracies. We select juries through some type of lottery process , and the opportunity to have random selection in different levels of government is, in that respect, a familiar one.

But you say 50 labs in the United States, 50 states, the classic laboratories of democracy, maybe go higher. Your proposal there for humanity, I do want to, especially in light of your service to the government of Taiwan, we live in a world of nation-states that are in some form of competition with one another. We can talk about the geopolitics of AI very easily. What would recommend random sortition from humanity? Or maybe can you imagine a political pathway in which nation-states would ever agree to such a thing?

Yeah. All right, with the last couple minutes we’ve got, I want to take advantage of your Taiwanese knowledge and another geopolitics question. So Taiwan, of course, occupies this incredibly essential point within the AI development stack with TSMC , the manufacturer of the chips, the power of the AI revolution. Many people see the race dynamic as potentially threatening a different kind of geopolitical conflict, no longer around the chips themselves, but an actual cold or hard war if it should turn out that this relatively tiny place that you come from is the center of a geopolitical contest.

Tell me how you rate the chances of actual warfare, what you think the prospects of Taiwanese independence happen to be. Give me your take and what you want to communicate about the geopolitical complications of Taiwan.

All right. We’ve reached exactly the end of our time. Would you please join me in thanking Audrey Tang for joining us?