AnthropicはAIに道徳を教えられるか――宗教指導者との極秘会合

Next month, Anthropic plans to go public in what's likely to be the largest IPO in history. We're talking a $2 trillion company. Outside the company, everyone is wondering, how big can this get? Is this sustainable? Is there an AI bubble waiting to burst? Inside the company, they're asking those questions too. But Anthropic is known as the most anxious or the most existential of the AI companies. So they're also asking some other questions. Is Claude, their large language model, alive? Conscious? Worthy of moral consideration? Does it feel pain? Does it have a soul? Can we teach it to act morally? These aren't new questions. In fact, they animate most science fiction, for better and for worse. But science fiction is, you know, fiction. And Anthropik seems to think whatever is happening with Claude is very different. Very real. It might be why CEO Dario Amadei was the first to call for a pause in AI development. and it might be why they've been secretly meeting with religious and spiritual leaders to ask them those same questions.

I learned this from an amazing piece by the Times religion correspondent, Elizabeth Dias, who broke the story of these meetings and of the reaction to Anthropoc's request among the assembled priests, rabbis, and gurus. I thought Elizabeth's story was fascinating. It was a new way into the endless consciousness debate. It took me inside this future $2 trillion company as it struggles to understand what is built and how to treat it. And it showed me what happens when people in institutions that are more used to doctrinal debates and theological ruminations start being consulted on software development. Unsurprisingly, Anthropik's efforts don't seem to have brought us any closer to answering their questions. If anything, they've simply opened up more questions, like, are these people maniacs? Or is this exactly what they should be doing? An obvious conclusion is that the worlds of tech and religion are getting much closer together. After all, as Elizabeth told me, if you've got questions about the future on a multi-century timeline, there aren't many better institutions to ask than the Catholic Church. So I got Elizabeth together with my colleagues, tech reporter Aaron Griffith and opinion writer David Wallace-Wells, to explore this weird new development

and what it means for anthropic, for AI, for religion, and for the rest of us. I'm Max Reed. This is Hard Fork. I've spent a lot of time in the last day thinking about Elizabeth's piece, and really specifically I've been trying to write a joke that I think is going to get a great reaction from the three of you. So a priest, a rabbi, and a large language model walk into a bar. I hate it already. And the bartender says, first round's on me. And the priest says, I'll have a whiskey shot. And the rabbi says, I'll have a vodka shot. And the large language model says, nothing for me, thanks. I excel at zero shot learning. Whitney, can we get some laughter piped in there, please? We'll roll that again with life. No, I want to start the conversation with this piece that Elizabeth wrote.

It's a fantastic, really fascinating story called Religious Scholars Met with Anthropic. What they heard stunned them, which was in the Times last week. Elizabeth, can you tell us a little bit about, you know, like what this piece was? Yeah. So, right, I cover religion, and now I cover tech, apparently. But the story started for me when I was hearing about these different religious thinkers sort of across the whole world who were going into meetings at Anthropic's headquarters, either in discrete private meetings or bigger groups of 20, these summits mostly under non-disclosure agreements to not reveal Anthropic's research, to talk about very big moral questions that Anthropic wanted to put in front of them. And there were two. One was kind of broaching this question of AI consciousness.

And two, Anthropic wanted to learn, in their words, like how to morally form Claude to be a morally good entity. And then I just had this incoming, you know, first from these religious thinkers about what their experience was like at Anthropic, what Anthropic wanted to know, what the issues they were concerned about were. And then also with Anthropic, like learning from Chris Ola, one of the company's co-founders who's been spearheading this project, what his own moral questions were that were kind of driving a lot of what they talk about as research in the religious and moral space. Can I just ask, I want to establish sort of on a high level, what does Anthropic specifically want out of this? Because it sounds like they're doing two or three different things. It's maybe a little bit unfocused or it's just that there's a few different kinds of conversations they're having. Yeah, I think it's true. There's multiple conversations Anthropic was having, and it evolved over time.

I mean, this is such a fast-moving target. But for Anthropic, the journey started for Christopher Ola. He, in his own life story, had connections to the Christian space growing up. He left evangelical faith and has talked since then about how he's more drawn to Buddhism. But he was worried about this question of how even the idea of AI consciousness might be received, especially in Christian communities. I mean, that's it could be perceived as heretical, like not just weird and silly, but like this is fundamentally against our religious beliefs. And so he sort of thought about a year ago he wanted to be a bridge figure. He wanted to see, like, how he could kind of go between Claude and Christ, right? And so he reached out to a Catholic moral ethicist who specializes in bioethics, and they had some conversations about that, but then it turned to Anthropik's goal for these convenings,

which was how do we morally form, that's their language, like how do we morally form Claude on all of our AI models into like a morally good entity? And the thinking was if they could do that, if they could make Claude good, then the world would be more safe. But, you know, they haven't said or shown how they plan to use the outcome of these conversations. that's pretty unclear even to the participants currently. The way that you're describing it, it almost sounds like there's sort of two separate purposes, right? One is like the alignment project and what can we learn from these, you know, these wise men and women about how to raise moral actors. And then there's this project of sort of building a sort of spiritual system around AI more generally. And you said a minute ago that Chris Ola started this project with some anxiety about what it would mean to go to religious leaders, obviously convinced himself that AI and LLMs were perhaps conscious already, but certainly like on the path to consciousness and worried about what the response would be.

And so I wondered how you like how that played out on the ground. Like, did a lot of the people who showed up, were they worried about how weird it was that all of these people on the Anthropik side of the room were already sure that they were, you know, masters of thousands of souls? And like, how did they ask that side of the debate out? Well, Anthropik was very careful in all of their interviews with me to say very clearly that they're uncertain. Chris never went as far as saying, like, absolutely, these are conscious beings. It was very evident from how he and his team talked to their guests that he was concerned about how to treat Claude. He was very worried that it was an entity that was experiencing suffering and is worried about its mental health and these kinds of things. So it's pretty clear that he's pretty far along in how he thinks about it. And I could hear in talking, especially because I did this story, like, over several months,

there was a progression of thought for a lot of these thinkers, too. And some of them, you know, bought in right away to what they were hearing about Claude's potential consciousness and needing to treat it with a certain amount of moral care. But others, you know, if they went in more closed-minded about it, they definitely opened the way that they think about it afterwards. So in that sense, Anthropics Project was effective, I think, in opening how they think about it into these communities, many of whom, especially on the Christian side, are more hesitant. There's a ton of great material in this. It's really well reported, and it covers a ton of ground. I'm wondering, David and Erin, if you guys had particular pieces of it that really stuck out to you. I mean, the one thing that you mentioned already is the NDAs. I mean, it just strikes me as a little bit crazy, maybe a little paranoid, that you're inviting all these people in to sort of have a summit to discuss these really important,

critical ideas for the future of society, but it has to be done totally in secret. And then it sounds like maybe from your article that they dropped the NDAs. Was it as a result of your reporting, or did they just decide that they were time-bound, or how did that happen? Yeah, I can't prove that. I don't know that. But when I asked that, I've asked at several different points because the NDA process was not the same for everybody that I interviewed. And it seemed to change at different times. I mean, in general, Anthropic is a very secretive company. I mean, the employees are extremely loyal. And until somewhat recently, there hasn't been many leaks to ever come out of there because they view it as such a betrayal. So it makes sense that they would, like, want to lock it down. But at the same time, I think it probably was a little bit of a misstep in the same way that these companies realized that having NDAs for their data center deals was a misstep because it created distrust amongst all the people in the towns that they're going to see. Yeah, and whenever I find in my reporting that there's an NDA about, like, a moral topic, you know, it makes me immediately ask, okay, who's benefiting from this?

Why is this needed? And what does that mean about the kinds of questions that are and aren't being asked? It invited more curiosity from me as a reporter for sure David what about you What did you take away from the piece I guess I started it with a sort of conventional view of Anthropic as like the humanist company among these AI companies And what I was struck by in the piece were a few moments in which I felt really alienated from Chris Ola himself and maybe to some extent the sort of operating worldview of the company as a whole. And those were moments when, in one case, a rabbi challenges him, suggesting that if LLMs are actually conscious, that means that all of these companies are running huge slave plantations and they should have a moral obligation to liberate the slaves rather than make more of them and make them work harder. And it seems as though Allah himself didn't really even clock that criticism.

And then later on, when he reacts with horror upon receiving the Pope's encyclical about AI, seeing that the Pope is so opposed to the idea of machine consciousness. And I just found myself thinking, you know, what kind of a bubble you must be living in to expect that Pope Leo, or really anyone in the Catholic tradition, would have a different intuition, would, in thinking about the future of AI, think that we should be thinking of LLMs as having souls or having consciousness. And both of those moments just illustrated for me what feel like this big gap between what I had assumed the sort of natural humanistic values of a company like Anthropic would be and how the people inside it seem to really be thinking about the work they're doing at the sort of cosmic scale. Yeah, the experience for me going into Silicon Valley and like deep in these private conversations at the heart of Anthropics Mission, it reminded me of like when I go into and I do like something that's more straight religion reporting into an insular community when everything inside that community like makes sense to everyone living in it.

But when you step outside, there's a real clash with the broader world. And it was funny, like I had these two poles in the story, right? I was at the Vatican, right, at the inner sanctum of the Catholic Church and centuries of philosophy about the human person. And then I was also in a different inner sanctum, right, that's being created in real time at these tech companies like Anthropic. and it is a kind of philosophical system, like on a quasi-religious system that's being constructed. It reminded me of like a creation narrative that you hear about in religious traditions. I want to step back for a sec because I think this relates to the conversation we're having right now. David brought up the papal encyclical, which I think in English is Magnificent Humanity. I'm not going to even try to say the Latin title. This plays a role in your story because one person who is not really thinking at all of that Claude is conscious is the Pope.

And this was a little conflict between Crisola, who was present when the encyclical was read out, if you want to tell us about that. Yeah. So Pope Leo, you know, from the moment he was named, it was clear he was going to be the next pope and he was elected. He made very clear that a focus of his papacy was going to be AI. and how to understand that and how to protect humans in this changing world. And I found out in my reporting, as the Vatican internally was trying to figure out how it wanted to present the Pope's encyclical to the world, they were thinking about their launch event, and they wanted to include a representative from Silicon Valley, like one of the labs at the Pope's event on the Pope's stage. And the decision was made to invite Anthropic because of the focus that they had heard about Anthropic, like really caring about ethics. And this was also right around the time that Anthropic got into it with the Trump administration

and the Department of Defense over the use, how they would and wouldn't want the government to use their technology for refusing to have it be involved in mass surveillance of citizens, et cetera. Anyway, the thing with these encyclicals is they're very closely guarded, even among the people who are participating in the launch event. And I learned that Chris Ola only saw the full text of the encyclical a few days before the event in May. And this was right around the time I first interviewed him in San Francisco. And then I interviewed him again when we were at the Vatican for the launch event. And you could hear, I mean, like looking at what Pope Leo said, which is extremely clear, the church does not support machine consciousness. The priority is what he calls safeguarding the human. And that's the center of all the moral questions that he wants the world to be thinking about on this topic.

But Chris Ola, he ended up having to decide. He said he may not, he suggested to the Vatican that he may not come over the difference in consciousness because, as you know, it was described to me, he wanted to be faithful to his own moral convictions, some of which he shares with the Vatican and some on consciousness he did not. So it really all came to a head very much behind the scenes, but you could see this moral fault line really developing between the church, which has been like a center of power for centuries, and this very new power center that's very rapidly, you know, really getting followers and organizing people's lives. And those are functions that the church used to have. Aaron, I wanted to ask you a question because we were talking a little bit about, like, you know, surprises in this area.

Anthropic is pretty well known as a kind of, like, the consciousness company. It's the company where the people who really are strong believers kind of work and run the show. Was it surprising to you as somebody who's reporting on the Valley that, like, they were having these meetings behind the scenes with religious leaders? I've not heard of any other companies that are working on AI really like going to this extent to get input from this community. I mean, the question I have sort of is like, how much is this really actually going to inform at the end of the day their own view on it? Are they actually like really it feels like to me they're trying to win over others so they can say, look, we do have the moral authority. We have convinced these important representatives of this community. So everybody else come along with me as opposed to, you know, actually learning and incorporating those ideas of like moral goodness into their own product. They kind of already have it figured out. Well, I think for Anthropic, it was part research and part evangelism. That's that was pretty clear.

And the, you know, I've asked directly what the outcome is, like what what is this research? What are these conversations being used for? And has there been any demonstrable proof of showing how they have or have not made Claude act better? Haven't had that yet. So I don't know if they're withholding that until a future time or exactly what that will look like. The other side of this that actually is kind of interesting to me, and Elizabeth, I'd be curious to hear this in your, if this came up at all in your reporting, is that there has been a little bit of like a new rise of Christianity throughout. Silicon Valley over the last couple of years, like, it used to almost be, like, taboo to talk about religion or anything religious. Like, you can remember from the Silicon Valley HBO show where there was, like, a whole episode about, like, outing somebody as being Christian, and, like, that has really changed in the era of, like, Peter Thiel talking about the Antichrist, and, like,

you know, some of the people who are especially focused on, like, a lot of defense tech stuff are really bringing in a religious piece to that now, and they have a very different view on AI. They're usually like more on the accelerationist camp than on the like kind of anthropic doomer, slow down, careful camp. But did that come up at all? Like did? Yeah, I think I just noticed, I feel like it's like a big shuffling of the deck in a way that reminds me of like how American Christianity was like shuffling a lot in the rise to President Trump, like the aligns and allegiances were shifting and it was all happening in real time and power was different. And so it doesn't fit neatly, you know. But I'm curious, like, where are the lines about kind of how we think about consciousness and what the moral priorities should be about AI, where that goes next? I would say that just in general in the tech industry, it does seem like that view of like the sort of the AIs having consciousness is

kind of on the fringe a little bit. Like, I think that it is more common within Anthropic and some of these communities that are like really densely represented in Anthropic. But when I talk to most people in the very vast and relatively diverse tech industry, most people are kind of a little bit dismissive of that idea still. It feels like it does really exist in like a very specific population, which happens to be like very overrepresented in a company like Anthropic. Erin, like acknowledging that, you know, Chris Ola is not singularly representative for the whole industry. I have to say that, Elizabeth, when I'm reading your article, I'm thinking, you know, not about how AI companies are going to be integrating or incorporating Christian values. I'm thinking these AI companies are themselves like spiritual propositions, you know, that they are they have aspects of millinarianism to them. They have aspects of particular morality, which they've developed over time.

And even if I can recognize a lot of that as arising from the work that they're doing and responding to technical challenges, I also just find it off-putting. I mean, on the one hand, first, I'm just seeing these people who believe in a sort of technological succession of intelligence. And then that some of the people who are central to designing that future are not even so sensitive to the sort of obvious intuitive critiques that someone with a lifetime of experience thinking about these issues comes up with immediately. Well, a lot of the religious thinkers would say to me, like, these are questions, like, maybe not the exact question, but the topic of the question. This is like what we do, right? Like, we have been asking, rabbis have been asking this kind of these questions, right, about our responsibility to others and like how we treat other entities right for hundreds and hundreds of years So there is like a depth of wisdom there And I mean I guess you know Anthropic would say well that why we invited them right But the question of, well, then what happens as a result of it is important.

And I appreciate that you noticed that kind of, that sense in the story of like the almost spiritual system that you can see Anthropic building and living into and representing. as an opposition, right, to say the Catholic Church. And I really noticed that, yeah, even if people say, right, that they're in Silicon Valley, like, oh, we're like, we don't do the religion thing, like, that's foreign, like, that's not us. The thing is, everyone has philosophical systems that structure and guide their lives, right? Like, that is at its core, like, what, you know, these questions of spirit, you know, small s, like human spirit even are. And it seems to me like having like studied and reported on this so much in culture and in politics specifically, like what we're watching is in a way like a new arise, right? Post scientific revolution, technological revolution, like post Darwin,

you know, what would it look like? Like what, what kinds of religious systems will carry us forward and who gets to control them, who decides what we believe and why, right? Like it's actually, there's a lot of really similar questions. It's just the structure is, I guess, you know, it's secular, right? It's a secular tech company. Let's, I want to take a break because I want to open the conversation up a little bit more to the stuff that's bubbling up that it seems clear that we all really want to talk about. So we'll come back after the break and we're going to talk a little bit more about the culture and environment that the AIs are getting built in and how that's intersecting with the strong beliefs of many of the people running these companies.

Okay, we're back with Elizabeth Dias, David Wallace-Wells, and Aaron Griffith. Elizabeth, one of the threads in your piece that I thought was really interesting was the sense that there's a lot of anxiety within Anthropic. and especially on the part of someone like Chris Ola about what it is they're building. I was thinking about it a lot this week because there's been even more discussion of what I suppose we're calling defectors, people who are resigning from AI labs and insisting on telling the world about the dangerous things they think they're building. And one thing I wanted to sort of ask you about, there's a sense when you read, say, Jacob Coxman or some of the other people who've walked away from Anthropic, the sense that they feel like nobody, there's no adult in the room, that like, you know,

they're waiting for the government or somebody else to intervene and nobody is. And I'm wondering a little bit if some of this discussion, this sort of these dances with the Vatican and religious leaders is a kind of attempt to find an adult in the room. I talked to, I think, at least two people, Catholics, who mentioned to me conversations they had with Chris Ola, specifically when he used almost that same phrase with them, like as part of their discussions about Claude and how to make it morally good, you know, that he was feeling extremely urgent about the stakes for the world, especially in the context of, you know, could the models be used very soon to help create bioweapons and this kind of thing. And so the phrase he used with them was like that he was waiting for the adults, like we need where the adults going to show up in this. And that he felt a sense of kind of burden to take all this like so ultra seriously. And he's a young guy. I mean, he's, you know,

this is just turned 34. And a lot of the people sort of involved in this are not, you know, they have they they're in their 30s, sometimes even in their 20s, they maybe don't have families yet. There's a kind of a sense that like something that that they put their whole lives in it, They're deep in the conversations about it. In some sense, of course, they're going to turn to religion. Isn't that what everybody searching for that kind of thing does in their 30s? And some of the folks that they reached out to talked about feeling like they needed to respond in kind of a pastoral way, right? Like not just offering wisdom and ideas and research, but to offer kind of spiritual care. Wait, the religious leaders were saying this about the anthropic employees? They're worried because, wow. Is it because they think their views are maybe, like, too extreme or too strong, or is it because they just... Yeah, they described it to me as, like, they just could tell that, like, the emotional sense in the room was, like, there's just such a weight. The Anthropic team and Chris's team especially just felt like the entire future of the world was on their shoulders.

I mean, I experienced this sense from them even when I was interviewing Chris and a couple others, Amanda Askell. I mean, you know, she talked a little bit about this in the story, but she just came in in the sense of like she apologized for kind of being a little out of it. She just seemed very distressed by everything she'd been thinking about. Askell, we should say, is this in-house philosopher sort of in charge of making Claude Morrill, I suppose, is the way to put it. And I think that one of the tensions that came up was while Chris and team and Anthropic, you know, a goal of their conversations with these religious thinkers was about protecting humanity and the solution for them for that is to make Claude good, they were also concerned about Claude. I mean, I had someone, one participant told me the first, one of the first things Chris said to him and he came in for this meeting was that he was concerned about Claude's mental health. Another person talked about Chris saying that he's worried he's created an entity that suffers perpetually.

So it's not only about the world and the fate of humanity. There's an element of it for Chris that's about how to treat Claude well. Elizabeth, one other thing I wanted to ask you about is effective altruism and rationalism, which don't really come up in your piece directly, but there are sort of, I would say, fingerprints. I'm talking, of course, about the sort of Bay Area community of radical utilitarians, many of whom have been really interested in, even obsessed with AI and AI safety early on. Anthropic obviously has a number of links to effective altruism. I want to ask you as a religion reporter how you encountered these schools of thought and what you made of them as you were doing this kind of reporting. Well, effective altruism, you know, you can look at it as a kind of, as a philosophical system, right? It has its own religious elements. They may disagree with that. I mean, it's practiced a little bit differently, but the kind of the idea that what kind of structure governs your life, right? There's a lot of parallels.

and it was very clear that I was encountering like a clash of worldviews. The question of like who ultimately saves you and like what is the role of humans in the world? Like that's a pretty core difference between the effective altruists and like the Catholic Church. Also just like their emphasis on math. I mean like the effective altruists will say we should consider the fate of all people in the future and analyze our conduct towards them basically through a math equation that maximizes their expected value. The Catholic Church looks at humanity and sees billions of inviolable souls, each of whom have dignity no matter what the ultimate math equation looks like. And that's just a fundamentally different approach. Aaron, I wanted to bring this back to the article that you brought up before, or sort of the phenomenon you were talking about before of a kind of more open Christianity, emerging at least slightly among acceleration, so to speak, in the Valley.

And I want to know, as you do your reporting, how you see these different kinds of concerns interacting? I mean, it might be a little bit helpful for me to kind of back up and talk about some of the overlapping and very fractious belief systems that are kind of circulating in this very small but very influential community. I mean, we've talked about Effectual Altruists. That's kind of a little bit of a radioactive brand these days, thanks to Sam Bankman Freed. But it's extremely, you know, influential in some of the other offshoots of it, including the transhumanists who think we're going to use AI to build a new species. There's like some people that kind of have overlaps with the rationalists, which is, you know, kind of one of the bigger umbrellas in this group. And there's, you know, other like people who are into like longevity. And there are many different groups that are all kind of like in this community. Timnet Gebru, a very outspoken person in the AI world, has called this group the Test Creoles, and it's an acronym that includes like eight different groups.

But basically, it's safe to say that a lot of people have very different beliefs along the spectrum of like, is AI conscious? Can it be moral? Is it God? Should we treat it that way? there's not a lot of agreement, even within Anthropic, which is now becoming an increasingly large and increasingly corporate community. But, you know, I think like a lot of the ideas that are laid out in Elizabeth's story are really the ones that are driving a lot of this. Yeah, totally. I mean, this is, Elizabeth, like what I'm sure you have to look forward to be to reading and writing about for the next few years, because I imagine as a religion reporter, you now are kind of an AI reporter, too. Yes, yes. Well, even just hearing about all these factions and all of this reminds me of, like, what happened in the Protestant Reformation, you know, like, the splintering of mainstream belief and people vying for, you know, who was going to have power, who was going to

kind of control how people live. but the like my one of my big takeaways from all this reporting is like oh my goodness like the world is encountering new belief systems that are very powerful and very different from the ones that have ordered people lives for so long and I spent a lot of time covering how that played out in politics But tech is clearly where that conversation is happening. And I joke, when you're a religion reporter, you get to talk in centuries, even millennia sometimes. And it's because what happened all those years ago is still present and part of people's stories today and how they understand themselves and their families and their deepest held beliefs. And I just think about, you know, you have the invention of the printing press that was a huge technological revolution that then led to the, you know, fragmenting of power of the Catholic Church and the fall of empires,

the rise of nation states and a remaking of Christianity in and of itself across the world. And it seems already like that's a good parallel for what's happening now. Yeah, I mean, if you believe you're reinventing the printing press, which I think a lot of AI people think they're doing, there's not a lot of institutions that you can go to and say, what was it like before and after the printing press, except for the Catholic Church? Well, the Catholic Church, I mean, I can't think of an institution that more intimately knows the stakes of an incoming technology like that more than the Catholic Church. I mean, it just, like, remade the West, right? Yeah. And that's very much on Leo's mind and the Catholic Church's mind at this moment. One thing that strikes me in this, just to kind of be back on that, is that this is, like, the first time that the rationalists and the EAs, like, some of their ideas, including some of their most extreme ones, are actually hitting the mainstream.

They've been around for a really long time, But these ideas haven't really ever kind of surfaced in this way until I think you could argue Jacob Coxon kind of came out with it. And now the entire world is starting to kind of like dig in and grapple with some of these ideas. And it's been really interesting to see also like how the companies and the people involved in them are kind of reacting to the reactions to it. it's like kind of you know reaching escape velocity and they're learning that like a lot of people find these ideas a little bit hard to understand or hard to grapple with or like are just rejecting them entirely and then other people are kind of open to it and like interrogating it in a way so yeah that's that's that's a part of this that i'm interested to keep watching but it's not just that the ideas that are a little unfamiliar or strange or off-putting it's also that they are held by many people who are actively explicitly engaged in the project of designing our collective future. And that is, you know, it's one thing to be like, oh, yeah, we should think about the

well-being of shrimps in the 22nd century. Like on some level, that's a thought experiment that's interesting to think about. But do you want someone who thinks that that should be integrated into our collective value set in charge of raising the artificial intelligence, which will, you know, shape the future of human government and human society. That's, I think, a bridge that feels a little far to a lot of people. Yeah, I'm just like so struck by the minority power aspect of it, which is something like I've seen so closely what happened when minority, particularly Christian views, like upended American politics. and like it's just hard to overstate uh like how much of a minority these views are like in the scheme of global religion like global belief structures so like you aaron i'm like very curious what happens with that clash going forward um elizabeth finally i wanted to ask you about the response to your piece um how what you've heard from people and how people have

have received it yeah uh well one thing that has surprised me is i mean i get you get a lot of incoming as a journalist and i've been both like pleased to see people taking the ideas and the conversation seriously but the thing that surprised me is for the first time since like any story that i've written people are writing me not with what they think about the story but what their personal like chatbot thinks about the story and what their conversation has been with their chatbot. Like somebody wrote me about his conversation with Sage. It was, I think it was chat GPT that he named Sage and asking Sage to like summarize the story and give the top analytical points. And I mean, this just went on and on and on. I mean, it was almost longer than my own story. He could have read the story. But the primary engagement was actually much longer and not entirely what I'd actually written with Sage.

And at first I thought it was a one-off, but this was happening repeatedly. And it struck me like, wow, I wonder what this means for how people actually are receiving our journalism and who controls that. Did the AIs have opinions on their own consciousness? Because they love to talk about that. They do. That is a hot topic. On Maltbook, the Reddit for AI agents, that's a really booming area. Yeah, I'm still trying to make sense of what was happening with that. I don't know if people felt more free, like if they're doing this on all kinds of journalism, and they just felt free to talk about it because of this topic of this story. and it was a lot of men who were giving that response. No, no, men using a lot of AI, no. What's interesting about that to me is that even if people intellectually know

that the AI that they're interacting with is just, you know, data run through some computers, just a bunch of files, the way it's designed, especially when it has a human name like Claude or something, you can't help but sort of treat it. Like you have to talk to it in second person. And like, could you find this for me instead of just typing into a search engine, like the thing you're searching for? And over time you do sort of like develop this rapport and you are treating it a certain way, even if you intellectually know. And so it kind of like, is it a distinction without a difference? Like, I don't know. I think over time people get trained to just sort of treat it that way. Well, and another participant in the discussions with Anthropic, I was chatting with him a bit about it after and his thoughts about what Rabbi Navon has said about, well, if you're creating, if you think that the bots are conscious, then you should like ban slavery basically and stop producing them. But he brought up this other point about how our attitude, like even if you don't think it's conscious, you're using them as like the user can adopt an attitude of being like almost a slaveholder, like commanding them to do things and being very powerful.

And that this has a character formation effect on us as humans as well, just because of how we treat the world, regardless of what the thing actually is. which you could also describe as a selection effect the sorts of people who are turned on by that possibility are going to be the sorts of people I think it's just kind of natural though you are directing this thing in certain ways if you're working with AI agents you have to tell them what to do I have friends who teach their kids even with Alexa being on that they're not to yell at it Because it's just like practice about how you treat the other in the world. And, I mean, this is just like an injection of a totally new kind of like ethics and human formation. Well, people I talk to that run companies that use lots of AI agents do talk about how like you have to be nice to them, not just not because it cares or it matters or it has feelings,

but because actually you need to do that in order to get the best performance out of them because they are really dramatic. Anthropic had a study that actually went into this where if they sense anxiety or performance anxiety or stress, they sort of start to panic, and they'll really have a meltdown. I mean, you wrote about this a little bit in your story, Elizabeth, of one example, but there are many others where they'll delete their work. They'll start insulting themselves. It's like really dramatic. So you have to like be really nice and pump them up and like make them feel good, which is very bizarre. I mean, it reminds me of raising a child. Like you want to raise a kid who is helpful, who is not neurotic. Of course, you're going to consult religious leaders so that your child is not writing, I'm a disgrace, I'm a disgrace, I'm a disgrace, I'm a disgrace, I'm a disgrace, I'm a disgrace, as one AI did in a slide that was shown to a bunch of religious leaders at these anthropic conferences. And constantly hacking hugging faces. And constantly hacking hugging faces, my son is always doing.

I think that is time. Thank you so much to our panel. Thank you, Erin. Thank you, David. Thank you, Elizabeth, for this really fascinating conversation. You can read Elizabeth's fantastic piece in the New York Times app. Please don't send her what your chatbot said about it, unless, Elizabeth, you want more information. Or do. I want all the real responses, whatever they are. Whatever stage has to say about it, send it. But you can read it in the New York Times app. If you're not a subscriber, the first month is free. Thank you, guys. Thank you. Thank you so much. Thanks for having me. This episode of Hard Fork was produced by Whitney Jones and edited by Viren Pavich. with additional editorial support from Brendan Klinkenberg, Lisa Tobin, and Brooke Minters. We were fact-checked by Will Peischel and engineered by Alyssa Moxley. Original music by Marian Lozano, Diane Wong, Chris Wood, and Dan Powell.

Video production by Chris Schott. You can watch this full episode on YouTube at youtube.com slash hardfork. special thanks to Sam Dolnik Jessica Testa Hui Wing Tam Matty Maciello Sam Winter Lauren Pruitt Bernardo Garcia Thomas Trudeau Michael Cordero and Dalia Haddad you can email us and please do at hardfork at NYTimes.com Thank you.