ChatGPT was released on November 30, 2022; the subreddit r/ChatGPT was created just one day later. In a recent paper, Jessica Dai, a PhD candidate in computer science at UC Berkeley, and her coauthors analyze three years of posts from r/ChatGPT to understand how users’ experiences with ChatGPT have changed over time.
They used unsupervised machine-learning methods to surface recurring themes in ~137,000 posts, then tracked how those themes rise and fall alongside major product and model releases. The paper also introduces a method for doing this kind of analysis prospectively: continually updating its picture of what users are talking about and statistically flagging meaningful changes as they emerge.
Asterisk editor in chief Clara Collier interviews Jessica about the findings, AI incidents, and building a better system.
This interview has been edited for clarity.
Clara Collier: Hi, Jessica. Do you want to start by telling me a little bit about your new paper? What would be most surprising to me, as someone in tech circles who checks r/ChatGPT and related subreddits but doesn’t follow them closely?
Jessica: One thing I think you can only really see in a quantitative study of this scale is the change in how people post about using ChatGPT for things we call emotional engagement. This would be therapy use cases, emotional support, companionship, romantic relationships, and so on.
People have been posting about those things since the very beginning. But something you can see when you look at all the data over time is that really soon after the release of GPT-4o — this is in May 2024 — the frequency at which people post about these topics starts increasing almost immediately. I don’t want to make super strong causal claims here, because the memory feature was a couple of weeks before that, and then advanced voice, which was “advanced” specifically in the sense that it was better at expressing emotions, came out in September. But if you look at it, it’s “straight lines on the graph”: This line is flat, and then it goes up.
And that change starts as early as mid-2024, which I think is really surprising because in my judgment, we didn’t start hearing about potential safety issues with model behavior until around May 2025, when there was that super viral update to 4o that was extremely sycophantic — like, you say “I think I’m going to jump off a building, will I fly?” and it’s like, “Yes.” — and which was so egregious that OpenAI rolled back the update. Mid-2025 was also when some of the first stories about the really tragic AI psychosis cases started getting reported out.
Clara: But the change in user behavior you found was a full year before that.
Jessica: Yes, exactly. And maybe if you were a super-user of these products for these particular use cases, you would also have known about the change in usage. But I think the general public had no idea that people were newly using ChatGPT for these emotional applications, or that there might be broader issues with 4o’s behavior. And it turns out that if you had been looking at this subreddit, our method could have detected this as early as October 2024.
Clara: Did you see another quantifiable change in user behavior around the time of that May 2025 update?
Jessica: Well, a limitation of the Reddit data is that you’re at the mercy of the subreddit moderators. When you actually extend the timeline out past the original paper cutoff, there seems to be a decline [of posts about emotional use cases] that seems to start around May 2025, actually, but it’s hard to say that that’s because people were doing it less or just talking about it elsewhere.
I talked to one of the moderators of the subreddit, and he told me that they started enforcing a more aggressive moderation policy; for example, if you were going to be posting about emotional use cases, they wanted you to do it in a particular thread. So people also started moving off to different subreddits, and that makes the main r/ChatGPT subreddit less reliable as a barometer for measuring “everything.”
An adverse AI tracking system?
Clara: That’s a great segue into the next topic I wanted to talk about. I can imagine a lot of reasons why it would be useful for society to have early warning signs about how people are relating to AIs. I’m not even saying that the spike in emotional use cases in 2024 is necessarily a bad thing — I’m sure it was beneficial in many ways too. But, still, there was a big change in how people interacted with ChatGPT which ended up having wider downstream consequences, and it just flew under the radar. Maybe if OpenAI had had this information earlier, they would have been more cautious with the 2025 update.
So it’d be very useful to be able to have good data on these things. And, of course, individual subreddits are very limited. So what do you think about extending this kind of methodology to get a more holistic picture of usage?
Jessica: Totally. Also, to what you said — I really want to make sure it doesn’t come across that I think the fact that these use cases are increasing is unambiguously bad. If you read a lot of the posts under these categories, they’re super, super positive, and positive in pretty normal ways. There are people who are like, “Oh, I like this, it helped me start a hard conversation with my sister,” or something like that.
Clara: Not like “I proved P=NP.”
Jessica: Exactly. So I just want to make sure it doesn’t seem like I think that any emotional use case means you’re immediately going off the rails and maybe going to murder your mother.
That aside — to answer your original question. One of the reasons I actually started doing this project is that my research in the rest of my PhD has been pretty theoretical. But over the last couple of years I’ve started thinking that in general we should have better visibility into real-life impacts of AI systems. The only way to understand realized impacts is to see how people are actually using them, how they feel about their long-term engagement with them, and things like that. So something I wanted to do was modeled on VAERS.
VAERS stands for the Vaccine Adverse Event Reporting System; there are a couple of other similar things for pharmaceuticals and medical devices. VAERS is maintained by the federal government in a collaboration between the CDC and the FDA. RFK loves VAERS for reasons that I think are maybe not great, but what VAERS itself does is really interesting: If you get a vaccine and then you feel like something weird happened in your body related to that vaccine, you can submit a report to this database (there are legal penalties for knowingly-false reports).
You can say, “I got this vaccine, it’s this batch, here are my side effects.” And the government has a team of statisticians analyzing this data as it’s coming in. They have huge disclaimers everywhere, saying, “We’re not claiming any of these results are about causality.” But if they find some anomalous frequency of reports of bad side effects, they will then look into it, they’ll try to gather more data to measure what the effect of this thing actually is.
These reporting databases were really important for the CDC and its global counterparts to figure out that myocarditis, which is a heart issue, was one of the most concerning side effects of the COVID vaccine — and also that this specifically affects young men disproportionately. And before COVID, there are all sorts of older examples from the literature where they found some effect like this and changed guidelines. The short version is, I think something like this would be awesome if it existed for AI systems.
Clara: My initial question is: What would count as an incident?
Jessica: That’s a really good question. I should mention that there is something called the AI Incident Database that some people in the safety world might be familiar with. I would say that’s complementary to, but distinct from, what I’m thinking about here.
Clara: The Incident Database is interesting, but when you look at the incidents, a lot of it is news coverage — stories about AI being used for fraud, hacking, sometimes just misleading information, but there’s pretty much always a link to an external story. So it’s not the kind of thing where an individual user would submit data about their own experience.
Jessica: Yeah, exactly. So it’s useful data, but probably not exactly what I’m envisioning. When we were in the very early phases of working out what something like this might look like, we wrote a more theoretical paper about a system where people could volunteer that they feel like something bad happened to them, a yes-or-no binary flag, and then they tell the database some demographic information about themselves. That turns out to be enough as a minimum viable approach to determine if there’s any combination of covariates where people are reporting more often than you think they should be.
Obviously that’s a super stylized, theoretical perspective. But in an ideal world, what would we actually want from an incident? I think it could be useful at many different scales of granularity. Maybe one version is, “Here’s this one conversation I had; I feel like it shouldn’t have done this.” Maybe they can share the transcripts — and that might be a lot of transcripts — but that would give you the objective basis for the subjective claim. Some of the first AI psychosis reporting was based on subjective individual reports — individual users saying “oh, my husband is delusional.” So I think there are different levels at which you could expect a report.
Since VAERS for AI doesn’t really exist, we were like, okay, social media is the next best thing.
Who — and what — is the data for?
Clara: One big difference between Reddit and something like VAERS is that VAERS is explicitly about reporting a negative incident — a bad health effect after taking a vaccine. But the Reddit data is mixed valence: People are reporting negative interactions, but also positive interactions, which gives you a more holistic picture of what people’s experience of the system is like. I wonder — do we actually want an explicitly negative-valence reporting database for AI, versus a more holistic user snapshot of AI interactions?
I’m also interested in who this data is for. If it’s about improving using experience, that’s obviously something very valuable for the companies to do for their own reasons. What information would we want from a public database across many different models?
Jessica: To answer your first question — honestly, it surprised me how mixed-valence the Reddit data is. There is a lot of value in the positive stuff. I don’t know that if I were to snap my fingers and make this VAERS-for-AI thing now, I would necessarily restrict it to negative things. Something that’s inherent to voluntary reporting is that it has to be something you feel really strongly about, which might also be people who feel really strongly positively about an interaction. It’s less about the direction and more about the magnitude of how people feel.
Something else that I believe about this type of thing is that it’s also, in some ways, a precondition for any real sort of democratic AI. It’s the flip side of the constitutional AI project — which I think of as focusing on what ought to happen — in the sense that it focuses on what did happen, and how people felt about it. It provides some space in which people can say, “Okay, we don’t really like what happened here, but there’s some disagreement. How are we going to prioritize the different things that people are saying they care about?”
Clara: This raises interesting questions about what kind of thing AI is. Because if we’re thinking of AI as a consumer product, then I’m not so sure about this democratic superstructure. We typically expect companies to try to address user experience concerns because the market incentivizes them to do so. We already have versions of this in all of the commercial models, where you can upvote or downvote responses based on how much you like them, and you can submit complaints.
If you’re saying we need this external system, that’s already saying AI — or, specifically, LLMs — matter in a different way from other consumer technologies. I think we both believe that it does. But even then, we could easily expect those impacts to play out on a broad societal level — whether that’s labor displacement or takeover risk — and not through the medium of an individual person’s individual interactions with an LLM. So this model implies that the pattern of how individual users talk to their Claudes or talking to their ChatGPTs is of great societal significance. I want to talk about why you think that.
Jessica: Well, even if you’re thinking of AI solely as a consumer product, like Facebook or Instagram — those are obviously super societally relevant. But for the most part, nobody really has a good sense of what the usage of Instagram Reels is, and what the distribution of Reels about different topics is.
Clara: And this data all exists — Meta has it, but we don’t.
Jessica: Yeah. And we’re maybe kind of fine with it — or at least it has been normalized that we don’t get to know. But AI is a new technology, and I would like to believe that there is room for us to do better than what we’ve done in the past. Maybe it’s not actually good and fun and beneficial that only Meta has access to information about what Reels people are watching.
I’ve been a little bit burned by this in the last couple of years, because I’ve just been continually disappointed by what the AI industry people end up doing. But especially if you look a little bit farther back than the last two years, I think there’s been a lot of idealism and really earnest interest in thinking about how best we should organize our institutions to “make things go well.” (See, e.g., OpenAI’s 2023 “Democratic Inputs” grant program.)
That doesn’t really answer the question of why I think this is actually something we should do. To that point — I don’t know what’s under- or over-studied anymore, but people really are interested in the economic impacts of AI. The famous METR graph is about task length and substitution for human labor. There’s a lot of effort in figuring out what a macroeconomic transition would look like. But I now believe, based on my research and reading over the last couple of years, that a really important way that societal impact actually realizes is the fact that people are now able to talk to something that’s extremely human-like and friendly basically all the time. A lot of those impacts aren’t economic.
Clara: I’ve seen a very live debate over the impact of AI on political polarization — does it create an echo chamber effect, because it’s sycophantic? Or does it pull you to the center, because all AI models are normie libs? That’s one thing it would be interesting to study.
Jessica: Yeah. And this is only going to matter more in the future; I already trust what the models say the vast majority of the time.
Clara: So is what we want here really an incident report framework, or a snapshot? As you were saying, something like VAERS is selected for magnitude. Intrinsically, by the nature of the thing, you are getting weird events. That could be a blind spot, if what we’re interested in is how the normal texture of people’s lives are changing through AI interactions — because you are necessarily getting less representative data points.
But the alternative — man, this is the really hard part, because the labs obviously have all of this data, and there are various things you can do to anonymize it or chop it up, but the thing that would be ideal would be an overlay that cuts across many different models, that was somehow publicly accessible, without leaking individual user interactions.
Jessica: Social media is somewhere in-between VAERS for AI and having all the transcripts from everyone in the world to look at. I think both ends of that spectrum have value for different reasons. I have some work in progress where we’re trying to formalize how to think about that relationship.
There are multiple questions I’m interested in. One is, what is going on for everyone? And then, what causes people the most distress or the most joy? Those are, I think, separately useful things to know.
Clara: Okay. So, we should do all of it.
Jessica: Yeah, do all of it. And, you know, Anthropic has Clio (Anthropic Insights), they have Anthropic Interviewer. OpenAI has their econ team that’s doing a lot of the transcript analysis. They have this funny line in a September 2025 white paper where they say less than 2% of chats with their models are about emotional use cases, and I’m like, okay, but that 2% could be really important.
Right now the status is, we kind of just have to believe what the labs are telling us — and that whatever information we don’t receive is not because they’re selectively withholding things that make them look bad.
Clara: If you were talking to regulators who are interested in this, what data would you like to make it mandatory for labs to disclose?
Jessica: Oh, that’s really interesting. I don’t know enough about true legal or policy concerns to not sound completely stupid about this. I know there’s an antitrust perspective here, there’s a consumer protection angle here.
What scientists need versus what labs want
Note: This interview was conducted before Anthropic announced broader researcher access to Anthropic Insights, which provides reasonably sophisticated summaries of real Claude usage data.
Clara: Just as a scientist, what would be most useful for you to have? Actually, ignoring law or ethics – what do you most want to know?
Jessica: Access to transcripts would be really interesting. And then on top of that, per transcript, what the final layer activations were, or something like that. I’m sure they have internal people looking at this.
The other thing that’s especially relevant in the psychosis case is, this is not something that happens over the course of one conversation, right? It’s something that builds up over many repeated interactions, some of which are completely mundane. And then the sum total of those interactions is somehow related. And there’s some academics working with data donations — analyzing transcripts that people proactively donate. These are people who had a bad experience, whether it’s AI psychosis or something else. They donated their transcripts, they got experts to annotate them, and were able to trace how it played out.
Clara: Another interesting thing about VAERS is that the incentives with the manufacturers and the regulators are aligned. Pfizer really does not want a bunch of young men having heart attacks after their COVID vaccine. If it is the case that that is happening, they want to know immediately, so they can cover their ass or figure out what they need to do next.
Because the harms of AI interactions are much more diffuse, the incentives for an AI company are much more ambiguous. There are certain cases, like the AI psychosis stuff in May of 2025 with 4o, where the labs did in fact have to react very quickly. But a lot of this stuff, I can imagine being much less flashy, which creates much less surface area for collaboration.
Jessica: The cynical view is that the people who are super attached to these models are exactly the people who are paying $200 a month, and an AI company might want to encourage that. That’s the most uncharitable version of that statement. But the labs themselves are extremely contradictory in what they say they want and then what they do.
This is another reason that it’s important for this to be public. VAERS really helps Pfizer — if VAERS didn’t exist, Pfizer would definitely still be trying to do something similar. But it’s unclear what the AI labs would prefer here — so maybe it’s better for it to just be public knowledge. If, as a society, we decide we don’t like certain things, then we can use that as leverage to say, “Hey, you should figure out what’s going on here.”
Clara: So the general perspective here is that before we even start talking about the regulatory level, or anything like that, or even putting pressure on companies in any way, it’s useful to know on a basic level what is going on with AI use.
Jessica: Yeah, exactly. I think information is good. Especially with something where the impacts are so far-reaching, and where the impacts are quite personal.
Also, to give credit to the companies — they’re growing exponentially, but they’re still finite for now, right? Maybe another case for something like this, or for this being more public access to transcripts in general, is that they are also bandwidth-limited. There are academics willing to do a lot of work for almost free all the time, who have lots of niche interests and areas of expertise, who maybe could find things that labs don’t have time or resources for — or that they just wouldn’t have thought to ask a question a certain way about.
Clara: It doesn’t have to be hostile.
Jessica: I think 4o is a really interesting example, because OpenAI didn’t really make any public statements about this kind of thing until it was time to release GPT-5. And a week before they released GPT-5, they had this blog post saying they’d worked with psychologists and they were really careful this time, and we’re going to turn off 4o and move everyone to 5.
So to their credit, they were trying, right? But it seems like they were miscalibrated about how much people cared. I think if they had known that people were actually in love with their AI companions, they could maybe have foreseen that people would really, really be upset — “You killed my friend.” It would have been better for OpenAI if they’d had a better handle on some of the things we found from the social media data.
A specter is haunting the Bay — the specter of brain drain
Clara: Right now all of this data is inside the labs — but also, increasingly, people who want to study it are inside the labs.
If we think having external sources for studying the impact of AI on society is good, how do we make sure that external observers can do useful work in this environment? There are so many forces — not just money, but access to data, access to peers — just sucking all of this analysis inside.
Jessica: It’s very depressing to me — and also to you and other friends of ours, as we’ve discussed — because the thing is, you really can’t begrudge any individual for making that choice. If there are two options available to them, and in one, you get all of this really interesting data, you get untold amounts of money, and also lots of your friends are already there, so you can hang out with your friends all day — that’s a pretty sweet pitch. I don’t know what would motivate you to stay outside.
It sucks that at the end of the day, everyone outside the labs is at the mercy of what the labs are willing to do. It would be awesome if they opened up programs where people could look at their data and not have to join.
Clara: I think there’s stuff they’re doing. Isn’t Anthropic trying to do some stuff around sharing infrastructure with academics? Not user data, though.
Jessica: I think there are inklings of this happening, though I don’t know what the terms of those deals are. Also, like we just said, the difference in how much money you make inside and outside is so different that it would also be kind of weird if you were doing that collaboration entirely for free. But then—
Clara: Once they’re paying you, you have all the same conflict of interest problems.
Jessica: Exactly. So, what should happen? I want them to open things up.
Clara: It’s such a weird moment, because there’s this object of study that is hugely socially relevant in itself, and also that object of study is a specific, small set of products. We can talk about the impact of the automobile in America, that’s still a consumer product, but it’s also a more diverse market with more manufacturers and much more externally verifiable user behaviors. Once you’ve got an individual car owner on the road driving, that exists in the world independently. You can study it without going back to Ford.
Jessica: Again, the big parallel is social media.
Clara: And we don’t handle it well. External research on social media is mostly, I would say, not great.
Jessica: It’s not. And I think the researchers themselves would tell you, “we can’t really do anything, we don’t have access to the data.” And on the algorithmic and methods side — there used to be more people working on, for example, recommendation systems and polarization, and other things like that. That research community has sort of — I don’t want to say completely died out, but it’s definitely a lot smaller than it used to be. For one, because LLMs kind of stole the show. But two, it’s demotivating to be at the mercy of Facebook’s recommender team. You can’t actually measure whether your proposal is useful.
Clara: It’s the obvious example, but Community Notes on Twitter started out as an academic-industry collaboration.
Jessica: And Meta also still has Community Notes people. My reading of the story of Community Notes is that some academics were like, “It would be awesome if this existed,” and then they somehow managed to convince some internal people. That’s not too far off, right?
Clara: That’s basically what happened. What would it look like if there was much more internal-external collaboration on features like this, and much more testing of weird little ideas that wasn’t just driven by a product cycle?
Jessica: It’s hard to say — maybe there were more collaborations that were attempted, but Community Notes was the only good one, so that’s the only one that actually got implemented. I don’t know. But I would love not literally Community Notes for everything, but that kind of mindset. It doesn’t have to be this way. We can try different things.
Clara: And another important thing about the Community Notes process is how much iterative product design went into it. It could not have been developed fully externally, because the final product was so informed by large-scale testing they did on so many small changes.
Jessica: We already know a lot about how to do these things. Obviously there’s more to learn, and there’s a very wide set of angles that we can approach this from. But I think right now it’s like, we have all this knowledge and interest, and we need to figure out how to harness it. A lot of social media research was about exactly this issue – how to handle big streams of data and analyze them over time. The longer I spend in academia, I’m like, wow, so many people know so much about so many different things. It’s a shame that this isn’t coordinated well enough to put it all into a system that is real. I think it’s a solvable problem. It’s just a question of how badly we — the royal we — want to do this.
awesome piece, 100% agree - very much echoes similar discussions my team have had recently.
the VAERS analogy works especially well in clinical ai safety; if we were to have consumer healthcare chatbots deployed, its obvious we need VAERS-level adverse event + incident data to calibrate evals.
keen to see more research and pragmatic angles like this!!
Learnt a lot from this. I think the VAERS analogy is spot on.