ChatGPT: Productivity Boost or Innovation Killer?

In late November of 2022, ChatGPT was released to the general public and it was adopted like wildfire. A new artificial intelligence tool is going viral for cranking out entire essays in a matter of seconds. This has caused more excitement in the tech world than anything for several years. It was the most rapidly-adopted app in history. They acquired millions of users in the very first few days and it became highly talked about in the popular press. Everybody was convinced that this is going to have a massive effect on productivity of workers. It’s going to change the way work is done in so many contexts. The stuff that’s harder to measure, which is what happens to social connections.

This is kind of a continuation of a dynamic that’s been happening for a long time, for decades. So first it was email. Once we got email, people stopped talking to each other face-to-face as much. And then we got remote work, which didn’t take off a lot for a long time until Covid happened. And you lose some face-to-face interaction. You have to schedule a Zoom call with somebody if you want to talk to them. So it’s almost like this long-run dynamic that technology is slowly eroding social connection. And we’re kind of finding that LLMs are kind of amplifying that or exacerbating the problem. Basically, the question we ask is if people had not had access to a large language model like ChatGPT, what would they have been doing instead?

Oftentimes they’d be going and asking a connection a question, or they’d be ideating with somebody else, or maybe they’d be at the watercooler having a conversation. The fact that these tools may be replacing peer connections in many organizations essentially creates this concern. And peer connections are pretty important, right? For a variety of reasons. One reason is the thing I mentioned already, that you may lose some innovation from ad hoc interactions at the watercooler. But even if we just think about peer connection of a new employee in the workplace. They got a question when they first showed up on the job, they’d go ask somebody. Now they don’t necessarily need to do that. So maybe they can instead ask their question of ChatGPT. We’re losing bonds in the organization, which might mean that people are less attached to the organization and they leave more quickly.

It might mean that they don’t advance as quickly on the job. We collected a bunch of data from StackOverflow that was the largest online knowledge community for coding and technical work. And we looked at what happened before versus right after the introduction of ChatGPT into the marketplace. And we saw huge discontinuous declines. First of all, we saw that these drop offs in participation were really concentrated on certain topics, specifically topics that were where a lot of training data was essentially available to train a large language model like ChatGPT. Other programing languages that are less popular or less common, less training data, ChatGPT is not as good at answering questions about those topics. Beyond looking just at Stack Overflow, we collected similar data from Reddit developer communities and we see absolutely no effect on Reddit. Reddit is well known for having a social component to them. People go there to socialize, not just to ask questions about coding.

I think we need to start doing experiments and studying what happens to group dynamics when you drop this technology into the mix. But of course, we also need to start looking at what happens in actual business organizational contexts, right? Companies need to start thinking about what kind of tools do I allow my employees to use? The driver behind banning use of these tools in the workplace is information security concerns. Essentially, when you type questions and text into these tools, they ingest your text and it can be used as training data later to improve the models, etc. So both Samsung and Apple were worried that their employees would put corporate secrets essentially into these tools. If you’re not setting a policy in place, you’re allowing it, because industry surveys show that a huge fraction of workers in the workplace are using this.

Whether you want them to or not. And by the way, that also means even if you do have a policy against their use, a lot of your workers are probably still using it anyway. So the best path forward here is to think about actively adopting these tools and therefore being able to manage how they’re adopted, where and how they’re used. We actually outline different examples of strategies that can be used to help overcome these problems within an organization. So we talk about user interface design. So how people actually interface with the tool. We talk about incentivizing people where you can get rewarded by your peers for doing things that help them. So LinkedIn has a point system like that within its organization.

And one of the things you can earn points for is socially interacting with other people and helping them with their job. So you can think about doing that as a company, because that can actually have benefits for connection in the company and innovation. All the stuff I talked about previously. And as a byproduct, it may actually help with the effects that LLMs are having on people’s behavior.

Technology is always a double-edged sword. You always get these benefits, but there’s always some possible downside. So we have to figure out how to deal with it. And it’s the same with the LLMs. They’re here to stay because they’re so productivity enhancing. So the question is how do we manage them to mitigate the downsides?