Is your organisation more productive as a result of GenAI?

The world’s most powerful autocomplete service powered by AI - ChatGPT - celebrated its first birthday on 30 November 2023. In the year since launch, ChatGPT has given people access to the remarkably lifelike outputs of the generalist knowledge and assemblage of words based on the connections in its unique and massive dataset.

With it came the doomsayers who claimed that AI tools were going to replace jobs in our organisations. Research by OpenAI from March predicted that up to 49% of workers would see half or more of their tasks exposed to LLMs (large language models) like ChatGPT. Further research conducted by Hui (et al) shows that there has been a sharp decrease in freelance jobs (and freelancers’ earnings) since ChatGPT launched in November 2022.

However, I have also engaged with many business leaders who can point to moments where they have felt productive using ChatGPT, but can’t see much in terms of effectiveness across the organisation.

In this blog, we’ll explore the impacts of ChatGPT and other large language models (LLMs) on team and individual productivity, giving you some key factors to bear in mind when trying to answer the elusive question - “Are we more productive as a result of using Generative AI?”

Driving productivity gains through Generative AI: 4 factors to consider

Before you start trying to determine productivity, begin with organisational purpose

The purpose of most organisations isn’t to generate text. Normally it’s to solve a customer problem, convince the customer that you have solved the problem, and then persuade them to part with their money so that they can benefit from the solved problem.

Or, as Peter Drucker said, “The purpose of a business is to create a customer.”

Unless that problem is generating text (possibly based on a prompt), you’ll be unlikely to see a direct effect from the use of ChatGPT on your organisational efficiency. Even if you did, what is stopping the customer from writing their own prompt and solving the problem themselves?

True productivity is all about collaboration

To execute on your company's purpose you need smart people working together towards a shared goal. Some of them even create text in pursuit of that goal. Take, for example, your software development function. Is the purpose of your software developers to type code as quickly as possible?

No.

Your software developers are trying to translate your user needs into software-powered solutions. Typing is the very last piece of this puzzle. First, they need to understand the problem to solve and prioritise the minimum effort to solve it. They need to agree on a solution that is not only safe and free from causing additional problems but also adaptable for future modifications.

They need to collaborate.

Then they need to write some of that knowledge in the form of code and ensure that it does what they say it does.

Experimenting with code generation and optimisation is key

We’ve worked with companies and conducted research with code optimisation and generation tools to see if it makes developers more productive. The results show that it does indeed shorten the time it takes to deliver a solution. In one particular study, we identified a mobile application that was built in a legacy technology. We measured the time it took for a team to rebuild the application in a new and supported technology and compared this to another team who built the same feature set with the support of GenAI tools.

We found that the development team was able to deliver 25% faster using GenAI tools.

This is, of course, subject to the software development factors for this particular research. It’s also true that software design and writing clean code in a specific paradigm/language is a skill. Few AI models will have been trained with high-quality software, especially as much of it is often closed source, yet the pool of available open-source material is so vast that finding good quality source code (from which the AI can learn) is like looking for a really small needle in a particularly enormous haystack. There remains a risk currently that injudicious use of LLMs to generate code can represent dated common practice rather than cutting edge insight.

Despite this, however, our experiment showed a clear effect: when you are delivering software based on a baseline and measuring it, productivity gains are there to be had.

Yet companies rarely have the opportunity to time a rewrite, nor can they apply the data in a manipulated way. What they instead see is “development took three sprints” rather than “development took four sprints”, or “we can deliver the same scope with only 75% of the team”. It’s hard to spot the difference when the timescale is so large. To truly see the effect of GenAI tools in your own teams you must introduce the tools and measure your new baseline (together with the quality of the code written and the documentation generated) over a period of time. Then you can see the real productivity gain your teams can achieve from GenAI tools.

The real work is in prioritising which problems to solve

In mature software development teams, the real key is to have a continuous delivery experience where they deliver the most valuable piece of work possible to the user as quickly as possible (i.e. flow). Obviously this requires some code to achieve it but the hardest piece of the process is understanding the right problem to solve and how the team can best solve it.

Generative AI tools might solve some rudimentary problems in this space, but the most important qualities are the way the team can collaborate best, and the specialist domain knowledge they have accrued so far in the pursuit of that goal. Neither of these are problems suited to generative AI.

Conclusion: Generative AI is just a tool

Generative AI solves hundreds of tiny problems, but it doesn’t do the work in organisations. It’s just a tool. What it can do, however, is accelerate speed to value, one of the key factors we see of successful generative AI adoption (alongside improving the human experience and adopting an innovation mindset), and an essential capability for any organisation seeking to achieve digital greatness.

GenAI can help people complete text-based work a little faster, freeing them to focus more on the way they collaborate and less on the amount they type. Happy birthday autocomplete 2.0.

If you’re ready to accelerate speed to value through Generative AI in your organisation, take a look at our GenAI Hub.