What’s Up with GPT-3.5 and GPT-4?
A fresh-out-the-oven study is showing us how the performance and behavior of big language models – GPT-3.5 and GPT-4 – have changed between March and June 2023. And it ain’t pretty.
Why Should You Care?
These large language models, or LLMs, are used a lot. But when they get updated, we don’t always know what changes. The researchers wanted to find out how these updates shake things up and if the improvements might end up hurting some aspects of how the LLMs work.
ChatGPT Quality Drop: Real or Imaginary?
Many users (me included) have noticed that ChatGPT using GPT-4 has been giving worse answers over the past two months. This might be why fewer people are using ChatGPT. It was super popular when it first came out, but now folks seem to be losing interest.
Some folks on Twitter think it’s because school’s out, but I reckon it’s more about the quality drop. I’m using ChatGPT less, but still getting my work done – I just use other tools when GPT-4 can’t do the job. Anyone else doing the same?
How Did They Study It?
The study looked at how these LLMs behave over time. They watched GPT-3.5 and GPT-4 from March to June 2023, focusing on four tasks: solving math problems, answering sensitive questions, making code, and visual reasoning. They also measured how long answers were and if different versions of the same model gave the same answers.
Key Discoveries
Here’s the shocker: both GPT-3.5 and GPT-4 changed a lot in just three months. GPT-4 was super good at spotting prime numbers in March (97.6% correct) but really bad by June (2.4% correct). And while GPT-3.5 actually got better at this, both models got worse at making code and answering sensitive questions.
What It All Means
The way an LLM works can change a lot in a short time, sometimes for the worse. This matches what users have been saying about things getting worse, without any official word from the folks who make these models.
What Next?
The researchers think we need to keep an eye on LLMs and suggest users and companies do the same. They plan to keep studying LLMs, including GPT-3.5 and GPT-4, and sharing their findings on Github.
It’s clear that businesses using these LLMs need to be ready for changes in how they work. And it’s high time the providers of these services faced up to these issues, fixed them, and kept their users in the loop.
Related Articles
