A pretty big shift in the large language model game just happened, and not only that, the first GPT-5 competitor was essentially just announced. It all comes from anthropic AI, which has always been a strong competitor to OpenAI's large language models.
Claude 3.5 Sonet: The New Contender
Alright, folks, let's go right to the source: Claude 3.5 Sonet. For those of you who don't know, the CLA 3 naming scheme means that Sonet here is the midsized model. This is a new 3.5 updated Sonet model, and the crazy part about 3.5 Sonet is that not only does it crush the previous CLA 3 Sonet, it also crushes the previous Claude 3 Opus, which was the best CLA model that Anthropic previously had. So, this is the new best Claude model.
Even better, the Claude 3.5 Sonet crushes the GPT-4 Omni, which is currently OpenAI's flagship model. While OpenAI's flagship model, the GPT-4 Omni, does have some multimodal capabilities, it has not yet been released to the public. 3.5 Sonet flattens GPT-4 Omni for actual large language model tasks, which are, of course, the most important. And of course, 3.5 Sonet is completely free for any of us to use, and that's a pretty big deal.
Performance and Availability
Yeah, pretty much their blog post just reflects this: it raises the industry bar for intelligence and outperforms competitors and their previous flagship Claude 3 Opus with the speed and cost of their previous mid-tier model. It's free on the Claude website and the iOS app, and it's already available on their API, which is great for developers.
Token Context and Benchmark Results
Oh yeah, don’t forget it also has the 200,000 token context window, which is better than GPT-4 Omni. They provide you with a nice little bar graph to help you understand this. You can see that the Claude 3.5 Sonet is significantly more intelligent while also being significantly less costly than their previous flagship, which was the Claude 3 Opus. I mean, this is a pretty high bar to set yourself. This model is ideal for complex tasks and orchestrating multi-step workflows.
Agent Capabilities and Coding Evaluation
If you remember when the original Claude 3 set of models was announced, they also mentioned that some level of autonomy was on the horizon. This could be the first model that utilizes Anthropic’s multiple-agent automated workflow. Agents might be the big thing this year. In an internal agentic coding evaluation, Claude 3 Sonet solved 64% of problems, outperforming Claude 3 Opus, which solved only 38%. That's a pretty big leap in the world of large language models.
Advanced Features and Benchmarks
With sophisticated reasoning and troubleshooting capabilities, Claude 3 Sonet can independently write, edit, and execute code. It handles code translations with ease, making it particularly effective for updating legacy applications and migrating code bases. Here is that big comparison chart we always look for when large language models crop up. The Claude 3.5 Sonet pretty much wins out in every single benchmark, especially in comparison to the Claude 3 Opus, Gemini 1.5, and Llama 400B.
In the comparison to GPT-4 Omni, in zero-shot learning, it is true that in MMLU, GPT-4 Omni wins out, but only by 0.4%, so they are about equal in this benchmark. There is one more benchmark, the math problem-solving benchmark, where GPT-4 Omni wins out by 5%. So, Omni does win in that benchmark, but not by a whole lot. Overall, you could say that the CLA 3.5 Sonet model is better than the GPT-4 Omni. This is the mid-tier model for the 3.5 series, so there is an Opus model that’s going to be coming, CLA 3.5 Opus, which will be better than Sonet.
Multimodal Capabilities and New Features
The vision multimodal capability has been increased. Claude 3.5 Sonet is their strongest vision model yet. There is a demo video showcasing its capabilities, and the video comments are full of excitement. People are hyping up this model.
Artifacts: A New Feature
A new feature announcement: Claude artefacts. This feature expands how users can interact with Claude. When a user asks Claude to generate content like code snippets, text documents, or website designs, these artefacts appear in a dedicated window alongside their conversation. This creates a dynamic workspace where they can see, edit, and build upon Claude's creations in real-time. This is the beginning of a broader vision for Claude AI.
Looking Ahead
Anthropic has been cooking up some impressive advancements. This will allow you to securely centralize your knowledge documents and ongoing work in a shared space, with Claude serving as an on-demand teammate. Imagine this with more agentic capabilities—things that can work for you—and then come back with finished products.
Conclusion
Anthropic’s commitment to safety and privacy is emphasized, as always. They aim to substantially improve the tradeoff curve between intelligence, speed and cost every few months. To complete the Claude 3.5 model family, they’ll be releasing a Haiku model, which is going to be a lot cheaper and a little bit less smart, and then the Claude 3.5 Opus, which should be a lot more intelligent. This may be our first GPT-5-level model to be released, and they may get it out before OpenAI. This will be the beginning of a new era for large-language models.
0 Comments