MCP-Atlas open-sourced

There is a growing realisation in the AI world that being good at language is not the same as being good at getting things done. That is the thinking behind MCP-Atlas, a new benchmark that looks at how well large language models actually use real tools, not just talk about them. The team behind it has now open-sourced the whole thing, including the research paper, the dataset and the evaluation setup.

At a basic level, MCP-Atlas is about testing whether AI models can cope when they have to interact with real software tools through the Model Context Protocol, or MCP. Instead of asking models trivia questions or giving them neat, self-contained puzzles, it throws them tasks that require calling the right tools, in the right order, with the right inputs.

The release comes with three main pieces. First, there is a research paper explaining how the benchmark works and what it is trying to measure. Second, there is a public dataset of 500 tasks written by humans, available on Hugging Face. Third, there is a containerised evaluation environment that exposes real MCP servers, so models are dealing with live tools rather than mocked-up examples.

Another 500 tasks exist, but you cannot see them. They are held back as a private validation set, which is meant to stop people from gaming the system or overfitting their models to the test. That matters if you want a leaderboard that actually means something.

What makes MCP-Atlas interesting is how realistic the setup is. The prompts are written in natural language and deliberately avoid naming the tools that need to be used. Models are given access to a limited set of tools, some of which are red herrings. In other words, the model has to work out not only how to use a tool, but whether it needs one at all.

Early results suggest this is where things start to fall apart. In recent evaluations, Claude Opus 4.5 came out on top with a 62.3% pass rate. GPT-5 followed with a 44.5% pass rate. That means even the best models are still failing around 40% of tasks that require coordinating multiple tools.

The failures are revealing. According to the benchmark results, the main problem is not reasoning or language generation. It is tool use. Models often pick the wrong tool, pass the wrong parameters, or call tools in the wrong sequence. Quite a few simply give up too early or do not realise that a tool is needed in the first place. When a model does manage to use the tools correctly, pulling the results together into a final answer is usually the easy part.

That lines up with what many developers are seeing in practice. Large language models sound confident, but as soon as they have to operate software, query systems or chain actions together, the cracks start to show.

The team behind MCP-Atlas has published links to everything: the paper, the dataset, the GitHub repository for the evaluation environment and a public leaderboard. They are actively inviting feedback and contributions as the benchmark evolves.

The message is fairly clear. If AI is going to move from clever demos to reliable agents, tool use is still the big hurdle. MCP-Atlas is less about celebrating how far models have come and more about putting a spotlight on how far they still have to go.The Recap

• MCP-Atlas benchmark and evaluation tools are publicly available.
• Claude Opus 4.5 leads with a 62.3% pass rate.
• Remaining 500 tasks withheld as private validation for leaderboard.