E-Commerce AI Agents Fail a New Benchmark Built for Real Work

Why Old Benchmarks Don't Cut It

For years, testing an AI model was simple: you gave it a set of questions, it gave you answers, and you scored it. Writing, math, coding—all turned into a single number that supposedly told you how smart the model was. That worked when AI was just a chatbot.

But now AI has stepped out of the chat window. It can call tools, browse websites, handle files, and execute tasks across systems. It's an agent, not just a talker. And that changes everything about how we should judge it.

In e-commerce, for instance, you might see an AI agent that can browse Taobao, add items to a cart, and place an order. But on the merchant side, AI is doing far more complex work: sourcing products, managing listings, handling logistics, and analyzing operations across multiple platforms.

To find out how well AI actually performs these real-world jobs, the team behind Accio Work built a new benchmark called RealReplicaBench. It's not a multiple-choice test. It's a simulated business environment where agents must complete actual tasks. And the results are humbling.

All Models Fail the Real-World Test

RealReplicaBench put 13 AI models through 107 real business tasks. None of them passed. The highest score was 56.1 out of 100, achieved by Claude Opus 5. GPT, Claude, GLM, Qwen, DeepSeek, and Gemini (which, unsurprisingly, came in last) all fell short of the 60-point passing mark.

The benchmark isn't designed to be easy. It's built to reflect the messy, interconnected nature of actual work. As the team behind it puts it, “There's no 'good enough' here. If a task isn't fully completed, you get zero points.”

That's a stark contrast to traditional benchmarks, where partial credit is given for partial effort. In a high school math exam, you might get a point for writing down the formula even if you can't solve the problem. But in real business, a half-finished task is often as good as no task at all.

What “Completion” Really Means

The core philosophy of RealReplicaBench is that a task is only complete when its output can be directly used by the next step in the workflow. If an agent does 80% of the work but leaves the most critical 20% for a human to fix, it's a failure.

Take supplier sourcing as an example. An agent must go through about 300 noisy emails, extract the actual sourcing requirements, select suppliers, draft replies, and set up a kickoff calendar. It's not enough to summarize emails. The agent has to turn that information into a set of actions that move the business forward.

Another task requires the agent to transform 5,383 customs records into a cross-system procurement control tower. It has to aggregate data by category and supplier, filter the top three suppliers based on purchasing policy, and then build a dashboard in Google Workspace, a folder in Box, and a project task in Jira—all while keeping the relationships between these objects consistent.

If a dynamic ID is wrong, the entire handoff can break. That's what the benchmark is testing: can the agent maintain correct state across a complex environment and push a business goal to its true end?

Simulating Real Work, Not Just Questions

To test real work, you have to recreate the conditions of real work. RealReplicaBench doesn't just give the agent a text prompt. It provides a fully simulated environment with front-end UIs, browser operations, CLI, API/MCP, file systems, and backend states.

This is a far cry from the old approach, where you'd write a question like “Summarize this email” and check if the model could produce a decent paragraph. In the real world, page states change, historical info conflicts with new info, and systems don't share a common vocabulary. The benchmark forces agents to deal with that mess.

One task, for instance, involves logistics fulfillment. The agent must list all viable routes for a shipment from China to the U.S., factoring in ocean freight, trucking, last-mile delivery, insurance, customs clearance, bond, and platform fees. It has to eliminate any route that exceeds 30 days or has port transfer issues. But it doesn't stop there—it also has to book the ocean freight and verify the shipment. The final check isn't whether the agent wrote a convincing plan. It's whether a real Shipment ID was generated.

Verification Is the Hard Part

Another challenge in testing agents is that they can claim success even when nothing actually happened. A chatbot just needs to produce text. An agent's text output is only a means to an end.

So RealReplicaBench uses a verifier that reads the final state of the environment, not the agent's self-report. It checks whether the listing was actually created, the booking was actually made, the calendar event was actually set, and the files actually exist in the right places. Only then does it award points.

This strictness is what makes the benchmark so demanding. It's one thing to write a plausible answer. It's another to change the world in a way that can be independently checked.

Built on Real Data, Not Theory

The 107 tasks in RealReplicaBench aren't invented by researchers. They come from real business needs, distilled from about 1.6 million conversations, 200,000 execution traces, and 2,000 high-value workflows.

The design of a benchmark reflects how its creators think about AI. If you believe an agent is just a smarter Q&A system, you'll test it on answers. If you believe an agent's value lies in getting work done, you'll test it on completion. Accio Work clearly believes the latter.

That's why the benchmark is so focused on environment, tools, state, verification, and failure attribution. It's not just a test—it's a methodology for building agents that can be trusted with real responsibilities.

What This Means for Industrial Networking

While this benchmark is built for e-commerce, the lessons apply directly to industrial networking. In factories, warehouses, and supply chains, AI agents are increasingly being asked to do real work—not just answer questions. They might monitor network health, adjust production schedules, or coordinate between machines and enterprise systems.

The same principle holds: a task isn't done until the next step can pick it up without human intervention. An AI that can diagnose a network issue but can't automatically open a ticket or update the inventory system isn't truly completing the job.

In industrial settings, the cost of a half-finished task can be enormous. A misconfigured network device, a missed maintenance window, or a wrong part number can ripple through an entire production line. That's why the standard for AI in industrial networking must be just as strict as RealReplicaBench.

The Future of AI Evaluation

Accio Work plans to keep expanding RealReplicaBench with new real-world tasks and rolling updates. They also intend to use it for model selection and training optimization—routing tasks to the models that perform best on them.

This kind of benchmark isn't just useful for merchants. It's valuable for the entire AI community. As AI moves into business workflows, the ability to build good harnesses—the systems that connect models to real work—will become as important as the models themselves.

After all, what users are willing to pay for isn't an agent that's “almost good enough.” It's a system that can take over a job and hand back a finished result. RealReplicaBench turns that simple-sounding requirement into a repeatable, verifiable standard.

In the past, we asked how much an AI knows. Then we asked if it can reason. Now, in the age of agents, the question is whether it can actually finish the job. That's the new yardstick, and it's a much harder one to measure up to.