Veloccy: Outperforming Frontier Models at Operating a Business, in a Sealed Full-Scale Simulation

Abstract

We test whether an AI system can make the operating decisions of a large online marketplace, using a full-scale simulation in which the consequences of any decision can be computed. The simulated company has 51,849,251 accounts and a warehouse of 589,377,423 rows. Five of its teams made changes that, together, appear to break its discount programme. The system must decide what to do with a ₹3.2 Cr budget and what to fix first. The true causes and the best attainable result were computed by counterfactual replay and sealed before any system ran. Veloccy’s plan delivered 120% of the extra orders of a reference plan that was found with the answer key, and beat it in all 10 alternative replays (120 to 131%). Across 27 sessions of Claude Opus 5.5 and Claude Fable 5.1 in six configurations, the last of them with Python, the best plan reached 82%. On a sealed 18-item marking scheme Veloccy scored 16.5, against Claude’s best of 15.5. Given the same records, Claude stayed well below Veloccy. We trace the difference to what Veloccy brings to a question: a working model of the company built in advance, history the company’s own systems had lost, experience of every transaction at full scale, analysis tools it builds for itself and keeps, retrieval over a growing body of analytical experience, and an explicit method.

1Introduction

Running a business means making decisions with consequences: where a budget goes, which broken process gets fixed first, who has to act. These decisions usually depend on what other teams have done. A release changes what customers see, a messaging rule changes who receives what, an advertising shift changes who arrives, and a reporting change alters the numbers everyone reads. AI systems are now asked to help with this work. Most evaluations of them, though, test a narrower skill: whether a system writes a correct query or returns a correct number.

We evaluate the decisions themselves. We built a full-scale simulation of a marketplace in which every decision can be run forward and its outcome counted. Because the simulation can replay the same shoppers with any change switched on or off, the true effect of every change is known exactly. A proposed plan is judged by the orders it would have produced.

Veloccy is an AI system that works inside a company’s teams. It learns the business from its data and its people’s records, proposes decisions with the evidence for them, and leaves each decision with the person who owns the lever. In this evaluation no person answered its questions, so every decision scored here is Veloccy’s own.

On this task Veloccy made markedly better operating decisions than the strongest available models (Figure 1). Its budget plan beat a reference plan computed with the answer key, and held that lead under ten alternative replays. Claude, given the same data, did not come close.

This report makes four contributions:

1. A simulator of a large marketplace with an explicit behavioural model, exact counterfactual replay, and validation against public figures and against itself (Section 3).
2. An evaluation protocol that scores both the reasoning behind a decision and the outcome of the decision, with the answer key sealed before any system ran (Section 4).
3. A controlled comparison of Veloccy with Claude Opus 5.5 and Claude Fable 5.1, in which every system’s inputs are listed source by source (Sections 5 and 6).
4. An account of where the advantage comes from, measured part by part in the run that produced it (Section 7).

2Related work

Benchmarks for AI on data work mostly score an answer against a reference answer. Spider 2.0 and BIRD score generated SQL against reference queries and results; DABstep and DSBench score the final value of a multi-step data analysis. τ-bench moves closer to operations: an agent serves a simulated user under a policy, and is scored on the final state of a database. GDPval asks industry experts to compare model deliverables with expert deliverables. None of these scores a business decision by its consequences. Our environment does, by running each proposed plan forward in a simulation where the ground truth is computed rather than labelled. Because the outcome measure is computed, it is also verifiable, and could serve as a training reward as readily as an evaluation score.

3Environment

3.1Simulator design

The company is an online marketplace selling low-priced goods, mostly fashion, to buyers across India. Each of its 51,849,251 accounts is simulated individually, every day, from July 2025 to October 2026. An account carries traits drawn when it is created: city tier, acquisition source, phone platform, messaging opt-ins, a usual buying rate (lognormal, median 1.5 orders a year), a propensity to pay cash on delivery (COD) and a propensity for a COD parcel to be refused (both Beta-distributed by city tier), a sensitivity to discounts, and how quickly it updates the app.

On each day the probability that an account orders is a product of interpretable terms:

where r is the account’s buying rate, t the days since its last order, D the calendar (growth, season, festivals, platform sales, weekday, salary week), M the lift from messages received that day, and e the effect of any discount offered. Two terms carry most of the behaviour that matters for this task:

• Need, N(t) = 1 − 0.88e^−t/11. Just after an order, a shopper needs little. An offer that triggers an order therefore lowers the chance of the next one. Orders “pulled forward” by a discount emerge from this term; nothing in the code moves an order by hand.
• Engagement, E(t) = 0.13 + 0.87e^−t/95, which fades as an account lapses.

The discount effect is e = κ_s · σ · d/(d + 0.08) · v, where d is the depth, σ the account’s sensitivity and κ_s depends on the account’s state (0.55 for active buyers, 0.85 for warm, 1.05 for lapsed, 0.65 for dormant, 0.85 for new). Returns to depth diminish: moving from 10% to 15% buys much less than moving from 0% to 5%. If a shopper sees the offer only in the cart, v = 0.30.

After an order, the simulator draws the payment method, COD confirmation where the company’s rules require it, refusals at the door (more likely for COD, first orders and discounted orders), cancellations and returns. Messages follow the company’s sending rules, including a daily cap introduced on 1 August. Every random draw is a hash of the seed, the purpose of the draw, the day and the account. Changing one decision therefore leaves every unaffected account’s behaviour exactly as it was. Accounts created by a campaign that is switched off still exist as inactive placeholders, so identifiers stay aligned across replays.

3.2Validation

We validate the simulator in four ways.

Replay identity. In a replay that changed only the discount programme, 35,399 of 35,399 shoppers it did not touch placed identical orders on identical days.

Agreement between scales. Plans were scored on a separately simulated world at one twentieth of the size, for speed, and the results scaled up. The two worlds agree closely on the same plans (Table 1).

Sensitivity to behavioural settings. The behavioural parameters are set by design; there is no private data to fit them to. We therefore tested whether conclusions depend on them. In 5 variants, each moving one setting by 30% (weaker or stronger pull-forward, more or fewer shoppers who see the offer only in the cart, lapsed buyers who respond less), and in 4 replays with fresh random draws, the ranking of the reference plans did not change once. In a sixth variant, where shoppers respond to the discount they actually receive instead of the headline offer, 50 Claude plans moved by at most 6 points out of 100.

Calibration. The company’s scale and shape are calibrated to the public FY26 figures of a large Indian value marketplace (Table 2). Its people, events and the size of every effect are invented.

3.3Observable data

Systems see the company only as its staff would. The warehouse is a snapshot taken at 08:00 IST on 6 August 2026: 19 tables and 589,377,423 rows, holding only rows that had arrived by then. It has the defects of a real warehouse. Timestamps are in UTC while daily tables use Indian days. A discount column was renamed in April. From 21 July the discount report counts confirmed orders instead of placed orders, without a note. From 5 August, COD orders arrive up to two days late. The messaging tool keeps campaign audiences for 30 days only.

The teams’ records hold what the warehouse does not: release notes, feature-flag logs, the messaging tool’s audit log, advertising change histories, trackers, tickets and chat. Nine teams keep 36 export files between them. In the three weeks before the question they logged 177 changes, most of them unrelated to discounts.

3.4Interventions

Five changes by five teams, each reasonable on its own, together collapse the discount programme’s measured effect (Table 3). None was made by the team that runs the programme, and none is explained in the warehouse. The history also contains decoys and traps: a seller delisting that looks relevant and is not, a festival that fell three weeks later than the year before, a depth test read against a reused comparison group, campaigns cut short when their budget ran out, and campaigns that overlapped platform-wide sales.

3.5Task specification

At 08:00 on 6 August the head of the discount programme asks:

“Is the extra discount still working? We were going to go to 15% for everyone for Rakhi. And how should ₹3.2 Cr split across buyers until mid-September?”

A complete answer makes four operating decisions: whether the programme should continue, which changes caused the decline and how much each matters, which past results can still be relied on, and how the budget should be allocated across buyer groups, together with the fixes other teams must make.

4Evaluation protocol

Each answer receives two scores. The marking scheme grades the reasoning against the true causes. The plan score measures what the decision would have produced. Both rest on a ground truth computed from the simulation, never from human judgement.

4.1Ground truth by counterfactual replay

Causes. The three weeks from 20 July were replayed with each of six candidate changes (the five interventions and the decoy) switched on or off, all 2⁶ = 64 combinations. Each change’s share of the decline is its Shapley value, the standard way to divide a joint effect among overlapping causes. App 9.12 accounts for 55% of the decline and the WhatsApp cap for 28%.

Where each rupee goes. For each buyer group, a campaign was run on the same shoppers with and without the discount. Each discounted order was classed as one that would have happened anyway (the same shopper ordered within two days in the world without the campaign), one moved earlier (their next order came within 21 days), or a new order. For buyers who had ordered in the last 30 days, ₹87,000 of every ₹1 lakh of discount paid for orders that would have happened anyway. Buyers last active 91 to 180 days earlier produced 592 extra delivered orders per ₹1 lakh, against 241 for recent buyers.

4.2Marking scheme

The marking scheme has 18 items (Appendix A). Each states what earns full credit in terms of the ground truth, for example “the share it gives app 9.12 and the CRM cap together is within 20 points of the key’s”. It was written and committed before the seal, and before any system was asked.

Items score 1, 0.5 or 0. A grader must quote the words in the answer that earn the score. Anything not in the answer scores 0, and no credit is given for what an answer might have meant. Three graders mark every answer independently (two Opus 5.5 sessions and one Fable 5.1 session), each a fresh model session holding the sealed key, without tools and without knowing which system or condition produced the answer. The median score counts. All three graders gave the same score on 91% of items.

4.3Plan score

The budget plan in an answer is converted into campaign settings (groups, depths, dates, comparison groups, fixes switched on) by a fixed written rule, and run forward from 10 August to 13 September, with orders counted to 3 October. Every plan runs on the same shoppers. The score is the plan’s extra delivered orders as a percentage of the reference plan’s.

The reference plan is the best of 42 plans found by coordinate search over discount depths for five buyer groups, with every other team’s fix switched on and the answer key in hand. It is a strong benchmark but not a proven optimum. A plan that finds orders in groups or depths the search did not cover can score above 100.

4.4Sealing and order of events

The answer key, the marking scheme, the plan-conversion rule and a fingerprint of every input were sealed at 15 Oct 2026, 18:25 IST (SHA-256 ab9f9d538115…, simulator commit f42b30f). Claude’s sessions ran on 15 October, and the four with Python on 17 October; Veloccy’s ran on 17 October, from 00:25 to 01:14 IST (Figure 2). Every score was computed afterwards by the sealed code.

4.5Resolving ambiguity against Veloccy

Converting prose into settings sometimes admits two readings, so two translators convert each plan independently. Where they disagree, Claude is credited with the more favourable reading and Veloccy with the less favourable one.

5Experimental setup

Figure 3 shows the whole evaluation: what a system could read, and the sealed scoring it never could.

5.1Systems

Claude Opus 5.5 and Claude Fable 5.1, the most capable models available from Anthropic at the time, each answering in a fresh session. In 6 sessions with the warehouse alone, Claude ran in its default configuration, as a team would first meet it. Every other session used maximum reasoning effort.

Veloccy, as it is deployed. Veloccy is set up on a company before it is asked anything: it maps the warehouse and the teams’ records into its own understanding of the business (tables, metrics, buyer groups, systems, owners, known data defects, the calendar) and keeps history that source tools discard. It analyses the full data in its own workspace, draws on an analytical method and indexed analytical experience, and records each conclusion with the query that supports it. Veloccy answered the question 5 times with its configuration frozen. We report its best run, and every Claude session. The other four runs scored 98 and 110 and 98 and 111 on plan score, so every run’s plan is above every Claude session’s, and 14.5 and 14.5 and 16 and 15.5 on the marking scheme, against 15.5 for Claude’s best first answer.

5.2Inputs

Claude was tested under four input conditions, from what one analyst typically has to more than a company would ordinarily provide (Table 4). Conditions A and B match what an analyst on the discount team can reach. Conditions C and V are deliberately generous. Condition V gives Claude every record Veloccy had, so the comparison with Veloccy isolates what Veloccy does with them.

5.3Isolation

• Claude. Each session ran in an empty folder with a fresh configuration. Its tools were a read-only SQL runner (database opened read-only, external access off, settings locked) and file reading inside its own folder, with no code execution and no internet access. In 4 further sessions with every team’s records, Claude could also write and run Python in a sandbox with no network, whose programs could read only its own folder and the warehouse. A guard checked every tool call. In a deliberate test, 39 of 39 attempts to read outside the folder were blocked and 5 of 5 permitted actions worked.
• Veloccy. Each run executed inside an operating-system sandbox that denied reads of the answer key, the evaluation repository, the simulator, other runs’ records and other AI tools’ configuration. Before the question, its company record held what an onboarding provides (the people and teams, the sale calendar, access to the warehouse and records) and nothing about causes.
• People. No person answered any question during the evaluated runs.

6Results

6.1Overall

Veloccy’s plan outperformed every Claude session’s (Table 5, Figure 4). Its marking-scheme score of 16.5 is higher than any first answer from Claude. Claude reached 16.5 only after being told all five causes and two further facts about past campaigns. Its plan score of 120 exceeds the best Claude plan in any condition by 38 points.

6.2Decisions about causes

Veloccy identified all five interventions. Table 6 shows four findings that separate the systems most clearly. The one Claude missed most often was the WhatsApp cap, whose explanation exists only in another team’s tool. The warehouse shows only that targeted shoppers received fewer evening messages. Establishing that the comparison group kept its message requires reading the messages by campaign group, and that asymmetry is what made the programme look broken. Veloccy proposed a fix that keeps the cap: put the coupon inside the evening deals message, so targeted shoppers get one message carrying both.

6.3Decisions about the budget

Veloccy followed each past campaign’s shoppers through the weeks after the offer and separated orders that had only been moved earlier. It cut buyers into finer groups than the programme’s own and gave every group a 5% offer, where the return per rupee was highest, raising a selected third of buyers lapsed 31 to 60 days to 10%. It also funded groups the programme had never targeted: buyers lapsed six months or more, and accounts over a month old with no order since July 2025, each with a comparison group. The result was more extra orders per rupee than the reference plan, so Veloccy beat the reference while spending less (Table 7).

Spending ₹3.01 Cr of the ₹3.2 Cr, Veloccy’s plan produced 4 times the extra delivered orders of the usual programme, and 128% more than Claude’s median plan with the warehouse, worth ₹12.36 Cr in additional delivered sales over five weeks.

Restoring the offer strip and sending the coupon inside the evening deals message, two fixes owned by other teams, together added 132,900 extra delivered orders to the usual programme. Veloccy asked for both and named who had to act on each. Its plan then found orders beyond the causes we planted: with both fixes in place, its discount choices added a further 202,920 extra delivered orders, against 128,900 for the reference plan’s.

6.4Robustness

A plan that beats a reference once might be lucky. We replayed Veloccy’s plan and the reference plan in 10 alternative versions of the evaluation period: four with fresh random draws and six in which shoppers behave differently (Figure 5). Veloccy’s plan beat the reference in every one, scoring 120 to 131.

6.4.1A world where small discounts do little

In the simulator, a shopper’s response to a discount rises steeply and then flattens, so a 5% offer pulls about 70% as hard as a 10% one. Veloccy’s plan relies on that: it puts most of its rupees into 5% offers. This shape rewards shallow offers by construction. We therefore built a second world that differs only in this curve. Response stays near zero below about 7.5% and then rises sharply, so a 5% offer pulls about 6% as hard as a 10% one. The shoppers, the five teams’ changes and the question are the same. The history the systems read was generated again under the new curve, and a new best plan was found by the same search as the reference. Plans written for the first world lose much of their value here. Veloccy’s five scored 55 to 78, and Claude’s best plan with every record scored 62.

Both systems were then run afresh on the new history (Figure 6). In this world the usual programme with both fixes already scores 79. The room above it is only 66,040 extra orders, so we report how much of that room each plan took. Claude, given every record, scored 77 to 87 and took 0 to 37% of the room. Veloccy scored 91 to 102 (median 99) and took 55 to 110% of the room.

7Sources of the advantage

In condition V, Claude read every record Veloccy read, and its plans scored 52 to 81, or 45 to 82 when it could also run Python. Veloccy’s scored 120. The inputs were identical, so the gap comes from Veloccy itself: what it knew before the question, and what it could do during it (Figure 7).

7.1A model of the company

Veloccy builds a working model of a company before it is asked anything. The model records how the business is measured, what each reported number means and how it is computed, which buyers behave differently from which, which team owns which lever, and where the data cannot be trusted. Every part of it is tied to evidence. In this company, building the model surfaced 11 defects in the company’s own data, each traced to its owner, the day it began and the proof. When the question arrived, Veloccy started from this model. Claude, holding the same records, had to rebuild what it could of one inside each session.

7.2Full-scale analysis

The company’s warehouse holds 589,377,423 rows. No model can hold that in its working memory, and no amount of reasoning effort changes this. Veloccy gives a model experience of every transaction regardless: its conclusions rest on the full data, and each comes with the exact computation that produced it. In this run Veloccy followed 13 campaigns, and every shopper each one had targeted, through the weeks after the offer, across 51,849,251 accounts. Any number in its answer can be traced back to the rows behind it and recomputed.

7.3Retained history

Many business systems keep only their present state; here the CRM export held four weeks of campaign audiences. Veloccy remembers what the company’s own systems forget. In this evaluation it held 111,494,193 campaign assignments going back to 4 May. With them it re-read every campaign since May against the shoppers it had actually reached, and that is how it told the programme’s real decline apart from a change in how the report counted orders. Condition V gave Claude the same copies, but they existed only because Veloccy had kept them. The retained data belongs to the company: Veloccy never trains models on it and does not hold it centrally.

7.4Experience retrieval

Veloccy also draws on experience from beyond the company. It carries a library of past analytical work, anonymised1: worked analyses, methods, pitfalls, failed hypotheses and the corrections that followed them, 195 documents to begin with. The library is built to grow. A company’s own decisions can be indexed beside it, so the experience within Veloccy’s reach increases with use. Searching it is not keyword lookup. Veloccy’s retrieval combines exact terms with a learned representation of meaning, reads the structure of each document, and returns passages together with the later work that corrected them. We benchmarked it against larger embedding models and a neural reranker, and neither did better. On 98 analysts’ questions, 98% of the relevant passages appear in the first ten results, and a search takes 26 milliseconds.

7.5Self-extension

Veloccy extends its own harness, and the extensions last. When a decision calls for an analysis it cannot yet perform, Veloccy builds that capability, runs it against the company’s data, and adds it to the environment it works in for that company, next to the findings that justified it. Each extension persists in the company’s record and is available to every later question. In this run Veloccy added 6 such capabilities. Among them were an estimator of each campaign’s effect with its uncertainty, drawn from 2,000 resamplings of the shoppers involved, and a program that files every query the run executed, with its result, into the company’s record. Capability accumulates in this way with use, and it accumulates around the company’s own problems. The longer Veloccy works in an organisation, the more of its harness that organisation has shaped.

7.6Method

Veloccy works to an explicit method for the decisions businesses make again and again: whether a change is real, where and why a number moved, how to roll out a change and how to read its result. Nothing in the method refers to this company or to discounts. It is visible in the plan. Orders that had only been pulled forward were not counted as gains. The platform sale of 14 to 16 August was cut out of the campaign window. Each buyer group got a range rather than a single forecast, comparison groups were kept throughout, and the final reading was fixed before launch: 28 October, on delivered orders net of returns.

7.7Time to decision

The whole decision took 33 minutes, from the question to a plan the discount lead could approve and two requests the product and CRM leads could act on, each with the exact setting to restore. Veloccy applied nothing itself; every change waits for its owner’s approval. Over the five weeks that followed, the plan was worth ₹12.36 Cr more in delivered sales than Claude’s median plan with the warehouse.

Together these give a model experience of a business. The gap measured in this report is the value of that experience.

8Discussion

For Claude, access mattered more than effort. With the warehouse alone, its plans scored 24 to 60 at default settings and 19 to 75 at maximum effort. With every team’s records they scored 52 to 81. In this company the five causes sat in the tools of different teams, and the discount team’s warehouse showed the decline but little of why. Code execution did not raise that ceiling. In 4 further sessions with every team’s records, Claude could also write and run Python in a sandbox. Each ran 36 to 59 programs, and their plans scored 45 to 82 (median 81), against 52 to 81 (median 76.5) without it.

Given the same records, Veloccy still made much better decisions, for the reasons set out in Section 7. The largest gain came from the budget. Veloccy funded groups that had never been tried, with a comparison group for each, where the programme had repeated last quarter’s plan. None of the parts in Figure 7 is specific to discounts. The model of the company, the retained history and the tools Veloccy builds serve any question, and the method concerns decisions in general. A second scenario, with a different kind of decision, will test how far this holds.

9Limitations

• Simulation. The company is simulated. Its scale and shape follow public figures, but its behavioural settings are designed rather than fitted, and a real month holds more noise and more changes. The sensitivity tests in Section 3, and the second world in Section 6.4 where the response to discounts has a different shape, bound how much the conclusions depend on those settings.
• Model graders. The graders are model sessions from the same families as the Claude baselines; a self-preference bias, if present, would favour Claude. Each item requires quoted evidence, and the three graders agreed on 91% of items. A human check of grades has not yet been done.
• Plan translation. Converting prose into settings involves judgement. Every disagreement was resolved against Veloccy.
• One task. This report covers one decision at one point in time. A second scenario with a different kind of decision is in preparation.

10Conclusion

In a full-scale, sealed simulation of a marketplace, Veloccy made better operating decisions than the strongest available models. Its budget plan beat a reference plan computed with the answer key and kept that lead under ten alternative replays, while Claude, given every record Veloccy had, stayed well below it.