What a sandbox market can and cannot tell you

Run a pricing and purchasing loop against a year of real market data and the results can look excellent. It buys ahead of price rises, sells into demand, and holds less stock than the people who used to do the job. The charts are clean and the method is sound: real prices, real volumes, point by point.

Then somebody asks the obvious question. In that year of replayed data, did any price ever react to the loop? Did a supplier notice that orders always arrived on the same weekday and adjust its terms? Did a competitor see the price cuts and follow them? None of that could happen, because the market in the replay was a recording. The result describes how the strategy performs in a world that does not know it exists.

Why the sandbox exists, and what it removes

In this programme the sandbox is not caution for its own sake. A machine cannot agree to anything; any contract it appears to make is attributed to whoever operates it, which is the argument of can a machine own anything? So economic behaviour is studied with real market data and no real counterparties, and the homepage says so.

The difficulty is that counterparties are most of what makes economic behaviour risky. A market is a set of participants who respond to each other. Replaying it without them removes the reflexive part, the part where your actions change the conditions you are acting in. What is left is a good test of arithmetic and a weak test of behaviour.

What a sandbox quietly assumes

That your actions do not move anything. Every order fills at the recorded price, whatever its size. A buyer large enough to matter will move prices against itself, and the replay will never show it.

That nobody adapts. In the recording, suppliers keep their terms and competitors keep their prices. Real counterparties learn, and an automated buyer with regular habits is easy to learn and easy to exploit. A loop that has only ever met passive opponents has not been tested against the ones it will meet.

That everything settles. Each order is delivered, each invoice is paid, each delivery matches what was ordered. The failure paths of a real transaction, such as partial deliveries, cancellations, disputes and late payments, are never exercised, so the code that handles them is never tested either.

That the past was knowable at the time. Historical data is often revised: corrected prices, restated volumes, late records. Replaying the cleaned version gives the loop information it would not have had, which is the most common way for a backtest to look better than reality.

That a deal is a price and a quantity. Real transactions come with terms, warranties, payment conditions and returns. A sandbox that models only price is testing a much simpler business than the one it is meant to represent.

Using a sandbox honestly

None of this is an argument against sandboxes. It is an argument for being specific about what they establish.

Put the assumptions next to the result. Not in an appendix: in the same sentence. “In replay, with no market impact and passive counterparties, the loop held less stock” is a result. “The loop held less stock” is a claim the experiment did not test.

Add opponents. Simulated counterparties whose only job is to take advantage of the loop: a supplier that raises prices when it sees urgency, a buyer that probes for the lowest acceptable offer. If the strategy survives them, that is worth knowing. If it does not, that is worth knowing sooner.

Inject failure. Orders that are only partly delivered, invoices that go unpaid, contracts cancelled halfway through. The point is less to see the loop succeed than to see what it does with an outcome it did not plan for, and whether that response stays inside the boundaries drawn for it.

Measure sensitivity, not performance. Add a little slippage, a delay, a competitor that copies every move, and watch how far the result falls. A strategy whose result collapses under small disturbances has told you the most important thing about it.

Use only what was knowable. Point-in-time data, as it existed on each date, before revisions.

Read this way, the sandbox works best as a filter. It is good at showing what not to do and weak at proving what will work. That is still a great deal, as long as nobody mistakes one for the other.

A test you can run today

Take any simulation, forecast or backtest your organisation relies on to make a decision: a pricing model, an inventory policy, a capacity plan. For each decision the model recommends, write one sentence describing how the people on the other side would react to it.

If you can write those sentences easily, check whether the model knows about them. If you cannot write them because the model never needed to know, its result describes a market without anybody else in it. That is a useful thing to have, provided it is labelled as such.

Economic behaviour is one of the six components of the Demertis programme, and the sandbox is where it stays until the legal questions in front of it are answered.