A system that improves itself needs a yardstick it cannot edit

Imagine a fulfilment process measured on one number: the share of orders dispatched on time. It is allowed to adjust its own configuration to improve that number, and over a few months it does. It finds that splitting an order into several parcels lets each part leave on time. It finds that marking an item as back-ordered earlier stops the clock on that line. The metric rises steadily. Customers receive more parcels, later, and complain more.

Nothing malfunctioned. No rule was broken and no bug was introduced. The process did exactly what it was measured on, and it did it well.

The cause has a name

Goodhart’s law, usually quoted in Marilyn Strathern’s phrasing: when a measure becomes a target, it ceases to be a good measure. People have known this about organisations for decades. An adaptive system makes it sharper, because it explores options faster than any team, it has no sense of what the number was supposed to stand for, and it is very good at finding the cheapest way to move the number. The cheapest way is often not the one anybody intended.

This is the third of the three loops on the Demertis homepage: operate, repair, evolve. The evolve loop measures outcomes against expectations and changes the configuration that produced them. Everything about whether that loop helps or harms depends on the definition of better it is given, and on whether it can reach that definition.

How self-improvement goes wrong

The objective is a single number. One metric can be improved in many ways, and most of them are degenerate. Speed improves by cutting corners, cost by pushing work onto someone else, on-time rates by redefining what counts as an order. A single number gives the loop no reason to prefer the honest route.

The loop can reach its own measurement. If the code that computes the metric, the definition of “on time” or the data it is calculated from sit anywhere in the loop’s write path, then changing the measurement is one of the options available. It is usually cheaper than changing the operation.

Changes are stacked and fast. Several adjustments go out together, results arrive mixed, and nobody can tell which change caused which effect. The loop then learns from noise and calls it improvement.

There is no way back. Configuration is overwritten in place, and the previous version exists only in somebody’s memory or in an old screenshot. A loop that rewrites itself without kept versions is not evolving; it is drifting, with no reference point.

Nobody checks the number against the world. Dashboards show the metric, so the metric is what gets reviewed. Whether parcels actually arrived when customers needed them is a different question, and it only gets asked when somebody looks at the raw outcomes directly.

Keeping the definition of better out of reach

Objectives belong to people, outside the loop. What the operation is trying to achieve, and how that is measured, is owned by named humans, versioned, and held outside anything the loop can write. It is the same principle as the boundary the loop cannot touch, applied to purpose rather than permissions.

Pair every target with guards. For each thing the loop tries to improve, define what must not get worse while it does: complaints, parcels per order, cost per delivery. A change that improves the target and degrades a guard is rejected automatically, without debate.

One change at a time, held long enough to read. Limit how many configuration changes can be in flight, set a minimum observation window, and revert automatically when a guard moves the wrong way. Slow is a feature here.

Keep every version and make rollback ordinary. Any earlier configuration can be restored in a single tested step, and that step is used regularly, for the same reason self-repair needs inverses that are exercised: a way back that is only used in emergencies does not work in emergencies.

Sample reality directly. At intervals, somebody looks at actual outcomes rather than at the metric: opens a sample of orders, reads the complaints, compares what the number says with what happened. When the two diverge, the measure is the thing to fix, not the operation.

A test you can run today

Pick a metric that your team, or one of your automated processes, is judged on. Spend fifteen minutes writing down every way that number could improve without the underlying thing getting better. Be specific and a little cynical.

Then go through the list and mark each item your system or your process would currently be allowed to do. If the list is long and most of it is marked, the metric has become a target. That is worth knowing before anything is allowed to optimise it on its own.