An automated system can look profitable on the screen and still lose money on the ledger — quietly, for years. This is the account of one we built, tested to destruction, and shut down on purpose. The method that shut it down is the method we bring to every automation we build.
A long-only spot crypto bot. It read two timeframes, scored each candidate against nine technical indicators, combined them into a single confidence number, and placed a bracketed order — fixed stop, target at twice the stop — whenever a coin cleared the threshold. Two processes: one scanned every pair on the exchange around the clock; the other checked portfolio state and executed. Risk was capped in dollars, not guesswork.
It ran unattended. Some days were green. On a chart of its equity you could talk yourself into it.
Every losing automated strategy has good weeks. The failure mode that matters is not a crash — it is a system that runs cleanly, produces plausible numbers, and bleeds slowly enough that you keep adjusting parameters instead of asking the harder question.
So we did not ask “how do we make it better.” We asked “does it have an edge at all, and how would we know.” Everything below was decided before looking at the answer.
Every hypothesis was written down first: the change, the direction it should move the result, and the exact pass/fail criterion. A result that came out backwards was recorded as a rejection, not reinterpreted as a discovery.
The bot only ever executes a fraction of what it sees. We logged every signal at the moment it was evaluated — roughly 24,000 matured signals — so we could measure the ones it didn’t take, not just the handful it did.
A sequential probability ratio test ran continuously against two hypotheses — “has an edge” vs “doesn’t” — so the decision to stop was a threshold crossing, not a mood.
Twenty-five improvements — better filters, adaptive exits, volatility sizing, regime gates — each pre-registered, control-tested, and recorded when it failed, so none of them could quietly come back.
| Idea | Pre-registered test | Result |
|---|---|---|
| Dynamic indicator weights | Forward + shadow | failed — closed |
| Higher confidence threshold | Yearly, all regimes | cost 3.5× return |
| Multi-day time exit | Cache replay, 3–10 d | −$77 to −$152 |
| ATR trailing exit | 72 real trades, OOS | one lucky coin only |
| Cross-sectional momentum | 713 pairs, 20-config grid | failed 3 of 4 criteria |
| … 20 more | … | documented |
At a 1:2 payoff, fees included, the strategy needed to win 41.3% of trades to break even. It won 31.7%. Expectancy was −$0.29 per trade — close to what buying at random into an uptrend and hoping would produce.
The nine-indicator confidence number — the core of the system — had a rank correlation with actual outcome of about 0.03. It could not tell a winning trade from a losing one. Higher-confidence trades did not win more often. The gate was theatre.
Four independent reviews — our own analysis, an outside decision panel, and two rounds of deep research — converged on the same conclusions:
One coin accounted for the entire visible profit. Remove it and every version of the strategy, under every exit rule we tried, was underwater.
Live trading was switched off. The stop criterion had been written in advance — hit any one of three thresholds and the project ends — and it was honoured rather than argued with. The code and every test are archived. The failure mode is written down in plain language so it cannot be rediscovered from scratch in six months.
This was not a disappointing outcome. Most people who run a system like this never reach a clear answer — they keep losing, slowly, and call it variance. We reached the answer in seven weeks, on paper, for the cost of the research and none of the capital.
Most automation is sold on “it works.” We would rather show you the measurement. Whatever we build for you — a pipeline, an integration, an agent — ships with the same discipline: the benefit stated as a hypothesis, an honest test of whether it holds, and a number instead of a claim. If it doesn’t hold, you find out early, from data — not two years in, from your bank balance.