Estimating an ML experiment

How to estimate an ML experiment: it's research with engineering attached — the model is easy, the data is the work. How to size a budget, not a forecast.

The model is the easy part. The data is the work.

An ML experiment is research with engineering attached. The model architecture is usually an off-the-shelf choice, and training is a known pattern. The work is upstream: getting clean training data, defining the evaluation metric the team will live or die by, building the pipeline that takes the model from notebook to production. The first 80% of the experiment is data work; the last 20% is the model.

Estimating an ML experiment with the feature deck is the same trap as estimating a research spike — you don’t know what you’ll find. The estimate has to be a budget for the experiment, not a forecast of the result. If the result is “the model doesn’t work,” that’s still a successful experiment that consumed the budget; the team can’t pretend that outcome makes the points wrong.

What gets said in the room

ML engineer: “Training is a day. The model is standard.”

Lead: “Where’s the training data coming from?”

Data: “We need to label about 5,000 examples first.”

PM: “What metric tells us the experiment succeeded?”

SRE: “If it works, what does shipping it look like? Inference latency? Cost?”

Questions worth asking before voting

  • Is the training data ready, or is labelling part of this story?
  • What’s the evaluation metric, and what’s the baseline to beat?
  • Notebook-only experiment, or end-to-end including a serving path?
  • What’s the time-box, and what’s the deliverable at the end of it?
  • If the model works, what’s the path to production — and is that in scope?
  • Cost of training and inference — does the team have the budget?

If labelling and a serving path are both in play, they’re not one story — split the data work, the experiment, and the productionisation, and size each on what it actually is.

Budget the experiment, don’t forecast the result. A model that fails is still a result the points paid for.

Like estimating a research spike and estimating a prototype, the deliverable is knowledge first. See the other worked estimation examples, or open a free planning poker session when the experiment has shipped its result.