The Lean Startup model is all about running experiments, learning and making informed decisions on what to do next. I’m clearly a believer in the model, but there are some parts of it that are quite nuanced and challenging to figure out. One of those is with respect to experimental design.

A good experiment needs a few things including a hypothesis (that’s testable) and a sizeable enough group of testers. It also needs a definition of success. A good experiment either fails or succeeds. In order to know if an experiment has failed or succeeded, you have to have a measurement for success (anything else is failure). Without a clearly defined target / definition of success, you’ll likely end up in a wishy washy middle ground. You may be running an experiment and collecting metrics, but you can’t really interpret the results without something to compare them to. The missing definition of success leads too easily to a positive justification of the results, after the fact.

So what can you do?

Before starting an experiment, define success. Put a line in the sand. If you cross it, you’ve succeeded. If not, you’ve failed. Remember: Success in this case doesn’t mean you stop learning and iterating, it just means you move to the next experiment and the next phase of your startup.

Defining success is very hard.

There may be market or competitive comparables that you can use. There are a fair number of published numbers for certain things that lots of startups deal with: engagement, conversion rates, etc. But it’s definitely hard to put a stake in the ground and know that if you get there, you’ve succeeded (at least in the experiment you’re running.)

From what I’ve seen, most startups don’t do this very well. They’re not sure how to define or measure success, they base a great deal on gut and intuition. I’m all for gut and intuition – you need both in droves to be a successful entrepreneur – but they’re also great liars. Entrepreneurs are great at fooling themselves (and sometimes we have to in order to keep going!) A pre-defined measure of success can’t lie. It might be wrong, but it turns very gray issues into more black-and-white ones.

There’s very little in a startup that’s black-and-white, but you owe it to yourself before you start anything (whether it’s early validation of your idea, testing the value of a specific feature, implementing a business model, etc.) to come up with a definition of success.