Defining Success

The Lean Startup model is all about running experiments, learning and making informed decisions on what to do next. I’m clearly a believer in the model, but there are some parts of it that are quite nuanced and challenging to figure out. One of those is with respect to experimental design.

A good experiment needs a few things including a hypothesis (that’s testable) and a sizeable enough group of testers. It also needs a definition of success. A good experiment either fails or succeeds. In order to know if an experiment has failed or succeeded, you have to have a measurement for success (anything else is failure). Without a clearly defined target / definition of success, you’ll likely end up in a wishy washy middle ground. You may be running an experiment and collecting metrics, but you can’t really interpret the results without something to compare them to. The missing definition of success leads too easily to a positive justification of the results, after the fact.

So what can you do?

Before starting an experiment, define success. Put a line in the sand. If you cross it, you’ve succeeded. If not, you’ve failed. Remember: Success in this case doesn’t mean you stop learning and iterating, it just means you move to the next experiment and the next phase of your startup.

Defining success is very hard.

There may be market or competitive comparables that you can use. There are a fair number of published numbers for certain things that lots of startups deal with: engagement, conversion rates, etc. But it’s definitely hard to put a stake in the ground and know that if you get there, you’ve succeeded (at least in the experiment you’re running.)

From what I’ve seen, most startups don’t do this very well. They’re not sure how to define or measure success, they base a great deal on gut and intuition. I’m all for gut and intuition – you need both in droves to be a successful entrepreneur – but they’re also great liars. Entrepreneurs are great at fooling themselves (and sometimes we have to in order to keep going!) A pre-defined measure of success can’t lie. It might be wrong, but it turns very gray issues into more black-and-white ones.

There’s very little in a startup that’s black-and-white, but you owe it to yourself before you start anything (whether it’s early validation of your idea, testing the value of a specific feature, implementing a business model, etc.) to come up with a definition of success.

July 7, 2011 Posted in Customer Development by

  • http://twitter.com/mikkojarvenpaa mikkojarvenpaa

    I appreciate the notion of getting the most out of experiments. This is the crux of the lean model.

    But since we often insist on comparing the lean model to the scientific model and construct experiments where we want to test our hypothesis, maybe success the wrong thing to define for experiments? Maybe it is more important to define the failure of the experiment. Or in terms of the scientific model, maybe we should be keener to falsify a hypothesis than to verify it? This would also force us to keep an open eye towards hypotheses that are unfalsifiable.

    Further, defining success of a hypothesis can be dangerous for an entrepreneur. It can lead to a positive selection bias in collecting and analyzing the evidence (i.e. if you want the experiment to succeed), or it can drive the experimenter to conduct the experiment for longer than is required. The history of scientific experiments is rife with such cases.

    What you are saying does not exclude this possibility, and maybe this is just the flipside of the coin I am presenting here. 

  • http://www.instigatorblog.com Benjamin Yoskovitz

    Thanks for the comment. Perhaps identifying failure is a more rigorous scientific parallel, but it’s a much more depressing way of handling things, which I don’t think is necessarily good in an entrepreneurial endeavor.

    I do think there’s a risk in defining success that’s too easy to achieve, and I agree that any experiment has to be falsifiable, otherwise you’re just lying to yourself. That’s why I stress that everyone has to agree on the measurement of success, which hopefully sets the bar higher. If you have strong mentors (with experience) they can be your intellectual honesty meter to determine if you’re just making an experiment that’s too easy or can’t fail.

  • http://www.ContentEqualsMoney.com Amie Marse

    I totally agree. And for those of who tend to stay too long on new ideas (myself included) I think it is wise to have multiple stepping stones. For example, if a new marketing technique yields x then we keep it, if it yields x+y then we throw more money at it, if it yields x+y+z we develop the idea to showcase it further next quarter. Otherwise, I think we as entrepreneurs tend to add variables later on and justification to keep things we shouldn’t. Or maybe that is just me :)

  • Me

    10k a month, personal defined startup line of success

  • Signs

    I agree with you. To define success is somewhat very hard as hard as the ways on it. Anyway, you have got the point. “There may be market or comparable competitive that you can use. There is a reasonable number of figures for some things, many startups face: commitment, exchange rates etc.” Signs

  • http://www.customessays.co.uk/dissertation.php dissertations

    Blogs are so interactive where we get lots of
    informative on any topics…… nice job keep it up !!

  • Shahnawaz Sadique

    good post but in my opinion nobody can define success actually its vary from person to person

    steel price in india

    government jobs in india

  • http://profiles.google.com/nabeelshaukatbutt Nabeel Shaukat

    Success comes after confidence. Confidence is the first step on the road of success and leadership development. If you understand this you can succeed. 

  • http://www.ithakaleadership.co.uk/ Leadership Development

    Incredible article. I think success is everything that you achieve in life. No matter how big it is. Drops make the ocean. So start from small achievements for a bigger one. 

  • Pingback: Finding Your Startup’s One Thing

  • Pingback: Twitter Link Roundup #97 – Small Business, Social Media, Design, Copywriting, Marketing And More « crowdSPRING Blog

  • Pingback: Twitter Link Roundup #97 – Small Business, Social Media, Design, Copywriting, Marketing And More

  • Pingback: How Many Metrics Do You Need to Run Your Startup?

  • Pingback: How to Use a Single Metric to Run Your Startup

  • Pingback: Metrics versus Goals « iHack, therefore iBlog

Ben Yoskovitz
I'm VP Product at Codified (makers of VarageSale).

I'm also a Founding Partner at Year One Labs, an early stage accelerator in Montreal. Previously I founded Standout Jobs (and sold it).

My bio »

or

Follow me on Twitter

Get updates and special content
When I publish new content, get it directly in your inbox. Subscribers will get special stuff as well not available on the blog (but I promise it will be infrequent + high quality.)
Get the Lean Analytics Book!
Awesome Jobs
Check out the job opportunities at my portfolio companies.
Startup & Investor Resources
Find Stuff
My Photos