Seth Godin today writes:
Netflix tests everything. [...]
Except they didn’t test the model of renting DVDs by mail for a monthly fee.
And they didn’t test the idea of betting the company on a switch to online delivery.
The three biggest assets of the company weren’t tested, because they couldn’t be.
Sure, go ahead and test what’s testable. But the real victories come when you have the guts to launch the untestable.
I was there. To the extent that these could be tested, they were. The company started out as an a-la-carte DVD rental-by-mail business. The a-la-carte and subscription models ran side by side for a while. No, they weren’t controlled A/B tests, but there was ALWAYS a rigor around metrics and what worked best.
You can’t test everything, and to Seth’s point, if you wait to launch something that hasn’t been tested, you may miss out on a great innovation. But for the risk you take, there’d better be a good reason not to test. Because if you don’t take the “test everything” credo to heart, you run the risk of being intellectually sloppy and justifying it as bravery. And you could cost yourself a lot of money and a lot of opportunity that way.
Some decisions need to be made with very little data, but there’s always SOME data. Don’t let yourself off the hook without knowing you’ve given big decisions the intellectual discipline they’re due.
Edited to add: Additional perspective from Bob Warfield – Seth: Netflix Avoided the Tragically Knowable