What’s the best way to measure product usage?

Donald T. Campbell (American social scientist)
There’s a Name for the Big Flaw in Our Obsession With Assessment and Metrics

The more a given metric—say, a national college ranking—is used to evaluate performance in some domain, the less reliable it becomes as a measure of overall success. Why? The people whose performance is being measured will neglect other parts of their job just to focus on boosting the relevant numbers, sometimes to the point of cheating.

What’s the best way to measure product usage?

David Jackson (Founder, Seeking Alpha)
Chartbeat’s suggestion for how to measure content quality | A Founder’s Notebook

Sites that optimize for pageviews inevitably publish provocative headlines and populist content. But time on page isn’t a solution, because shorter articles aren’t necessarily lower quality or less valuable.

Ben Erez (Product at Breeze)
22 Mistakes I Made as a First Time Founder — Viabilify

In hindsight, seeking positive feedback was toxic. It was toxic because it was giving us the impression we were on the right track. I mean, we were on the right track, but we were moving at 1 mph on a track built for bullet trains. Now I know that shipping product, getting sales and not running out of money are the most important things for a B2B startup.

Danielle Morrill (Co-Founder & CEO at Mattermark)
Revenue vs. Value | Danielle Morrill

While revenue is a useful signal to founders, indicating they are creating something people want, it is also a lagging indicator of success. By the time a startup has a predictable and steadily growing revenue stream that means it has built a product and brought it to market successfully.

Brian Balfour (Co-Founder, CMO @ Boundless)
Avoiding The Wheel Of Meaningless Growth

There is a cycle that plagues a lot of companies. It typically works like this: 1. A startup wants some press, so they look for some bloated number to give the writer (downloads, registrations, visits, etc). 2. The startup then celebrates that press article, internally supporting the message that the bloated metric is worth pursuing. 3. In order to get additional press hits, the startup needs to increase that bloated number, so they focus on incr… (read more)

Alistair Croll (Founder of UEM, Author of Lean Analtyics)
The One Metric That Matters | Lean Analytics Book

Transactional sites are about shopping cart conversion, cart size, and abandonment.
Collaboration is about the amount of good content versus bad, and the percent of users that are lurkers versus creators.
SaaS is about time-to-complete-a-task, SLA, and recency of use; and maybe uptime and SLA refunds.
Media is about time on page, pages per visit, and clickthrough rates.
Game startups care about Average Revenue Per User Per Month and Lifetime … (read more)

Josh Kopelman (Partner at First Round)
Founder Office Hours With Chris Dixon And Josh Kopelman: Schedit | TechCrunch

“The real data is retention and repeat usage. ” Startups that focus on the real metrics can make their products better, attract more customers, and make them happier.

Brian Balfour (Co-Founder, CMO @ Boundless)
Avoiding The Wheel Of Meaningless Growth

The one metric that guides us like our north star is Weekly Active Users. We defined this as our authentic growth metric using a few criteria: 1. Retention. In the consumer world things like Daily Active User (DAU) and Weekly Active User (WAU) are most commonly used. Celebrating metrics such as total registrations or downloads over time tell you nothing about whether or not you are building real value. 2. Meaningful interaction. Qualifying events… (read more)

David Jackson (Founder, Seeking Alpha)
The key metric for your startup must satisfy these 4 criteria | A Founder’s Notebook

Reporting metrics as growth rates (or absolute growth) focuses the company on growth, and encourages everyone to think big.

Donald T. Campbell (American social scientist)
There’s a Name for the Big Flaw in Our Obsession With Assessment and Metrics

The more a given metric—say, a national college ranking—is used to evaluate performance in some domain, the less reliable it becomes as a measure of overall success. Why? The people whose performance is being measured will neglect other parts of their job just to focus on boosting the relevant numbers, sometimes to the point of cheating.

Josh Elman (Partner at Greylock Partners)
The only metric that matters — Medium

I always ask the same question: How many people are really using your product? You need a metric that specifically answers this. It can be “x people did 3 searches in the past week”. Or “y people visited my site 9 times in the past month”. Or “z people made at least one purchase in the last 90 days. ” But whatever it is, it should be a signal that they are using their product in the way you expected and that they use it enough so that you believe… (read more)

Mariya Yao (Founder at Xanadu)
The key success metric for mobile apps | A Founder’s Notebook

A popular metric for measuring retention in the mobile games industry is DAU / MAU, or daily active users divided by monthly active users, and I highly recommend that consumer-facing mobile app developers keep track of that metric as well.

David Jackson (Founder, Seeking Alpha)
The corrosive impact of pageviews as the target metric for content websites | A Founder’s Notebook

While quality positively impacts pageviews, there are cheaper levers that are far more powerful. This leads to volumes of low quality content, with no attempt to create or surface the “great stuff”.

David Jackson (Founder, Seeking Alpha)
Why websites shouldn’t optimize for page views | A Founder’s Notebook

Our key metric at Seeking Alpha is daily, direct users. “Daily” gives you credit for returning visitors, and “direct” only counts people who come for your product and brand, not people who came because they were enticed to click on a syndicated or shared headline.

David Jackson (Founder, Seeking Alpha)
The best startup metric: Share of habit? | A Founder’s Notebook

Tom Tunguz argues that share of habit is a better metric for startups to focus on than engagement. Share of habit, however, has disadvantages. You can’t measure it if you don’t have access to accurate market size data and who has that? And its not a good operating metric, as its impacted by external factors out of your control. In that respect, its a vanity metric.

Fred Wilson (Co-Founder and Partner at Union Square Ventures)
30/10/10 – AVC

I call this ratio 30/10/10 and so many services that we see exhibit it within a few percentage points here and there. Here’s how it works: 30% of the registered users or number of downloads (if its a mobile app) will use the service each month. 10% of the registered users or number of downloads (if its a mobile app) will use the service each day. The max number of concurrent users of a real-time service will be 10% of the number of daily users.

Paul Graham (Co-Founder & Partner at Y Combinator)
Do Things that Don’t Scale

We encourage every startup to measure their progress by weekly growth rate.

Paul Graham (Co-Founder & Partner at Y Combinator)
Do Things that Don’t Scale

Focusing on hitting a growth rate reduces the otherwise bewilderingly multifarious problem of starting a startup to a single problem. You can use that target growth rate to make all your decisions for you; anything that gets you the growth you need is ipso facto right.

Paul Graham (Co-Founder & Partner at Y Combinator)
Startup = Growth

A good growth rate during [Y Combinator] is 5-7% a week. If you can hit 10% a week you’re doing exceptionally well. If you can only manage 1%, it’s a sign you haven’t yet figured out what you’re doing. We usually advise startups to pick a growth rate they think they can hit, and then just try to hit it every week.

Fred Wilson (Co-Founder and Partner at Union Square Ventures)
Social Commerce Is Commerce With A Social Layer – AVC

Conversion rates are critical. They tell you what systems perform best for the end user. When a system converts north of 5% of users visits to a transaction, it is working extremely well for the end user. When a system converts 0.1% of user visits to a transaction, it doesn’t work as well for the end user.

Anamitra Banerji (Partner @ Foundation Capital)
The Shape Of The Curve — Medium

Most consumer software products have a DAU:MAU of 20% or lower. Zero products get to 100% but some rare ones come close to this ceiling (like Facebook 65% and Whatsapp 72%) [i.e. you should be getting better than 20% DAU:MAU]

Eric Ries (Author, The Lean Startup)
The Lean Startup: How Today’s Entrepreneurs Use Continuous Innovation to … – Eric Ries – Google Books

The most important thing to measure is behavior: would the early participants actively spread the word?

Paul Graham (Co-Founder & Partner at Y Combinator)
Do Things that Don’t Scale

That’s a reasonable proxy for revenue growth because whenever the startup does start trying to make money, their revenues will probably be a constant multiple of active users

Fred Wilson (Co-Founder and Partner at Union Square Ventures)
Having Empathy For Your Users – AVC

I feel like the companies we meet with and work with generally do a good job of instrumenting their products and collecting on data on what is working and what is not working. But they often don’t have good answers for why the behavior they are seeing is happening. It’s hard to fix something you know is broken unless you understand why it is broken.

Fred Wilson (Co-Founder and Partner at Union Square Ventures)
The Behavior Of Your Users Normally Doesn’t Change Overnight – AVC

I tell this story because we all encounter this sort of thing along the way of building and launching and growing a product. We make tweaks and something changes right away. That immediate change is usually related to something that brought traffic (google, twitter, rss, email, appstore) and not a design change. More gradual changes (up or down) are usually because of design changes. There’s a difference between these two kinds of effects and it … (read more)

Fred Wilson (Co-Founder and Partner at Union Square Ventures)
Growth vs Retention – AVC

Entrepreneurs always ask what the one number they should focus on for raising money. I always say “90 day retention numbers for your acquisition cohorts”. There’s a common view in silicon valley and around the tech sector that growth is the one thing you should focus on. But it’s hard to grow if you are churning your users. And if you are paying for user acquisition, as many startups do in search of growth, then retention/churn becomes even more … (read more)

Vasu Vadlamudi (Director of Product, Oscar Insurance)
Growth vs Retention – AVC

For early retention, if your product can be used by a normal user on a daily basis, the “40/20/10” bar is a solid goal. (Of 100 installs on D0, 40 return on D1, 20 on D7, and 10 on D28.)