DISQUS

Ian McAllister · 1 year ago

Great points. In my experience, if you're not watching a metric (e.g. content creation, unique users, service latency) it is probably going South, for a mature business, or at least not growing fast enough, for a startup. It also prevents you from noticing when specific changes you make have a positive or negative effect. You end up building new features just assuming the old ones are working as desired and miss out on opportunities to grow your business by fixing/improving existing ones.

Albert Lai · 1 year ago

That was a fantastic post.

Mind you, as you know, being a founder of a viral analytics company, I'm a bit biased. ;)

Keep up the great writing Andrew.

albert
www.kontagent.com

Andrew Chen · 1 year ago

albert, nice website plug ;-)

simonhk · 1 year ago

Great post ! It's always a tough one - as an overloaded startup, spending time on things that don't seem to directly affect user experience seems like a bad idea when you have a million things to code.

Yet agree it's foolhardy to not have a good understanding about _how_ people are using your service - why spend time developing a feature that people don't or don't know how to use, or know what things you need to fix / or even change direction.

So at the end of the day, have to view analytics development not as a tax, but as an investment..

Shayan Zadeh · 1 year ago

I would even go further and say if you don't have analytics baked in it's like driving blind. think about a car without a dashboard to tell you that the car is running out of gas in 3 miles or is going to be pulled over by the next state trooper.

you can launch a product without it and some things about it might work. but you won't know what's clicking, what's not. where should you focus next. etc...

in our case, we make almost every decision for Zoosk based on the data we collect. If we don't know the answer to a question, we instrument it first, wait for 2 hours (or a day sometimes!) and make a decision based on data. then if it's important enough we trend it over time and revisit.

the beautify of web development is that your feedback loop is immediate. we have had cases that nobody knew exactly how users behaved in a certain portion of the product. so we released an instrumented version and 10 minutes later we had pretty good estimates about how to improve product virality based on user behavior.

long story short, I would say don't release anything that you think is important without somehow measuring it's impact almost immediately. and if it's not important enough to measure, maybe you shouldn't be building/releasing it in the first place.

Andrew Chen · 1 year ago

great points, and good idea for how to step into analytics... I think you can generally go through a couple steps, from most lazy:

1) keep a row in a flat file log (or DB) which you use grep and wc to count, or select queries in the latter case
2) write a script file that runs a multi-query report and spits it out into a flat file
3) same as #2, but hook it up into a web page
4) then the final option is to check it into your codebase and integrate it with the live site, so the queries can be pulled in real time
5) after that, then you set up a slave DB and run it on a separate machine so that your queries don't overload the main server
....
100) run hadoop/cassandra/etc and write all your queries in mapreduce format ;-)

scottshapiro · 1 year ago

Great post!
I see a lot of myself in this post and will definitely reference it in the future when I need some grounding.

Rajat · 1 year ago

To put things into perspective, I have tried to give examples of the granularity to which you can go for measuring your performance + size of the company (at which it is realistic to expect).

Disclaimer - This is my personal experience, so it may differ greatly for you.

For startup, highest level metric is -
Revenue, Expense, Money in bank left.

2nd level is - This is required for most startups (>2 employees)
Revenue -> 1-3 major revenue sources and what they are doing WOW, MOM (if your revenue is directly tied to traffic coming to your site, then you must have a number for traffic)
Expense -> Break it into 1 to 3 major costs - hopefully, tied directly to revenue lines (may not be from an accounting perspective but from a team setup, it is ideal).

3rd level -> Companies with 10-200 people (pick an area that is the focal point for your company)

Assuming that revenue is directly proportional to traffic to your website (e.g. CPM/CPC advertising etc) -
Break traffic into channels -
1) Free Search
2) Sponsored Links
3) Email
4) Inbound Links - Affiliate, Datafeed, Referral
5) Direct
Measure WOW, MOM trends by each channel, measure overall Cost per visit, leads ($) per visit.

4th level - This is mostly for bigger companies.
Now at this level, you can analyze each channel in detail to decide where to invest your time and money + which ones have the highest potential to growth.

Lets pick a traffic channel say, Sponsored Links. Now, you can measure -
1) Life Time value for a channel - Most important metric in my view
If you don't have a mechanism or sufficient history to calculate life time value, measure the following (its better not to measure it then have it wrong).
2) Page Views per visit for the channel
3) Cost per visit for the channel
4) Revenue per visit for the channel

5th Level - Be a guru
Start looking at each channel DOD and determine user's click behavior, past and post activity etc to determine what you can do to keep and grow the customer base.


Bets of luck.

Andrew Chen · 1 year ago

I want to add that Rajat's advice is great, particularly for ecommerce. Anyone looking at stats would be well-served to read the above comment and apply it to their particular industry. Obviously some social media metrics would be different, but I agree with the general premise.

Gates VP · 1 year ago

Great post Andrew.

From a development standpoint, I think it's important to focus on this as very "TDD-ish".

Instrumenting "as you go" can really help deliver at least some fundamental data. Obviously, the Google Analytics are cheap and free, but some things are "almost free" and that's always the first step.

Take a simple one, you have a client reporting screen with filters. You're already running this reporting query, but why not track the filters that are actually being used? The same goes for "timestamping" creation and modification times of primary entities or even just writing "triggers" to start tracking history on this stuff.

A lot of useful "how often does X happen" data is just disappearing into the aether, even though tracking it can just be built into the framework without the 25% "product tax".

Don't get me wrong, you'll still incur that tax as you grow, but it's been my experience that most young companies do too little instrumentation not too much.

bradcooper · 1 year ago

Hey Andrew, I love your blog and usually agree with almost everything you have to say. Keep it up, good stuff!

But, sorry if I have to disagree with you on how hard analytics is/should be and how many resources it takes in product development process. I've never heard of it called a "tax"... i like "investment" from the other poster, but I prefer to call it "air" or "water"....it's simply fundamental. In an organization with experienced web developers and marketers, analytics isn't even a discussion topic or issue... it's core to business and simply must be done. In 1999, with log file analysis and everything being still new to many folks in a company, yes, it consumed resources and was very expensive (~$5,000 a month webtrends bills). Today, it shoud be no more difficult (for a company of any size) than making sure every employee's phone works or that everyone has benefits or that everyone has Microsoft Office installed (though i realize even these things can be challenges for a start-up, but so is everything, right?) ;-)

To illustrate how easy it can be, just yesterday, I sat down with one of our newer marketing coordinators who had never seen/used analytics before and showed her Google Analytics. She literally "got it" after only 5 minutes of showing her around and she's already able to do just about anything we need with it and report on or analyze the information. wow. Even an old school guy like me was impressed with how easy Google (and others) have made it.

It's an awesome thing and has come a long way.

Andrew Chen · 1 year ago

fair enough! I love the "air" terminology. I agree that positioning it more like a positive investment rather than something as hated as "tax" is a good idea.