Jamie's Blog

Lessons from a life of startups, coding, countryside, and kids

On dashboards, metrics and chartporn

There’s a trend, nay a bubble (bubbles are all the rage these days), in all sorts of dashboards and metrics services. Librato, StatsMix, Klipfolio, and Ducksboard are just a few that I’ve looked at over the last few days They’re all really really pretty, achingly beautiful… but what are they telling us? They can show infinitesimally detailed charts, headline numbers and general trends all collected in real-time from hundreds of metrics. That’s great if you’re in operations (detailed data is important for recognising & diagnosing problems) but this isn’t something that the business owner should ever see.

  • Real-time dashboards encourage “chart watching”. Oh, my, look at the pretty! Oh, look at that lunchtime peak! Oh, look, New York just woke up!
  • Real-time dashboards discourage comprehensive reflection. A dashboard, yes just like the dashboard of your car, is for clearly communicating information about the now. They’re not for analysing history or for reflecting on the previous week. Dashboards live in the now. Dashboards are tactical, not strategic.
  • Detailed data encourages micro-management. Do you want to explain to your boss for the hundredth time that the 3am spike in CPU load is actually the backup process kicking off? That the Pingdom “error” was actually a local DNS failure and the server wasn’t “down”? Do you want the CEO’s meeting interrupted because a metric on his dashboard flips from green to red?
  • Detailed data is mistaken for control. Just because you can monitor the number of visitors in real-time doesn’t mean you have any control over it. Your sales are down by 10% today compared to yesterday so now your boss is stressed out and you need to explain the concepts of “variance”. Sometimes things change, often out of your control. Why are you monitoring things at a level that you can’t control? Do you hate yourself?

What would a client report look like?

This brief sojourn into metrics-land was prompted by the thought: if I could create a weekly report for clients, what would they care about? I’m responsible for developing and running the application so that’s the data which I have available (rather than the social or conversion funnel data). I might need to know about CPU & network performance but they don’t.

  • They might want to know about any downtime events but they really need to know which companies (i.e., the actual customer) were impacted.
  • They should be told about errors but they really care about who was affected and how.
  • They might want a headline stat about the average page rendering time but, mostly meh, that’s not going to help them
  • They need to know which companies signed up last week. They really need to know who in that company is using the app and how the adoption is progressing.
  • They really need to understand which companies (not users!) are deriving value from the application, and which are not.

These aren’t standardised metrics; these are metrics which have to be custom developed for each specific application. In the case of a HR application, what percentage of a company’s employees have a current performance plan? How long are these plans? — because companies using annual plans will derive less value than those using quarterly plans. Is this company tracking goal progress using milestones or metrics? — because tracking goals increases the chance that they will be completed.

This is not metrics. This is not numbers.

This is not “Big Data”.

This is about customer intelligence.

This is about understanding your customers so that you can help them succeed better with your service.

To be fair, there are some services making in-roads to this space (Intercom and Customer.io) but they are still user- not customer-focused. To various degrees, they also rely on high-level metrics and focus on automated messaging rather than reporting.

Perhaps I need to build something…