Moving from Reports to Insights
Dashboards are compilations of reports. Do your reports tell you anything beyond counting? Here are some tips to get more insights from your counting. Everything Starts Out Looking Like a Toy, #126
Subscribe now for free to join curious folks who get the “Everything Starts Out Looking Like a Toy” 📊 newsletter
Hi, I’m Greg 👋! I write essays on product development. Some key topics for me are system “handshakes”, the expectations for workflow, and the jobs we expect data to do. This all started when I tried to define What is Data Operations?
This week’s toy: a website that makes it easy to play a radio station from anywhere in the world. (Try listening to 80s music from Rekjavik). Edition 126 of this newsletter is here - it’s January 2nd, 2023.
The Big Idea
A short long-form essay about data things
⚙️ Moving from Reports to Insights
Almost all organizations maintain reports based on their collection of 1st party data. First-party data – data collected directly from user actions – delivers feedback on what’s happening in your system based on customer and user activity.
First-party data looks like user activity in your application, or information entered into web forms by users. This is the gold standard for understanding what’s going on. By counting the actions people take in your app and seeing if they are successful in completing flows, you can make a pretty informed statement about which users are successful.
Or is it more complicated than that? There are some users who complete flows successfully and then churn. There are other users who can’t do much on their own without help from a customer success team and maintain their status during a trial or subscription for other reasons. What can we do to move beyond simple counting in reports to get true insights about customer and prospect behavior?
What are Reports?
Let’s start with a few definitions. Reports are compilations of information in your system that let you measure based on filters, groupings, and limits. For example, you might want to count the number of customers in your system by selecting unique accounts from the closed-won sales opportunities that happened this year. By grouping this list by account ID, you will be able to count the number of opportunities for each account and the number of unique accounts. Filtering this list to “closed-won” opportunities and new or expansion opportunities removes accounts that are not yet customers.
This is a simplified report - your real report might also include a subquery to remove accounts that stopped being customers during the previous year, or a limit to return only your top 100 customers.
Reports don’t need to be complicated. They need to have clear definitions that clarify what they are counting. “All Sales” is ambiguous; “All sales this year” is more descriptive because it explains what changed over what period of time.
A general suggestion here is to explain X to Y by when - a useful metric tells you what happened during a specific time period, allowing you to compare it with another metric during that same time period.
What are Insights?
Insights are second-order benefits that result from analyzing and understanding the data that you collect in reports. Successful insights provide a deeper connection between the data and what it means. What do you want from your reports? Information that helps you make meaningful decisions.
However, most reports that count provide only “easy” insights. “Easy” in this case means they don’t go beyond the counting and give you strategy or decision support to help you make a prediction for the business.
Why does this matter, you think … I know how to extrapolate from multiple quarters of sales data to predict whether sales might continue to accelerate in the future.
It matters because your first impulse to find an insight may be correct, or it may need additional validation to confirm that it is in fact an insight.
For example, here are a few first takes at insights. You can get to these by looking at one or more reports, but they don’t necessarily tell you the why.
Customer count: identifying the number of customers without understanding how they were acquired
Relatedness of metrics: comparing any one metric during a time period with another metric during the same time period without considering how they might be related (Note: this one’s tricky, because sometimes they are related)
Customer health: measuring in-app actions for a customer without looking at their overall account health and satisfaction
Stronger and more effective insights for the same information might look like:
Placing the customer count in context: looking at the customer count alongside the blended cost of acquisition, the cost by source, and the rate of acquisition lets you know how customers entered the system. It also might help you understand which channels are performing well.
Identifying and chaining input metrics to the metrics you’re watching: if you know the conversion rate for a given source and you know the rate that leads are arriving from that source, that gives you information on which sources to invest in to grow customers.
Changes in customer patterns usually mean something: a drop-off in the number of logins per month might be significant. Or it might mean that things are working well and are on autopilot. Which one is it?
Start with the reports to get basic information, and then build secondary reports that help you to see what’s really happening in the business.
What can you do to get more from reports?
Better reports give you the opportunity to produce insights by doing a few things:
They provide clear, actionable labels that make it easy to understand the time period for the report, the population and data that it covers, and the change over time
They indicate the lineage of the data that went into that report, so it’s possible to understand the inputs. For example, when looking at a list of opportunities, you are able to see the progression of lead to contact to opportunity contact to explore the source of those opportunities.
They provide to store your guesses and validate later whether these hypotheses turned out to be right.
Insights, explained this way, start to look more like the kind of exploration you see in a notebook (Jupyter notebook or spiral-bound, it depends on the way you arrange your work). They are educated guesses that build on your understanding and are fueled by the basic information from reports that count information.
What’s the takeaway? Counting things is the “table-stakes” version of analysis. If you want to deliver insights to the business that drive more strategic decisions, move beyond the basics to build hypotheses to measure with your data.
Links for Reading and Sharing
These are links that caught my 👀
1/ On the internet, no one knows you’re a dog - Whether you love making pictures using generative AI or hate the fact that a computer system is manufacturing images without the consent of artists who contributed, it’s still fun to make pictures of cats in unusual situations. Prompthunt makes it easy to build images using Midjourney, DALL-E, or Stable Diffusion from a simple prompt. I’d be surprised if these tools aren’t widely available within 12-18 months in most software.
2/ You need a Jack of All Trades - I love this brief essay about wildcard people. A wildcard, Rob Merki writes, is the kind of person who can do just about anything. Need to learn a new process? Need to invent a new process? A wildcard is your friend. They might need new shiny projects to keep them interested, but every team needs a few people like this to deploy on the strange, random problems of startups.
3/ More about Dunbar - Matt Webb wrote a wonderful piece on Dunbar's Number. If that doesn’t ring a bell, it’s the number researcher Robin Dunbar has posited as the optimal size for human tribes: about 150. One interesting takeaway from this piece is the potential optimal size for small groups within a group: about 4-5. The next time you’re in a meeting of 10 people and wondering why more work doesn’t get done, perhaps that meeting should be two meetings held among two groups.
What to do next
Hit reply if you’ve got links to share, data stories, or want to say hello.
Want more essays? Read on Data Operations or other writings at gregmeyer.com.
The next big thing always starts out being dismissed as a “toy.” - Chris Dixon