#data #product - 5 mins read

Unlocking your product data

Good, meaningful, actionable data is far from a given.

Sometimes there isn’t any data. Other times there is, but you can’t find it. Or you know where it is, but you can’t access it. You may even get your hands on it, but it’s out of context. I’ve even seen teams wade through this entire obstacle course, only to have their insights censored—or worse, condemned—because they haven’t arrived via the appropriate channel, or because they don’t happen to fully align with “exec metrics”.

In this article, I’ll share what I’ve learned about unlocking product data and developing good data practices in teams and organisations.

• Sniff it out
• Join forces
• Forgiveness over permission
• Validate it
• Document it
• Visualise it
• Share it
• Make your own
• Up your team's game
• Bring in expertise

Sniff it out

First, I would encourage you to create your own directory, even if one already exists. Every source you come across, list it out. The tech, the owner, the context, its limitations. Begin to build up a picture of what already exists. This is just for your own consumption to begin with, but can often become a team or org artefact down the line.

Next, the real detective work begins. Dig through your knowledge bases (Confluence, Miro, GitHub, whatever you use that you can get your hands on). Ask the question in all of your intro chats. Ask your team. Ask your user researchers or those closest to your users. Is there any third-party data that’s in the public domain, such as Google reviews?

Join forces

If you work in an organisation with multiple teams, get your counterpart’s take on this. What do they use? What is missing? What plans do they have to improve the data landscape? Are there any obvious opportunities you could work together on, or look to divide and conquer? Is there some joined-up messaging you can deliver to your head of product that would strengthen the case for better data?

Is there a dedicated data or BI/MI team? Reach out to them. Try to understand their worldview. The constraints they’re under. Their ambitions.

It can get lonely really quickly if you go too far on your own. A handful of conversations like this can provide a few short circuits and can bring a lot of reassurance, if not the odd course correction.

Forgiveness over permission

I seem to be more bullish in each new role, as more and more I find that the door is usually open - you just need to give it a push. The overwhelming majority have welcomed this level of curiosity. Of enthusiasm. Scrutiny even. In a good chunk of organisations, data is not a first-class citizen and a lot of people really appreciate any attempts to change this.

To this end, just go for it. By all means, communicate your plans, but there is no need to be tentative. Seek the data out. Get access to all the necessary tooling. Ask for invites to data and research playbacks. Volunteer to talk about this in a show and tell or a lunch and learn. Publish your progress as you go.

Often, this is the best way to learn. People come to you once they see what you’re up to.

“Oh you might want to have a look at Amplitude as well,” a head of product once advised, which I didn’t even know we had.

“Have you seen the Trustpilot analysis from last quarter?” another asked.

Organisations are not good at knowledge management. At sharing valuable artefacts. Forcing the issue can shine a light on insights (and paths to insight) that would otherwise have been forever in the dark.

Validate it

So you’ve got a list of data sources. You’ve even got hold of some actual data and have a feel for the context and how you might make use of it. Now it’s time to apply some rigour.

Once you start talking about data and how it supports your decision-making, someone will inevitably challenge you.

“What do you mean by ‘conversion rate’?”

“I thought Adobe Analytics had been deprecated?”

“But what about those who don’t accept cookies?”

It pays to know your stuff here. Or even better—to get in front of this. To have an appendix ready to go that you can refer others to in the event there is uncertainty or disagreement.

The first step towards this is to validate your data. A lot of the time—with analytics and BI tooling—integrations are sadly fire and forget. They are done once, and rarely returned to. There is no continued ownership of these integrations as products in their own right. Often this leads to definitions being forgotten or warped, or even worse the integrations themselves can mutate over time.

If you intend to hang your hat on any data, make sure that you test it. That it indicates what you think it indicates. There are three angles to come at this:

• Seek out those closest to it. Are there any docs from the initial integration work? These can make for a great starting point.
• Reverse engineer. Dedicate some time to spiking out the integration—to bottom out the context. This is a great way to spot gaps and identify areas for improvement. It can also set you up for any further work in this space, ensuring it’s fresh on the mind.
• Test it. If you’ve got the luxury of being able to use your product in production, do it, and then compare and contrast the data with the activities you carried out. If not, could you do it in a different environment, such as dev, test, or staging? Sometimes it can even be better to do it away from prod so you can easily isolate the data that your actions have produced.

Document it

Now it’s time to go a level deeper than your directory. What are the actual metrics that you have at your disposal? The format illustrated in the table below has served me well in the past.

I’ve also been impressed by GitLab’s product data docs recently and would recommend having a read through and taking inspiration from the way they describe their data and supporting processes.

Visualise it

Crucially, this doesn’t mean “make some charts”, or “we need a dashboard”. By this point, you still might not know exactly what you’re going to use this data for, if you’re even going to use it all, so my strong steer here would be to get the data in a form that’s easy to manipulate. A tabular view across a date range is the gold standard, perhaps with the ability to slice by some dimensions. An obvious e-commerce example could be a view of sales by day, broken out by acquisition.

This doesn’t need to be perfect, and it doesn’t need to be in a BI tool at this stage. As long as your docs give you a way to get hold of the data you need in a form you can utilise, this is probably enough.

If you’ve managed to go one step further and you already have some idea of the metrics that you want to track, by all means have a go at something a bit easier on the eye. This could be something maintained manually—a Confluence page with tables and charts updated once a week—or it could be something more elaborate like a Tableau dashboard that pulls data in from multiple sources.

To help you pick out a few metrics that might be worthy of study—if, for nothing else, to help you better understand your product's users, and how they interact with you—I’d recommend having a read of my pirate metrics article.

Share it

We might be awesome at working on our products iteratively with super-tight feedback loops, but we don’t always apply that to our internal-facing activities.

Make sure that you regularly check in with your team as you work your way through your data stack. Play it back to others outside of your team often—particularly other product folk, leadership, or anyone else with an obvious interest in data, such as user researchers and any data-focused roles.

And it doesn’t need to be all-singing, all-dancing. I initially piloted this for one metric to get feedback on the end-to-end approach as early as possible, and I would highly recommend this approach.

Make your own

I’ve never joined a team where I have all the data I need. There is always more baseline data to be had. And every new hypothesis necessitates even more data. Again, you don’t need permission to push on with this. What other data points would strengthen your hand? Richer product analytics? Deeper user research? Third-party validation?

Regularly review your data directory. Throw rocks. Pick holes. Where else could you improve?

Up your team’s game

I’m a firm believer that it shouldn’t fall on any one team member to pull together and, ultimately, “own” your product data. Not the data analyst. Not your product owner. The more that this ownership is shared, the better. I’ve loved working in teams where the insights bubble up from all over the show.

A few tactics that can help here:

• Give your data the respect it deserves. Referring to it alongside roadmaps, in show and tells, against user stories. If you want others to obsess about it, you have to bring it to the fore whenever you can.
• Go over data in stand-ups, even if it’s just a cursory glance at your KPIs, or an experiment that’s in flight
• A regular data playback (fortnightly)
• A regular data directory review session (quarterly)
• Use OKRs or similar. Even if that’s not how your organisation works, having your own outcomes to strive for automatically shines a light on the data that matters

Bring in expertise

There is no shame in asking for help. I’ve done this myself. I would encourage this if you’re not able to get the traction you need within your team. This could be in the form of a dedicated data analyst. It could be data engineers to help you build out your data pipelines and stack to enable you to get at the data. It could be a reshuffle of the members of each team, to ramp up your overall data competence.

In summary

You probably don’t have all the data that you need to support your decision-making. Some of it may well be there. Some of it may necessitate further thought and likely development effort. This can be a rocky road to drive down, but it is one most certainly worth pursuing.

Remember: you’re not alone! There are lots of resources out there to help you with this, and probably lots of others within your organisation who are just waiting to help you with this.

Written by Dave Baines

Co-founder, Product Lead

07 Jul 2023

link