Season 3 - Episode 9: Feedback Loops and Data Goldmines
In the early days of many businesses, including ours, data is at a premium. You want to take a data-driven approach to decision making, but because of frankly low system usage, or a product that may not have a ton of data you’re at a loss as to how to “find” data.
But data is important. Having a healthy feedback loop with material data sets is crucial to your success. In episode 9 of season 3, and the first episode of our “Big Data for Small Companies” series, we tackle how to find data and build feedback loops for your nascent product.
Background Story
It’s 2013. We had a handful of customers and a couple hundred end users out there using our personal mobile safety app platform. It wasn’t exactly a mountain of data but we were keen to find ways to continue to iteratively improve our platform.
Still, our product had a problem: it was only really actively used slightly before, or during an active emergency or threat to an end user. The rest of the time it was a “peace of mind” product, doing little more than sitting on the end user’s phone. This was fine with us; we thought of our software as the fire alarm pull station on a wall - you didn’t use it every day but you’re glad you have it when you need it. But this low daily usage made it tough to discern what product enhancements our end users were looking for. In short, we had to go out and solicit feedback and it was not a trivial process.
Outline
Feedback Loops and Why They Matter.
How to Get in the Loop
How AI Can Help (And Hurt)
Busted Myths
Myth: You know how people are using the product.
Myth: Product feedback will come to your firm naturally.
Learnings
Feedback Loops and Why They Matter
Feedback loop: part of the “MVP” process - build, measure, learn. To improve the product, you need the data (measure + learn).
The reality is that some products have natural feedback loops. Think search engines, spotify, etc. They are always learning and get better as they’re used based on user action.
Others not so much - because they don't have data, like dumb products or lower usage products - and others because the data can't really be improved. For example, fitness wearables have a mountain of data, but it’s very user-specific and can’t necessarily be extrapolated to create universal product enhancements.
How to Get in the Loop
So how do you deal with this? How do the small time data businesses get enough data to create a feedback loop. The HBR article, To Get Better Customer Data, Build Feedback Loops into Your Products, by Andrei Hagiu and Julian Wright - suggest a few interesting ideas:
Consider modifying the product or service that in the usage of the product insight is naturally given. You get feedback automatically.
This is obviously kinda tricky. If you’re just making your product now, there may be ways to make this happen. But if you have a more mature product, this might not be on the table. Plus, some platforms just don’t function this way.
Integrate with something else that actually has data, and then compare that data with activity with your product. LLMs and integrating with salesforce or word or something to see what content is used.
This sounds easy but is also probably tricky. Integrations kinda suck - they can be a ton of work and sometimes break over time. Still, it may be helpful in getting more insight into how your product is used.
For example, during the pandemic we did vaccine tracking and passed the data on to Tableau or Microsoft Power BI - it would have been great to have more visibility on how data was being collected and used so that we could make changes to improve the experience for end users. We did start collecting this later in the game.
Other than that, just ask. AMA! Make it painless if you can. Expect low rates of response.
This was our go-to in the early days. But interestingly, we usually ended up asking our buyers (campus safety) as opposed to end users, simply because campus safety was more engaged (and paying money).
Good outcomes though. We incentives feedback either with hard cash or that we’d build a unique feature.
How AI can Help (and Hurt)
What’s really cool that’s happening now is AI is helping understand and boost feedback. It can see patterns that humans can’t pick up as quickly.
But it’s not a panacea. For one thing, the consensus is that while AI can do some analysis, make sure real humans weigh in.
And, in a catch 22, AI will need a lot of data to be effective. It’s going to take time for AI to help you on this one.
Summary
Data is good! But your product may or may not be generating enough it.
You want feedback loops. Some products are better suited than others to create feedback loops.
There are options to make feedback loops. Worst case: try asking your customers.
AI is here, but it won’t necessarily help early.
Data And References
To Get Better Customer Data, Build Feedback Loops into Your Products, Harvard Business Review by Andrei Hagiu and Julian Wright
https://hbr.org/2023/07/to-get-better-customer-data-build-feedback-loops-into-your-products