An early step to manage a product is to map infrastructure that maximizes user feedback — what they think and do. This article provides an example of how I plan and start capturing user insights, particularly if few or no systems are in place. As I tell all of my teams — we can only build the market-leading product if we can understand our users better than our competitors can understand theirs.
Goals Determine Analytics Coverage
The name of the game is to chip away at blind spots. We use methods and tools to achieve insights into our business and market with the hope of helping all team members make hundreds of better decisions each day. A common mistake is to treat the effort as “one and done.” To experience a large return, it’s critical to make ongoing investment by incorporating analytics into the product process. Not incorporating feedback loops into the product process accumulates product debt.
I start with goals — for the organization, various teams (functions and value streams), and specific stakeholders. I try to deeply understand what people are trying to achieve and the types of insights they will be pursuing across these “Layers”:
- Users: Predominantly leading indicators of customer satisfaction (aka love and loyalty) and desired business results across the user journey — acquisition, activation, engagement, and retention.
- Business: Predominantly lagging indicators of the business results that we are trying to drive — metrics related to financials, unit economics, and performance of growth loops.
- Ops: Proactive and reactive alerts to ensure the performance of mission-critical systems, including technology (e.g., application performance, dev / system / network operations) and physical operations (e.g., fulfilment, fleet).
There is no single, correct way to organize your metrics — it depends on the nature and stage of your product/business, goals, and circumstances. Remember, the point is to eliminate as many blind spots as possible.
A prerequisite to planning is deeply understanding the key metrics for each Layer above. Your metrics will evolve over time as your business and market change, or as you discover more meaningful ways to measure success. These key metrics are the levers for growth and must be sharpened in order to build an effective analytics infrastructure.
Plan for Evolving Needs
You won’t be able to plan for all the metrics that will become important but you can future proof by anticipating insights related to defensive or offensive goals:
- Defense — “Is my machine running smoothly?” Also called Health Metrics, these are signals that I keep an eye on to make sure nothing is going off the rails. My morning routine includes sipping coffee and checking key metrics across all Layers — business, users, and ops. I routinely check key metrics but also do variable spot checks of key breakpoints in the user journey — a deep dive which includes auditing qualitative sources, such as customer support conversations and user session videos.
- Offense — “Is my team driving a result?” These are the metrics to move (e.g., Key Results from quarterly OKRs). I try to thoroughly understand the company’s mission, goals, and strategy to anticipate which systems will need to be in place for everyone to successfully translate them into execution.
Today’s Offensive Metrics becomes tomorrow’s Defensive Metrics as successful new features are subsumed into the core experience. The key to finding Feature-User Fit is to build small pieces of your infrastructure just-in-time so that you can quickly and easily answer questions to validate assumptions.
Open Easy Feedback Loops ASAP
There is no one-size-fits-all blueprint. When I first start managing a product, I try to assess needs, constraints, and circumstances before strategizing what to build (while acknowledging things will change). Here’s a simple plan that I used for an early-stage product to start getting user insights quickly:
Qualitative-Attitudinal
In order to build a valuable product, it’s critical to speak frequently and purposefully with users. One quick win that I had was to release Intercom that day (using Segment), which let us live chat with users on our web app. Right away, I was able to do things such as ask prospects why they weren’t signing up (to increase registration conversion rate), what they thought the company offered (copy and design testing), and what they found most unique and compelling (UVP testing).
Live chat helped us accelerate the ability to observe, hypothesize, and validate problems we needed to fix. It also gave me a channel to start getting feedback on potential solutions.
In parallel, I try to start recruiting for in-person user interviews as soon as possible. Recruiting and scheduling interviews is a lot of work so I’ll frequently look for hacks. Aside from the method above (which trades ease for a visceral, in-person discussion), I have done things like use company events (in-person workshops for users) to convince people to chat before or after, use a company’s brand to offer an inside-peek and tour (which gave users a fun story to tell their friends), and, if all else fails, bribed them with food or cash.
It seems like recruiting for app interviews are getting more difficult these days. Beyond the typical challenge of finding evangelists, I speculate there is increasing fatigue and skepticism of new apps and startups this far into the business cycle.
So, when I do find users that provide great feedback, I ask that they join my product panel. This allows me to call on them regularly for quick feedback. Qualitative feedback is equally important after a release to proactively catch bugs or issues, such as UX problems or flawed assumptions.
If you have a UX researcher on your team — great! If not, you’ll have to fill the skill vacuum.
Qualitative-Behavioral
This category helps validate that a new feature is valuable or usable.
Pre-release, I have used mocks or prototypes, and various tools, to start validating a potential solution — both the value proposition and conceptual approach.
Qualitative-Behavioral insights are also important after release. Quantitative-Behavioral methods are good at telling you “what” while qualitative methods offers “how”, and sometimes “why,” something is happening.
One quick win I had was to implement a tool that allowed me to watch video clips of actual user sessions. Modern tools, such as Full Story or Hotjar, can even flag “rage clicks” or other UX choke points that you need to fix. If your team has already implemented a data collector, such as Segment or mParticle, you might be able to flip a switch to turn these tools on (without involving a dev). A product analytics or BI tool can help you see dropoff in your onboarding funnel, but Qualitative-Behavioral tools and methods can round out your perspective for diagnosing problems.
Quantitative-Attitudinal
Start collecting user input to make better decisions by opening and leveraging communication channels (e.g., individually email customers by using a mail merge plugin for Gmail, send behaviorally triggered messages to cohorts with marketing tools, Intercom for site intercepts, Twilio for texting). Even if it feels rough and dirty to start, hearing from your user is better than guessing.
Simple example: We were trying to prioritize which third-party Applicant Tracking Systems to integrate our platform with. Rather than guessing what customers wanted, we emailed and called customers (more effort, not repeatable) but also surveyed them with an Intercom popup (less effort, repeatable) that appeared when they clicked a new CONNECT ATS menu option.
Quantitative-Behavioral
As your product grows, this is the category you will invest heavily in. Effectively analyzing what users actually do (versus what they tell you) is necessary to develop successful features.
Split testing is the “holy grail” of proving Feature-User Fit, but, like all these methods and tools, should be triangulated with other sources. When you first release your product, you might not do much split testing because of a lack of users. But as you get traction, be ready to develop this capability.
Product analytics is also a critical investment to make for the first day you have users. The value of data compounds with time. The more (quality) data you have over time, the more valuable your insights can be — particularly as you benchmark your analyses (e.g., normalizing for cyclicality) or study longitudinal effects.
Product analytics is a part of your larger Business Intelligence (BI) stack. I get questions about what the difference is (and why you’d want product analytics then). Short answer is that product analytics is tailored for people working on product (e.g., product managers, designers, engineers, marketers). These tools can offer advantages across the board depending on your needs, such as simpler and quicker to implement and change, easier to visualize data and report, and offer product-specific analyses out-of-the-box (e.g., funnel conversion, retention, predicting churn). Product analytics tools also typically provide a fuller view of a user’s actions over time and across platforms and properties. BI tools can achieve similar outcomes but may require more work or specialized people (e.g., developers and data scientists). BI can offer power and customization. Product analytics can offer relevance, simplicity, and speed.
Which of These Capabilities Should I Roll Out First?
Depending on priorities, I typically recommend playing Defense first by covering blind spots. I also prioritize tools and features that collect priority data (as uncaptured data is lost forever), including specific metrics for BI / product analytics, customer support (live chat), and session recorders.
Then I get started on long-cycle activities such as building a pipeline of users to interview and creating a product panel.
For working on new features, I typically develop qualitative capabilities before quantitative capabilities. Below is an example of insights needed during a feature lifecycle. The oscillation is a fake representation of interactions (direct or indirect) between the product team and users. Pre-release, interactions are direct conversations and feedback with a smaller group of users. Post-release, interactions shift to measures or meta-measures of behavior in aggregate.
Buy vs. Build?
You might be able to trade money for speed. Or trade speed for customization. Yes, third-party tools can get expensive (although it’s getting easier to avoid vendor lock, and many tools offer generous free-tier plans) and you won’t be able to fully customize for power. But most of the time — what you’ll get is good enough to start. Just like everything product — you don’t want to over-build your analytics infrastructure. A common mistake that I see is building something that should have been bought (or better yet, rented).
For later-stage products and organizations with more sophisticated needs, a high-level rubric to decide:
- Is this capability mission critical? (Defense) AND/OR
- Key differentiator for our brand? (Offense)
In “Delivering Happiness,” Tony Hsieh talks about this lesson when Zappos first tried to use a third-party inventory management system for the sake of moving fast. The team then realized that fulfillment was a mission-critical capability and so decided to build and own that competency.
At OkCupid, we used a homegrown platform for split testing, called “DB Experiments.” Due to the needs of having multiple split tests with user-to-user interactions, experimental control was a challenge. It made sense to have a custom platform, versus using an off-the-shelf tool. DBX was even improved to allow for geo-based split testing. Clustering groups of users that were naturally interacting with each other gave us more meaningful results. We could see the performance of a feature in a set of comparable cities (e.g., NYC vs. LA).
Don’t forget about the total cost of ownership. An engineer might tell you that it’s not that big of an investment to create your own solution, but remember that support and maintenance will be the significantly larger cost, over time. When I was at OkC, more than 80% of backend engineering resources was spent on supporting and maintaining existing systems. That means Showtrends (a homegrown Application Performance Monitoring tool that evolved into meeting product analytics needs) did not get much love and couldn’t keep up with the business’ needs. I worked with the CTO and data science team to improve the BI stack to more heavily rely on outside vendors so developers would be freed up to work on projects that would improve the user experience and business.
Canada company registration for Canadian residents
We incorporate your new corporation federally or provincially in any province of Canada according to the provisions of federal and provincial Business Corporation Acts. Company Formations Canada provides fast and easy Canada Online Incorporation Service for only $99.99. Continue here to register today your new company in Canada.
https://companyformations.ca/corporations/
Canada Company Registration for Non-Canadian Residents
Incorporating in Canada – Canada Incorporation for Non-Canadians for only $2200 (All Inclusive.) Company Formations provides fast and easy Company Registration in Canada for non-Canadians residents and provides all the documents your new Canada corporation will need to stay up-to-date and in compliance with your province of registration corporations law.
Learn more about our services
https://companyformations.ca/canada-corporations-non-canadian-residents/
Canada Company Registration for Foreign Companies
Company Formations offers a full range of services designed to help global entrepreneurs and small and medium sized businesses create new businesses in Canada or expand current businesses to Canada.
Our services to non Canadian residents includes:
New Company Registration (Canada Corporations)
Registration of foreign companies in Canada (Extra Provincial Registration)
Canada Registered Agent Services
Canada Corporate Bank Account Introduction & bank account opening
Canada Residency Visa under the Canada Startup Program
This Article first appeared in medium
