<aside> 💡 Data is an important tool, but it’s not a panacea.

</aside>

Untitled

I’ve been considering this topic for a long time, especially since it played a crucial role in my second job. Over the years, my attitude towards “data” has gone through many changes.

In my first job, product design was mostly driven by the intuition of the founders or the team, with data playing a minimal role. After the product launch, process data was entirely ignored. Therefore, in my second job, I emphasized quantifying and tracking requirement goals from the start. The company culture also promoted quantitative assessments and emphasized clear benefits.

However, my attitude towards data changed over time. I realized the limitations of “every requirement must be evaluated by data,” grew to dislike blind data-driven approaches, and frequently argued with leadership over this concept. Here are some of my thoughts and observations on data-driven approaches.

Limitations of Data-Driven Approaches

Several examples frequently encountered in the past:

  1. Difficult to Quantify Individual Contributions: How can a product manager of a B2B product quantitatively explain the contribution of each feature iteration they oversaw to overall revenue growth? Why not attribute the success to the sales team? How can the long-term impact be assessed? Perhaps one of the challenges big companies face is the inability to accurately measure the contributions of each ordinary employee apart from sales, which explains the use of working hours as a metric.
  2. Multi-Factor Influence: If the average call duration of a customer service category decreases, how do we prove it’s due to an internal feature launch and not other factors?
  3. Evaluating Unimplemented Projects: How do we accurately assess the benefits of a project that hasn’t been implemented yet?

In Andrew Chen’s article, “Why data-driven product decisions are hard (sometimes impossible)”, he also lists several issues with data-driven approaches: