18Th5
Essential Knowledge About AI-Powered Product Management
78% of companies use AI. Only ~5% create real value from it. The gap isn’t technology — it’s product thinking.
Landscape of AI
The future of AI is predicted and investigated by governments and big tech companies. It is a beautiful future where we move away from traditional AI, known for problems such as face recognition, sentiment classification, ranking, and object detection, toward applications powered by genAI.
GenAI, as we know it today, is not new; LLMs are just a larger version of what we have used for decades. The performance of LLMs and their integration into our daily lives through services like ChatGPT, Gemini, and Claude have raised our expectations for using AI in the community.
The next step in AI is Artificial General Intelligence (AGI), the AI that can mimic human cognitive functions, automatically solve problems, conduct R&D, and develop novel healthcare treatments. AGI is predicted to happen in the 2030s. Even Elon Musk made a bold prediction that AGI will emerge by the end of 2026.
After the AGI era, the era of Artificial Super Intelligence (ASI) is expected to arrive in the 2040s to address several of humanity’s problems, such as climate change, poverty, and space exploration. However, ASI at the moment is just a hypothetical version of AI, and no one has really started working on it.
Image by Marily Nika — https://marily.substack.com/p/ai-is-not-just-gen-ai
The optimistic future of AI makes people so obsessed with making AI smarter. Every week, in the AI community, you can hear a new model break a benchmark. Every month, a new article claims that we’re closer to AGI. But several reports still confirm that AI products fail to deliver ROI. This effect is not because the model was too weak, but because nobody thought clearly about what the product was actually supposed to do.
I’ve been working on several AI products and AI projects for a while, and I keep seeing the same mistake: teams apply a regular management playbook to a product category that requires something different. The following guide is a compressed version of the handbook that I hope everyone should have in the first day working on AI.
AI Is Everywhere. Value Is Rare.
The numbers tell an uncomfortable story. 78% of companies now use AI in at least one function (McKinsey). ChatGPT hit 100 million users in 2 months (UBS) — the fastest adoption of any consumer product in history. 1 in 6 people on Earth use AI tools in some form (Microsoft). Adoption of AI increases from 55% to 78% in 1 year(Stanford HAI — AI Index Report 2025)
The gap between those two numbers is where most AI product work happens. Understanding why it exists is the first thing a good AI PM must do.
The six traits that make AI products difficult to ship
Traditional software is deterministic. Same input, same output, every time. AI is not. Until you internalize this, you’ll keep applying the wrong model, and your product will keep surprising you in the worst moments.
1. Probabilistic Nature
AI predictions are never 100% correct, so your AI team’s role is to set the right expectations with stakeholders.
A good way to manage the uncertainty of AI is to set up a feedback loop to monitor and adjust to maintain the expected performance.
Also, having a risk management plan for cases where AI cannot perform well. My team has been building an AI system to auto-review content uploaded to our platform, PixtaStock. Among several strict criteria for review, one is to detect all the faces that are not registered by uploaders due to legal risk. AI can achieve up to 98.8%, humans can detect 100%, not perfect for AI and here is how we have to deal with the problem:
- Analyse the missed case to ensure that your solution addresses the problem’s limitations. Unfortunately, not possible at a reasonable cost.
- We then have to estimate the risk if the violated content is introduced to buyers by analysing past conversion rates and the uploaders’ violation history.
- Thank God, the number shows an acceptable insight, just under 0.01%, tiny enough for a risk
- We are also preparing a legal treatment in case the tiny luck happened for the case bought by any buyers.
Important note: Please compare AI with human-level performance. The most underused action in every stakeholder presentation I’ve ever seen.
2. Data Dependency
Garbage in, garbage out. High-quality data means less bias, less noise, and less irrelevance, but not many companies care about sourcing, cleansing, and validation from day one. Building an easy-to-access, easy-to-maintain data pipeline is a must. Specifically, nowadays, if you want to embed LLMs in data mining.
Important note: Even though we’re becoming dependent on the foundation model and AI APIs, your business data still contributes significantly to AI performance.
3. Model Drift
The world changes. Your model doesn’t automatically know that. AI is not a “one-and-done” traditional software, it’s a continuously evolving system that requires continuous monitoring, retraining, and a feedback mechanism into the product itself.
Even foundation models are not guaranteed to perform consistently over time. Plus, LLM API services like Gemini or ChatGPT are often updated with new models, and a single update requires new prompts and new evaluation for the AI solution. A quick way to bypass this pain point is to auto-prompt for your AI solution using DSPy, but in several practical cases, I cannot find a good sweet spot.
4. Interpretability & Explainability
We often find it difficult to explain AI models, especially complex ones. Yet, the demand for making AI predictions transparent to end users remains a hot topic. Since end users tend to love using what they actually understand in a nutshell. In healthcare, finance, and legal, you must explain every significant AI decision. Transparency and accountability aren’t nice-to-haves — they’re the key to gaining trust from users.
It’s great if your UI can provide information on why the AI models make decisions, yes, like the way we tell LLMs to give reasoning for their outputs. At ClientScan, a UK startup we found in London, we showed a confidence score for every AI prediction, and our customers are convinced by AI’s decisions.
5. Automated Decision-Making
AI’s most powerful feature is also its most dangerous. Knowing when to keep a human in the loop and designing the fallback, escalation protocol, and fail-safe are product decisions, not just technical ones.
Once we build a customer support chatbot at Pixta, if the AI cannot make a decision with high confidence, it transfers the question to a human. The fallback rate may be high, but the risk reduction is worth it for the sensitive information.
6. Oh my cost
AI cost is about everything involved in using the AI feature. It includes initial system cost, scaling cost, monitoring cost, and model cost, without exception. AI doesn’t get cheaper as you grow without deliberate optimisation. The promise of cheap AI in the future is hype (even though I hope it’s not hype). Always ask whether the value created justifies the fully-loaded cost.
Complex models often provide better performance at the cost of speed and money. Have to balance several factors to select the best suite probability for your AI solution.
What Does an AI PM Actually Do?
A generalist PM launches the right products by finding user needs and aligning them with business goals. An AI PM does the same — but with a data-driven approach that leverages AI’s unique capabilities to create personalized, intelligent experiences at scale.
The skill stack has four pillars: core PM knowledge, engineering foundations (enough to talk honestly with your ML team), leadership and collaboration skills, and AI lifecycle development knowledge,out training, evaluation, deployment, monitoring, and retraining.
There are actually three distinct archetypes, and most job postings blur them together:
Different types of AI PMs align with different project purposes. AI Builder PMs, who you often see in big tech companies like OpenAI, Google, and Anthropic, to build core AI engine used by global people. AI Experience PMs are more familiar with general IT companies to improve user experiences. While AI-Enhanced PMs, which help boost productivity, could be the future trend as AI agents become co-workers in daily life.
Before accepting any AI PM role, ask yourself honestly: which of these three am I actually being hired to do?
The AI Product Development Lifecycle (AIPDL)
AIPDL is the lifecycle that loops more than you expect. Traditional product development flows in a mostly linear path: discover, design, build, test, then launch. AI products have a fundamentally different shape. They loop more. The failure modes arrive simultaneously, earlier and later. And there’s an entire sub-lifecycle, the AI lifecycle, nested inside the main one.
Marily Nika has drawn one of the most perfect illustration for AIPDL
1. Ideation — Map AI capabilities to real pain points
The question isn’t “what can AI do?” It’s “what does this specific user desperately need, and can AI solve it better than anything else?” Often starting from users’ pain points, not from your favourite technologies or beliefs. If not, you will pay a high cost in money, mental health, and time. Once you deeply understand the problem, converting it into an AI/ML problem helps you select the appropriate AI solutions.
2. Opportunity — Assess market fit potential
Don’t rush this. Most AI products die because the problem wasn’t real or urgent enough, not because the model was too weak. In several cases, if you can do well at the first step, it’s only about finding a suitable system and business model for your AI product.
3. Concept / Prototype — Scope, train, integrate
Scope out the work, train the model(s), and integrate them into the user experience. Expect surprises at every step. The model will behave differently from what you imagined in the spec. The POC should be quick; under 1 week per POC*, in case you need to dive deep into the UI/UX. With Vibe code’s support, some hours are too much.
*The time is estimated based on our internal projects.
4. AI Lifecycle — The nested loop
This loop runs continuously. Your job as PM: define MVQ (Minimum Viable Quality) clearly and don’t proceed with the rollout without it. Let me repeat again, MVQ, for example, 90% accuracy for an AI face detection. Your team is safer if the metric for the quality is aligned with the business problem, which you should do well in step 1 when converting a business problem into an AI problem.
5. Testing & Analysis — Validate at MVQ
Gather feedback to ensure Minimum Viable Quality and validate hypotheses. AI evaluation requires specific proxy metrics. More on those below.
Testing methods must be aligned with stakeholders from the very first product timeline and be obsessed with each testing milestone. Focus on it at every iteration, and gain greater motivation after achieving the previous goal. That’s the fuel for your AI development team.
6. Roll-out — Productionize and open for users
Launch is not the finish line. It’s the starting gun for monitoring and iteration. AI products that stop evolving at launch start failing from the start. Monitoring is a must; don’t ship without a monitoring plan.
A good way is to implement MLOPs into your AI product.
Measuring the Right Things
Most PMs know how to measure a regular product. Fewer know how to measure an AI one. The difference is that AI products have three distinct metric layers , each with a different owner and implications when something goes wrong.
The easiest way to remember this: Product health metrics tell you if users care. System health metrics indicate whether the system is running. AI proxy metrics tell you if the model actually works. You need all three but most teams only track the first one seriously.
Each stakeholder group has a “favorite taste.” Product owners, business managers, or bosses care about engagement and financial metrics. Your engineering director lives in uptime and scalability. Your ML team argues about accuracy, precision, and recall. Know your audience before your next metrics review and know which layer is actually on fire before you walk into the room.
The following table summarises the metrics an AI PM needs to collect.
Product health metrics
A simpler way for you to remember is illustrated in the figure below. Learn it, please!
Image created by the author
Final thought
-
AI is not a product strategy. It’s a capability.
-
The strategy is still: find the right pain, measure the right things, iterate faster than everyone else.
The companies winning with AI aren’t the ones with the biggest models or the most GPUs. They’re the ones where AI product members truly understand the game they’re playing, who set honest expectations, build feedback loops into the architecture, and know when to put a human back in the loop.
I write about practical AI product thinking — what actually works in production.
If you’re building AI-powered products and want to:
- reduce model cost
- improve performance
- design scalable AI systems
I’m happy to share ideas or suggest. Feel free to connect with me on LinkedIn.
#Ai Product Management #Generative Ai Products #Ai Product Metrics #Building Ai Products #Ai Product Strategy
