Understanding AI technologies is becoming an essential skill for product managers, but true value comes from the ability to develop a strategy that aligns AI capabilities with business objectives and user needs.
This is where product managers must know how to identify the problem to be solved, define the product vision, assess data requirements, establish success metrics, evaluate ethical considerations, and determine how machine learning and generative AI will create measurable value for users.
To put this into context, I created an AI Product Requirements Document (PRD) for an experience that would predict and recommend the players who were most likely to produce the most points helping me win my weekly matches and ultimately the league.
NHL fantasy leagues are quite popular and winning the league can come with added benefits such as financial gains and bragging rights amongst fellow leaguers. One of the hardest parts of managing a fantasy league team is figuring out which players to include on your roster each week. There are many factors that impact this decision such as looking at past player performance history, team history and current trends -- to make a weekly team roster prediction. While this research can be performed manually, it can be time consuming and unorganized - potentially causing an inaccurate database in which someone would make their roster prediction from.
Deliver ML and LLM tool that reviews fantasy player roster information (stats, game schedule) and compares data to roster of weekly competitor -- that evaluates and provides a recommendation for fantasy player roster weekly lineup that could produce the most amount of points, using historical data, during that weekly matchup.
A tool that simplifies data analysis and reduces manual load ultimately enabling end users to maximize weekly total points – with the aim to win weekly NHL Fantasy matchup, and thus, NHL Fantasy season competition.
I am…
a player in a NHL Fantasy league.
!’m Trying to…
win my weekly matchups.
choose the best players I should select to play each week.
But…
choose the best players I should select to play each week.
I am not experienced in using data and analytics to inform my decisions.
I do not have a lot of time to review player data and information.
Because…
I don't know all the places to get the data and information to help me make an informed decision.
I don't have historical data I can easily reference.
I'm unsure who will be the best player to play against my competitor.
Which makes me feel...
like I don't have the tools or information required to win my matchup.
"A tool that simplifies data analysis and reduces manual load ultimately enabling end user to maximize weekly total points – with the aim to win weekly NHL Fantasy matchup, and thus, NHL Fantasy season competition."
Target Group:
Which market or market segment does the product address?
Who are the target customers and users?
NHL Fantasy League Players
Sports fans
Gamblers
Sports Industry
Needs:
Which problem does the product solve?
What benefit does it provide?
Provides opportunity for better decision making
Saves time reviewing data
Reduces manual load on user
Provides readily available information when the user wants/needs it
Product:
What product is it?
What makes it stand out?
Is it feasible to develop the product?
The product is a mix of Machine Learning Models and Large language Models and while using Generative AI.
What makes it stand out is that it can provide a bespoke UI.
What would make the product stand out is that it is effective in achieving what it's supposed to do. .
Product leverages data from: Seasonal schedule, player performance and player performance against a specific team.
Business Goals:
How is the product going to benefit the company?
What are the business goals?
A fee to use the product could be implemented to generate revenue.
Product would be marketed as Fantasy Coach - giving players the edge they need.
Competitors:
Who are your main competitors?
What are their strengths and weaknesses?
Sports/NHL Fantasy bloggers
Sports news reporters
Sport Influencers
Weakness: They only provide the data and not an assessment based on roster
Strength: They provide trend information that the data may not be able to produce.
Podcasters
YouTubers
Revenue Streams:
How can you monetize your product and generate revenues?
Customers can pay a fee to access the product and input their rosters.
Page ads revolving betting, sports equipment and other sport related products.
Cost Factors:
What are the main cost factors to develop, market, sell, and service the product?
Costs associated with building the product.
Costs associated with maintaining and servicing the product.
Costs to promote the product - targeted ads.
Costs associated with hosting and storing data.
Costs associated with risk, legal and security.
Cost for leveraging AI platform.
Channels:
How will you market and sell your product?
Do the channels exist today?
Leverage digital ads to target audiences around betting, sports and fantasy leagues.
Product would be positioned as having your own digital Fantasy coach.
Once popular enough, could sell product to existing fantasy league product for them to integrate into their product offering. i.e. ESPN.
Identify which dataset(s) you will use, how you will acquire that data, and a baseline performance measurement (e.g. a heuristic or a current measure). How does the data get logged and stored? Will new data need to be collected, and if so, how?
Data Set: Seasonal Player Stats
Description: Player on-ice performance.
Collected: Leverage NHL.com/SAP API to pull in real-time data. Alternatively, manually upload data export from another authenticated statistic site.
Logged: Data would need to be logged everyday after the daily games have completed. This would be most beneficial to leverage an API for automation.
Stored: Data be hosted within a Cloud storage platform.
Data Set: Team calendar
Description: Data recorded based on the team's schedule.
Collected: This data is mapped out for the season and is available online. It can be exported into a data stream for logging.
Logged: Data would need to be logged at the beginning of the season and updated should there be any changes in the scheduling.
Stored: Data be hosted within a Cloud storage platform.
Data Set: Fantasy League Player Roster
Description: Fantasy league player roster for the week.
Collected: Data would be collected manually through input from Fantasy league players (end user).
Logged: Data roster could be inputted at the beginning of the season and be updated manually as needed.
Stored: Data be hosted within a Cloud storage platform.
Data Set: Fantasy Player Competitor Roster
Description: Fantasy league player competitor roster for the week.
Collected: Data would be collected manually through input from Fantasy league players (end user).
Logged: Data roster could be inputted at the beginning of each week and be updated manually as needed.
Stored: Data be hosted within a Cloud storage platform.
Data Set: Historical Data
Description: Player and team stats based on previous seasons.
Collected: Leverage NHL.com/SAP API to pull in previous seasonal stats. Alternatively, manually upload data export from other authenticated statistic site.
Logged: Data roster could be inputted at the beginning of the season to help train algorithms.
Stored: Data be hosted within a Cloud storage platform.
Which dataset fields will be important for the model, and why?
Player Name, Team Name, Competitor Player Name, Competitor Team Name
These are important fields because they will give the ML something to benchmark the data against.
Goals, Assists, Penalty Minutes, Powerplay Points, Hits, Shots Blocked
These are important fields because they provide the data to be compared and/or benchmarked.
Historical Points Against Team
This is an important field because it will provide trend data that will assist in providing a recommended analysis.
What will you use to measure baseline performance, and why?
Recommendation Accuracy: If the predicted roster analysis is correct or incorrect - leading to wins.
Validation: Whether the purpose of the tool still supports product vision.
User Engagement and Retention: How often people continue to use the tool.
Acquisition and Conversion: How many people sign up and use the tool.
Revenue: How much the tool generates.
Identify how machine learning will be implemented in your project, which models you will implement, and how you will measure performance. What role will ML serve (e.g. prediction, automation, augmentation)?
How will ML be implemented?
Machine learning will be implemented to analyze and predict potential player results based on a specific set of criteria. i.e. a fantasy league player roster.
Which models will you implement and why?
Linear regression: To simulate a mathematical relationship between variables such as goals, assists, and points to predict a total score outcome.
Neural Networking: For learning patterns in the data to predict trends such as whether a player historically plays better or worse against one specific team than others.
Naive Bayes: For making real-time predictions based on changes to a fantasy league players roster.
How will you measure performance and why?
Accuracy: The ML will be measured based on how well it predicts a winning roster or not. This is important as it is the main purpose of the ML.
Precision and Recall: While the ML provides a prediction, there is still a human factor it can not pull information from. Understanding and comparing the difference between what the ML predicts and what actually happens will be important to continuously train the model.
Processing Speed: The ML will be measured on how quickly it can assess the data and provide a result/prediction. This is important because the purpose of the ML is based on a pre-determined timeframe i.e. a weekly competition. If the ML takes too long to process and can not provide a result in time for the competition, then the ML is not useful.
Designing a machine learning system and the key decision factors.
Will your ML system be cloud-based, or edge-based, and why?
The ML system for the NHL fantasy coach would need to be cloud-based as the high volume data collection would need to be supported and leveraged from one single source, then pulled into a profile based on user roster. In addition:
There would be the potential for better accuracy in predictive analysis - pulling data from the same reference source as competitor data.
Concern for privacy and security leak is low since most of the data collection will be publicly available.
Real-time learning is not important since users would have already selected their rosters before a game has begun -- so the need to reduce latency is very low.
Will your ML system implement batch learning, or online learning, and why?
ML system will implement batch learning. This is because:
The model needs to retrain itself once a day - after games have been completed versus during a game.
The roster of a fantasy league competitor may change and updates would need to be made to the "dashboard" so that a new predictive analysis can be generated. This process could be managed through online learning but the cost and frequency of use would likely not support the ROI.
Will your ML system implement batch prediction, or online prediction, and why?
ML system will implement batch prediction. This is because:
Prediction analysis can only take place before or after a game and thus does not need to be in real-time.
Exploring the ethical implications and identifying actions to ensure positive outcomes.
Which issues are High Impact, Urgent; High Impact, Not Urgent; Low Impact, Urgent; and Low impact, Not Urgent?
High Impact, Urgent:
Data Collection: Ensuring data that is being pulled into ML is coming from a reliable source. This would need to be confirmed against other data sources for confirmation.
Model Optimization: It will be important to train the model quickly so that it can analyze and produce results based on an informed ML.
High Impact, Not-Urgent:
Transparency of ML and Data Collection: Letting users know how the data is being pulled, from where and how it's processed, adds legitimacy to the product. It will be important not to share too much so as to risk unique selling features.
Low Impact, Urgent:
N/A
Low Impact, Not Urgent:
N/A
Compare essential aspects of requirements The Deon Checklist.
Data Security: Ensuring user personal information is safe, secure and doesn't include any information like street address or location - especially since this information is not relevant to the product purpose.
Right to be forgotten: Ability to remove users from the database - should they wish.
Data Retention Plan: While historical player data may be important, keeping user information year over year may not be. Users who do not sign up within 3 "seasons" would have their profiles deleted.
Dataset Bias: ML needs to be constantly retrained so as not to favor one consistent outcome i.e. over favoring a player metric that seems to overrule others unfairly.
How will you resolve any potential issues from the comparison above?
Ideally taking a preventative approach versus a reactive/resolve approach.
As we build out the PRD, we would include a section on Ethics linking to the DEON checklist and ask the question, "Could this be used to hurt someone? If yes, how and how can it be prevented or governed to mitigate this from happening?"
In the case an issue does become apparent, we would roll back the product or feature and iterate accordingly to resolve the issue (if possible) before deploying again. If the issue is severe, the feature or product may need to be sunsetted and not deployed again.
Conclusion:
An AI PRD ensures all stakeholders share a common understanding of what is being built, why it is being built, and how success will be measured.
Key stakeholders typically include business sponsors and leadership teams for strategic alignment, product managers for vision and prioritization, UX designers for experience design, data scientists and AI/ML engineers for model development, software engineers for implementation, legal and compliance teams for governance and risk management, and security teams to ensure responsible handling of data.
By creating a well-defined AI strategy and PRD, product managers can bridge the gap between emerging AI technologies and practical business outcomes, ensuring AI solutions are both valuable to users and feasible to deliver.