Product Manager – Machine Learning Interview Questions

In this post, you will learn about some of the interview questions which can be asked in the AI / machine learning based product manager / business analyst job.  Some of the questions listed in this post can also prove to be useful for the interview for the job position of director  or vice president, product management. The interview questions can be categorized based on some of the following topics:

• Machine learning high level concepts
• Identifying a problem as machine learning problems
• Identifying business metrics vs value generation
• Feature engineering
• Working with data science team in model development lifecycle
• Monitoring model performance
• Model performance metrics presentation to key stakeholders
• Setting up product AI team (I will be doing a detailed post on how to set a product AI team)

You may want to check out one of the related topics I covered in another post – Top 10 Data Science Skills for Product Managers

Most Important Interview Questions

Here are some of the interview questions which you as a product manager / business analyst may want to get prepared with:

Q1. How would you define the terms – data science, machine learning, deep learning and artificial intelligence (AI)?

Simply speaking, AI is a broader term which represents the computer programs which mimic human intelligence. This can be done use a set of complex rules processing or training machine learning models. Here is a post on the difference between artificial intelligence and machine learning.

As a product manager / business analyst, you may want to note that all of the following can be solved using AI.

• Solving a problem using large set of complex rule sets
• Machine learning models related to predicting numerical outputs (regression) or classes of the data sets (classification)
• Natural language processing related problems
• Images classification, regeneration etc
• Audio / video classification

Machine learning is about training a machine (set of mathematical models) with historical dataset such that the machine can predict on the unseen data. The key part of machine learning systems is that it’s performance can be improved based on the new data set (experience).

Deep learning problems form the subset of machine learning problems. Deep learning represents the aspect of machine learning which mimics human brain for learning from data set and predicting the outcome on unseen data set. You may want to check some of the following posts to get an understanding of what is deep learning.

• Deep learning explained in layman’s term
• Why is deep learning called deep learning?

Q2. How do you identify / classify whether a problem requires machine learning solution?

Here are few rules based on which you can classify a problem as machine learning problem or otherwise:

• It is not easy to identify a finite set of rules based on which one can determine output related to numerical problem or classification problem.
• Although the finite set of rules can be identified, however, the fact that rules change very fast makes it difficult to deploy the solution changes in the production
• Whether the solution requires large volume of data for testing / quality assurance (QA)
• Whether the solution improves with the improvement in variety of data

Q3. What are different kinds of machine learning problems?

Here are the three common kinds of machine learning problems:

• Supervised learning problems: These are problems where the output labels or actual values related to response variable (variable which needs to be predicted) are available. The machine is trained using both the data and related output value. Later, the machine makes the prediction on unseen dataset. Supervised learning problems can be categorized into following different types:
  • Regression (Predict the numerical value given the data set)
  • Classification (Predict the class or the label of the dataset)
• Unsupervised learning problems: These are problems where output values or labels ain’t present. Clustering is one common type of unsupervised learning problems. The machine learns the clusters of data given the data set.
• Reinforcement learning problems: Given the environment, the machine learns to perform most optimal action based on feedback it gets by performing action in simulation or training environment. Some of the key aspects of reinforcement learning includes environment, current state, action, future state, reward.

Q4. What is feature engineering & what role product managers has to play?

Feature engineering is one of the key stages of machine learning model development lifecycle. It can be defined as the process of identifying the most important features which can be used to train a machine learning model which generalizes well for unseen dataset (larger population. You need to clearly understand the concept of features. Here is a post on this topic – What are features in machine learning?

Feature engineering comprises of the following tasks:

• Identifying raw features which can be obtained from the dataset
• Identifying derived features which can be obtained using the raw data set
• Extracting features from the existing features
• Selecting the most important features from features obtained in above stages

Feature selection and feature extraction are two important techniques in relation to feature engineering. Checkout the related post on this topic – Feature selection vs feature extraction

As product manager / business analyst, you play a key role in helping data scientists identify raw features and derived features. The other two tasks of feature extraction and selection are solely work of data scientists.

Q5. What is the roles & responsibilities of a product manager / business analyst through the model development lifecycle (MDLC)?

The following represents some of the key roles & responsibilities of a product manager / business analyst through the machine learning MDLC. These could also be taken as job description of AI / machine learning product manager. Ability to answer these questions with clarity may most likely help you crack the interview.

• ML problem analysis: Identify whether the problem is a machine learning problem; He / she may need to work with the data scientists
• Making data available: Play key role in making the data available to the data science team
• Business / Technical metrics: Set the business metrics / technical metrics for measuring the model performance vis-a-vis business value generation
• User acceptance criteria: Set the user acceptance criteria for models to be moved into production
• Data security: Play key role with data security team to ensure no critical customer data become available to anyone and everyone; You could come up with the concept of data profiles to determine who could get access to what kind of data set.
• Feature engineering: Work with data scientists in identifying features (raw and derived features)
• Model acceptance: Work with data scientists on making sure that model of only optimal quality gets moved into production
• Serving model predictions using REST endpoint: Work with project manager to ensure that software systems are made available to take the models into production; Models will need to be exposed as endpoint, preferably REST end point for integration with products.
• Production deployments of models: Work with software engineering and data science team on integration of model predictions with software products
• Model performance monitoring: Work with data scientists to monitor the model performance at regular intervals and plan strategies for production deployments

Q6. What is your approach towards model governance / monitoring?

Model performance can be classified into three categories namely green zone, yellow zone and red zone. One needs to identify thresholds for putting the model performance in green, yellow and red zone. Based on which zone model performance is found, model is scheduled for retraining.

• Green zone: If model performance is above a particular threshold say 85-90%, the model can said to be in green zone. One may not need to do anything.
• Yellow zone: If the model performance is between say 60-70% to green zone threshold, the model falls in yellow zone and requires scrutiny.
• Red zone: If the model performance is less than a particular threshold say, 60%, the model gets scheduled to be retrained.

Q7. What technical metrics you use for measuring classification model performance?

The following represents technical metrics which is used for measuring classification model performance:

• Accuracy: Measures the total misclassification done by the model. It is calculated as the ratio of total correct classification and total predictions.
• Precision: It is calculated as the ratio of total correct positive prediction (same as actual value) and the total positive prediction.
• Recall: It is calculated as the ratio of total correct positive prediction (same as actual value) and actual positive values.
• F1-Score: It is measured as harmonic mean of precision and recall value.

Q8. Who all are required to form an AI team?

The following represents some of the key teams of an AI team:

• Business analysts / Product Managers: A bunch of product managers belonging to different product teams who identify the business problems which require to be solved using machine learning solutions.
• Data Science Team: This is a team of data scientists (junior, mid-level, senior) who would work on training / fitting / building machine learning models
• Data Engineering Team: These are a bunch of data engineers who are involved in creating big data solution which will help process large to very large volume of data needed for machine learning models
• Software Engg. Team: These are a bunch of software engineers / developers who deploy machine learning models in production and expose the models as endpoint, preferably REST end point. This team includes architects who would architect / design the software system.
• Project Manager: Project manager who manages the AI projects
• Cloud / IT / Infrastructure Team: These are bunch of cloud specialists who help software team deploy machine learning models on cloud platforms including AWS, Azure, Google etc.

Q9. What are some of the challenges of building machine learning products?

This is one of the most common interview questions asked to the product managers. Here are some of the key challenges of building machine learning products:

• Sponsorship / funding as it requires investment in setting up team, setting up the cloud infrastructure for model training / retraining, production deployments
• Identifying the real machine learning problems which solves a real world business problem
• Setting up AI Team for taking care of key aspects of building ML products such as data science, data engineering, software engineering, cloud / IT team
• Setting up business and technical metrics as solution KPIs for measuring the effectiveness of the machine learning solution
• Project and program management with internal and external stakeholders for tracking the implementation of ML projects and adoption of ML solutions
• Educating / training the stakeholders / end users / customers on effectiveness of AI / machine learning based solutions
• Monitoring / retraining models at regular intervals and setting up related governance processes
• Training / educating customer success team to interface the end users in appropriate manner while dealing with their queries in relation to ML based solutions