Building Machine Learning Models & Dev Challenges

The machine learning models and AI implementation industry is booming. The demand for machine learning models has never been higher, but the challenges of machine learning development and deployment have also increased. In this post, we will discuss a few common machine learning development and deployment challenges. In future blogs, we will learn about solutions to overcome these challenges. This blog post will help you learn and understand some of the key challenges that you may face if you are planning to start machine learning practice in your organization. These challenges are also very much relevant if you have machine learning engineers and data scientists working across different offices/locations on different products in your global organization. Check out my post on what is machine learning?

The following is the list of the pain points/opportunities you might face while establishing machine learning models development and deployment practices:

• Need for Data Security across AI/ML Pipeline: Some of the key requirements related to building ML models are data preparation and exploratory data analysis. In this relation, data scientists across different offices would need easy and quick access to data while meeting the data security requirements. Different offices may have different levels of data scientists. In one location, the data scientists may be mostly interns who may not be given full data access due to different security reasons. Rather, they can be provided with selected data sets consisting of selected columns from selected tables in order to meet data security requirements. This throws up the challenge such as a need for data security, database, and data science team to work on data preparation, data masking, assessing data security and making data available in the desired format such as CSV. And, this requires the regular intervention of this team due to the need to constantly assess data security requirements. Given that the data science teams in current times may be working remotely, this throws up another layer of security challenges from the data management standpoint. From a business standpoint, this would not only impact data science team productivity but also delay the deployment of models in production leading to business impacts. Data security as a challenge is not only limited to machine learning model development and deployment but also impacts the entire data science pipeline.
• Data Management related Challenges: Given the constant need to manage and monitor data preparation/processing and data access across different office managers from a security standpoint, the database and data science team along with the data security team will need to collaborate at regular intervals to make sure data scientists have access to right data set in a secured manner. In this relation, the database team and the data science teams would need to collaborate at regular intervals for data preparation.
• Limited Data Availability for Feature Engineering / Building Models: Given that the business problems might cut across different product lines, data science team members working on the problems only get to access the data from one or only a few databases. This limits the ability to come up with a great set of features that could span across different product databases. For example, the team working on the particular problem do not get to study the data from other database related to other product which could provide useful insights in analyzing the problem and solution approaches. This is where the need for a data lake comes in. The data lake can help bring all the data sets together in one place. However, this comes with its own set of challenges.
• Computing intensive Feature Engineering / Data Processing: Given that laptops are having computing resources constraints vis-a-vis data / big data processing requirements, data scientists across different locations will be constrained to work on only a selected set of problems (building models) with only a limited set of features where data volume is not large enough. Given that your organization might be having clients having big data, this is going to impact business sooner than later in terms of providing high-quality AI-powered solutions in a timely manner.
• Computing intensive big data processing in production: In production, what is needed is the ability to process the predictions in a faster and efficient manner. This would require a large/big data infrastructure that would be able to support prediction requirements within the desired time period given computing-intensive feature calculations which are required to be done during runtime. What is desired is the distributed calculations for faster processing of predictions.
• Longer lead time for production deployments owing to Big Data: Given that there can be the need to deploy client-specific or problem-specific models in production in a faster manner, the fact that the data science team performs data processing and model building on his/her laptop would start becoming a big constraint. This will, in turn, lead to delays in relation to breaking data in chunks and then processing them on one’s laptop. Additionally, the need for training the model with a large volume of data would become a constraint due to the lack of big data infrastructure. Thus, this would result in delays related to building models and moving them in production.
• Limited data science skillset across different office locations: When you are operating across multiple different office locations, not everyone has similar machine learning and data science skills. This means that the team is going to be a mix of people with machine learning expertise as well as those without machine learning capabilities. This would mean having multiple different skill sets within the same data science team which will impact collaboration among them in terms of building machine learning models.
• Machine learning models collaborative development environments: When you have distributed teams, machine learning models development and collaboration becomes a challenge because machine learning model building tools & workflows are usually not well-defined. This means that there will be multiple different machine learning development environments that will confuse model-building teams due to lack of collaboration (model/code review) and hence slow down model development and deployment.

There are a variety of machine learning models development challenges that need to be overcome before these models can be deployed in production. One such challenge is the difficulty associated with training models on a global organization’s large data sets across different products, which may require more computing power than what laptops currently provide. Other challenges include limited access to data needed for feature engineering, difficulties related to deploying machine-learning models in production, and longer lead times for deployment due to big data processing requirements.  If you want help overcoming these challenges or would like some guidance on how best to develop and deploy machine learning models by leveraging distributed teams and cloud infrastructure, please reach out to us.