On the process for building software with ML components

ML Components in Software Systems

A simple architectural diagram of a transcription service
Sloppy architectural diagram of a possible online transcription service, with speech recognition as the central ML components and several other non-ML components.

Let’s talk about process

Simple iterative process model (software development life cycle)
System development process by decomposing the problem and integrating the components
Figure from Dogru, Ali H., and Murat M. Tanik. “A process model for component-oriented software engineering.” IEEE Software 20, no. 2 (2003): 34–41.

So, back to ML…

Machine learning workflow model from Microsoft
Machine Learning Workflow from Amershi, Saleema, et al. “Software engineering for machine learning: A case study.” 2019 IEEE/ACM 41st International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP). IEEE, 2019.
Three different machine learning life cycle models
left: CRISP-DM (cross-industry standard process for data mining) model, middle: Microsoft’s Team Data Science Process; right: refined CRISP-DM model from a recent study in the FinTech industry
Revision of previous process model, integrating the machine learning workflow into the box of one component

Beyond the ML model…

  • The pipeline component, that is, a component that takes data, processes it, and produces and packages an ML model.
  • The monitoring component, that is, a component that takes telemetry data collected by other parts of the system and produces reports and possibly triggers the pipeline component to rebuild a model.
  • The actual ML model component produced by the pipeline and integrated into the production system.
Revised architectural model that highlights monitoring and pipeline as two additional components
Revised process model that adapts the machine learning workflow in one component to emphasize the outputs
Unrolled process model showing an ML and a non-ML component in parallel

Thinking through Engineering Challenges for ML-Enabled Systems

  • Designing user interfaces has direct impact on what data can be collected as telemetry for retraining and monitoring. This requires careful thinking about how multiple non-ML and ML components interact and what the interfaces for the pipeline and monitoring components are.
  • Quality assurance needs to be planned holistically: Evaluating accuracy of the ML model is only one step, similar to unit testing. We also need to create unit tests for the pipeline component and the model component, as emphatically argued by this excellent paper from Google. And of course, we need integration testing and system testing that looks at the interactions of all these components, including one or more ML components.
  • Data quality is really at the interface between the ML pipeline component and the rest of the system. It requires planning and negotiation at the system level and likely requires to describe data quality requirements as part of the interface. The same holds for assumptions about data schema and drift.
  • Fairness considerations are really about requirements engineering and require to synchronize the system requirements with those in the machine learning pipeline, with consequences for how the pipeline component and probably also the monitoring component are implemented, but also with consequences for other components, say regarding what data is collected. This requires paying attention to fairness as part of system requirements, as part of high-level design, and then as part of several ML and non-ML components.
  • Safety is a system-level property and can, despite a lot of research on robustness of ML models, not be achieved at the level of the ML model alone. It’s really about designing how the ML model component interacts with other (safety) components of the system, such as fallback mechanisms, users interactions, and much more.

--

--

--

associate professor @ Carnegie Mellon; software engineering, configurations, open source, SE4AI, juggling

Love podcasts or audiobooks? Learn on the go with our new app.

Recommended from Medium

NLP Twitter Sentiment Classification Tutorial

My Recent Experience In Combinatorial Optimization — Journey From O(N⁴) To O(N)

Using C# & ML.NET to Predict Video Game Ratings

How to train an Object Detector with your own COCO dataset in PyTorch

The Class Every Reinforcement Learning Researcher Should Take

Kubernetes and Amazon SageMaker for machine learning — distributed training, hyperparameter tuning…

How I Taught A Machine To Take My Job

Classifying Cancer Tumours | Beginner’s Self Project

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
Christian Kästner

Christian Kästner

associate professor @ Carnegie Mellon; software engineering, configurations, open source, SE4AI, juggling

More from Medium

Getting the Most From Your Google Vertex ML Architecture

Machine Learning Pipelines: Everything You Need to Know

Machine Learning Pipelines

Improving the Machine Learning Lifecycle

Understanding ML monitoring debt