library / lib2d25dfa5d4547668
Designing Machine Learning Systems
Chip Huyen · 2022
In a sentence
A holistic, iterative framework for designing production-ready machine learning systems that are reliable, scalable, maintainable, and adaptive across every stage from data engineering to continual learning.
Designing Machine Learning Systems by Chip Huyen offers the comprehensive, end-to-end guide that ML engineers and data scientists have long needed to bridge the gap between academic model-building and the messy realities of production. Rather than treating the ML algorithm as the centerpiece, Huyen situates it as just one small component within a much larger system encompassing business objectives, data pipelines, feature engineering, deployment infrastructure, monitoring, and responsible AI. Drawing on her experience at NVIDIA, Netflix, Snorkel AI, and Stanford—where she teaches the course CS 329S: Machine Learning Systems Design—she walks readers through every stage of the ML project lifecycle with concrete case studies, trade-off discussions, and practical frameworks. The book covers everything from sampling strategies and labeling techniques, through model development and offline evaluation, to online prediction, data distribution shift detection, continual learning, MLOps infrastructure, and the human and ethical dimensions of deploying AI at scale. Whether you are deploying your first model or managing hundreds in production, this book provides the principled vocabulary and decision-making framework to do it right.
The four lenses
- Science
- Statistics
- Systems
- Strategy
Tags
The model
A causal model describing how design levers across the ML lifecycle—data quality, feature engineering, model development practices, deployment architecture, monitoring infrastructure, and organizational structure—produce intermediate system and behavioral states that ultimately drive production ML outcomes including reliability, business value, and responsible deployment.
Business Objective Alignmentdesign lever
The degree to which ML project goals, chosen metrics, and model optimization targets are explicitly mapped to and validated against measurable business outcomes such as revenue, conversion rate, cost reduction, or user retention, rather than being evaluated solely on academic ML metrics like accuracy or F1.
Training Data Qualitydesign lever
The composite quality of the dataset used to train ML models, encompassing representativeness of the real-world distribution, label accuracy, coverage of rare classes, absence of data leakage, freshness relative to the deployment distribution, and freedom from systematic sampling biases introduced during collection or labeling.
Feature Engineering Qualitydesign lever
The quality of the feature set provided to ML models, reflecting the predictive power of selected features, absence of data leakage, generalizability of feature values across train and production distributions, appropriate handling of missing values, and correct feature scaling and encoding relative to model assumptions.
Model Development Rigordesign lever
The thoroughness and discipline applied during the model development phase, including use of appropriate baselines, iterative experimentation starting from simple models, experiment tracking and versioning, slice-based and calibration-aware offline evaluation, hyperparameter tuning, and awareness of model assumptions relative to data properties.
Deployment Architecture Fitdesign lever
The degree to which the chosen prediction-serving architecture—batch versus online prediction, cloud versus edge, streaming pipeline integration—is matched to the latency, throughput, freshness, and cost requirements of the specific use case, including unification of training and serving feature pipelines to prevent train-serve skew.
Monitoring and Observability Maturitydesign lever
The extent to which an ML system is instrumented to continuously track operational metrics, ML-specific metrics (predictions, features, accuracy proxies), and data distribution statistics, enabling rapid detection of model degradation, data distribution shifts, and pipeline failures, with actionable alerts and sufficient telemetry for root-cause analysis.
Continual Learning Infrastructuredesign lever
The technical and organizational capability to update deployed ML models continuously or on-demand through automated pipelines that access fresh labeled data, retrain models (statefully or from scratch), evaluate updates safely before promotion, and deploy changes rapidly—enabling models to adapt to data distribution shifts without manual intervention.
MLOps Infrastructure Qualitydesign lever
The overall maturity and fit-for-purpose quality of the infrastructure stack supporting ML development and operations, including the development environment, resource management tooling, model store, feature store, and deployment services, as evaluated by how much engineering overhead these tools impose on data scientists versus how much they abstract and automate.
Responsible AI Practicesdesign lever
The degree to which systematic, proactive processes are embedded in the ML lifecycle to identify, measure, and mitigate biases, ensure fairness across demographic subgroups, maintain privacy, provide transparency through model cards and interpretability tools, and establish accountability mechanisms—acting early rather than treating ethics as a post-deployment concern.
Data Distribution Shiftcontextual condition
The divergence between the statistical distribution of data encountered by a deployed model in production and the distribution of data on which the model was trained, including covariate shift (input distribution changes), label shift (output distribution changes), concept drift (relationship between input and output changes), and feature or label schema changes, all of which degrade model predictive accuracy over time.
Model Accuracy (Offline)outcome metric
The measured predictive performance of an ML model on held-out evaluation data prior to deployment, encompassing aggregate metrics (accuracy, F1, AUC-ROC, calibration) as well as fine-grained slice-based performance across critical subgroups, evaluated against meaningful baselines including random, zero-rule, human, and existing system benchmarks.
Train-Serve Feature Consistencypsychological state
The degree to which the features computed during model training match the features computed at inference time, reflecting whether the same feature definitions, encodings, scaling statistics, and pipeline code are used in both contexts, and whether any divergence (train-serve skew) introduces systematic prediction errors in production.
Model Freshnesspsychological state
The degree to which a deployed ML model has been trained on data that is recent enough to accurately reflect the current production data distribution, capturing recent behavioral patterns, trends, and concept changes that would otherwise cause degraded predictions due to model staleness.
Degenerate Feedback Loop Riskcontextual condition
The tendency of a deployed ML system whose predictions influence user behavior, which in turn generates training labels, to reinforce and amplify initial biases over time—leading to increasingly homogeneous outputs, suppression of tail items or underrepresented groups, and self-fulfilling predictions that diverge from true user preferences or real-world distributions.
Production Model Performanceoutcome metric
The actual predictive accuracy and behavioral quality of a deployed ML model as measured on live production traffic, including natural label feedback, user engagement signals, and A/B test outcomes—representing the ultimate empirical test of model quality under real-world distribution, latency, and usage conditions.
Business Value Realizedoutcome metric
The measurable impact of the ML system on organizational business outcomes—including revenue uplift, cost savings, customer retention improvement, conversion rate gains, or operational efficiency—as distinct from ML performance metrics, representing the ultimate justification for ML investment and the metric business stakeholders actually care about.
System Reliability in Productionoutcome metric
The operational dependability of the end-to-end ML system in production, measured by uptime, latency SLA adherence, absence of silent failures, and the speed with which failures are detected and remediated—encompassing both software system reliability and ML-specific silent failure modes where predictions degrade without operational errors.
Fairness and Harm Mitigation in Productionoutcome metric
The degree to which a deployed ML system produces equitable outcomes across demographic subgroups and avoids disproportionate harm to underrepresented or vulnerable populations, as evidenced by slice-level performance parity, absence of disparate impact on protected classes, and absence of documented harmful incidents attributable to model predictions.
How they connect
- business objective alignment → influences model development rigor
- business objective alignment → predicts business value realized
- training data quality → predicts model accuracy offline
- training data quality − influences data distribution shift
- feature engineering quality → predicts model accuracy offline
- feature engineering quality → influences train serve consistency
- model development rigor → predicts model accuracy offline
- model accuracy offline → predicts production model performance
- train serve consistency → influences production model performance
- deployment architecture fit → influences production model performance
- deployment architecture fit → influences system reliability in production
- data distribution shift − influences production model performance
- monitoring observability maturity − influences data distribution shift
- monitoring observability maturity → predicts system reliability in production
- continual learning infrastructure → predicts model freshness
- model freshness → predicts production model performance
- degenerate feedback loop − influences production model performance
- mlops infrastructure quality → predicts continual learning infrastructure
- mlops infrastructure quality → predicts system reliability in production
- responsible ai practices → predicts fairness and harm mitigation
- responsible ai practices → influences production model performance
- production model performance → predicts business value realized
- monitoring observability maturity → influences continual learning infrastructure
- data distribution shift → moderates model freshness
The story
The reader ML engineers, data scientists, and engineering managers who want to build ML systems that actually work reliably in production—not just in notebooks—and who are frustrated by the gap between academic ML and real-world deployment complexity.
External problem
Their models perform well in development but degrade, fail silently, or cause harmful outcomes once deployed to real users at scale.
Internal problem
They feel overwhelmed, under-equipped, and uncertain about what to do next when production systems break in ways that unit tests and accuracy scores never revealed.
Philosophical problem
It is wrong for powerful ML systems to be deployed without the discipline, infrastructure, and ethical safeguards that protect users and society from their failures.
The plan
- Establish business objectives and translate them into ML objectives and system requirements (reliability, scalability, maintainability, adaptability).
- Frame the ML problem correctly: choose the right task type, objective function, and decoupled multi-objective structure.
- Master data engineering fundamentals: data sources, formats, storage engines, and dataflow modes.
- Create high-quality training data through principled sampling, labeling strategies (natural labels, weak supervision, active learning), and handling class imbalance and data augmentation.
- Engineer features carefully, avoiding data leakage, prioritizing feature importance and generalization.
- Develop models iteratively, starting simple, using ensembles where justified, tracking experiments, and evaluating with baselines plus slice-based and calibration-aware methods.
- Deploy models with an understanding of batch versus online prediction trade-offs, model compression, and edge-versus-cloud considerations.
- Monitor production systems continuously for data distribution shifts, using statistical methods and time-windowed telemetry.
- Implement continual learning infrastructure so models can be updated as frequently as business value requires.
- Build or buy the right MLOps infrastructure—dev environment, resource management, model store, feature store—and embed responsible AI practices throughout.
Success
- ML models remain accurate and reliable long after deployment, with degradation detected and corrected quickly.
- Teams move from manual, months-long model update cycles to automated, data-driven retraining triggered by real performance signals.
- Data scientists own the full ML lifecycle confidently, supported by infrastructure that abstracts away operational complexity.
- ML systems earn the trust of users and society because they are fair, transparent, well-monitored, and built with responsible AI practices from day one.
- Business stakeholders can clearly see how ML investments translate to measurable business outcomes.
At stake
- Models deployed without monitoring degrade silently, eroding user trust and business value until a crisis forces an expensive rebuild.
- Teams remain stuck in manual, ad hoc update cycles that cannot keep pace with shifting data distributions.
- Biased or opaque ML systems cause harm to underrepresented users, triggering public backlash, regulatory action, and organizational damage.
- Without proper infrastructure, ML projects remain one-off experiments that never reach the scale or reliability needed for real business impact.
Chapter by chapter
ch01Overview of Machine Learning Systems
The chapter outlines the essential components and considerations for operationalizing machine learning systems, emphasizing the distinction between machine learning in research and production environments.
- Machine learning is not merely about algorithms; it encompasses a systematic approach that includes data, stakeholder engagement, and operational processes.
- Successful ML systems in production are characterized by constant adaptation to changing patterns and rigorous monitoring of model performance.
- Understanding the differences between ML in research and production is crucial for deploying ML effectively in a real-world context.
- Stakeholder alignment on project goals and requirements can significantly enhance the effectiveness of ML deployments.
ch02Introduction to Machine Learning Systems Design
This chapter emphasizes the critical importance of aligning machine learning (ML) systems with business objectives, detailing essential requirements for their design and the iterative process required for successful implementation.
ch03Data Engineering Fundamentals
This chapter introduces the foundational concepts of data engineering, emphasizing the intricacies of data sources, formats, models, and storage techniques crucial for building machine learning systems.
- The relationship between machine learning and big data is critical, demanding a solid understanding of data engineering basics for successful implementation.
- Recognizing the importance of formatting and structuring data enhances the efficiency of ML systems.
- Data models are integral as they dictate how information is organized, which directly impacts system performance and integrity.
- The ETL process remains central to effective data management, ensuring data is clean, relevant, and ready for analysis.
ch04Data Engineering Fundamentals
This chapter contends that understanding the nuances of data passing through various architectural frameworks is essential for managing efficient data flow in modern applications, particularly in environments shaped by real-time requirements.
ch05Training Data
This chapter navigates the critical yet often overlooked realm of training data in machine learning, addressing essential techniques and challenges in obtaining and preparing data that significantly impact model performance.
- Training data is foundational for successful machine learning applications, warranting careful management and preparation.
- Sampling methods can introduce significant biases if not approached with a robust understanding of the underlying population.
- Label acquisition poses operational challenges that can be mitigated through innovative strategies like weak and semi-supervision.
- Class imbalance must be addressed proactively to ensure ML models are equitable and effective, particularly in critical applications like healthcare and finance.
ch06Feature Engineering
This chapter argues that effective feature engineering is the cornerstone of successful machine learning models, emphasizing its role in substantially improving performance beyond advanced algorithms.
ch07Model Development and Offline Evaluation
This chapter navigates the complexities of selecting and evaluating machine learning (ML) models, emphasizing a systematic approach to development, performance evaluation, and iterative improvement before deployment.
- Machine learning model development is an iterative process that thrives on continuous evaluation and refinement.
- It is crucial to select models based on problem-specific requirements rather than the latest trends or perceived ‘state-of-the-art’ techniques.
- Employing ensemble methods can lead to significant performance improvements by leveraging the strengths of multiple models.
- Comprehensive experiment tracking and versioning practices are vital for reproducible results and effective team collaboration.
ch08Model Development and Offline Evaluation
The chapter presents a comprehensive approach to developing machine learning models, emphasizing the crucial aspects of model evaluation, especially through offline methodologies, to ensure robust performance in real-world applications.
- Implementing a systematic approach to model development rooted in phases—ranging from basic heuristics to complex models—can fundamentally enhance performance outcomes.
- Baseline evaluations are essential to contextualizing model performance; without them, metrics lose their meaning and can lead to faulty conclusions.
- Incorporating evaluation methodologies such as perturbation and invariance tests helps in understanding how models may perform under various real-world conditions, including exposure to noise.
- Emphasizing slice-based evaluation can help avoid biases and ensure that models serve all segments of users fairly, avoiding issues like Simpson's Paradox.
ch09Model Deployment and Prediction Service
Deploying machine learning models is a critical step that transforms theoretical constructs into accessible, real-time applications; this chapter dissects the nuances, challenges, and methodologies of effective deployment.
- Deployment is not just an afterthought; it represents a critical phase that determines the long-term success of machine learning initiatives.
- Continuous updates to ML models should be the norm, not the exception; as user data evolves, so should the computations to maintain relevance.
- The choice between online and batch prediction carries significant implications for user experience and system architecture.
- Understanding environmental factors—whether cloud or edge—can drastically alter deployment outcomes and operational efficiency.
ch10Data Distribution Shifts and Monitoring
The chapter investigates the critical issue of data distribution shifts in machine learning (ML) models, arguing that continuous monitoring and adaptation are essential to maintain model performance over time.
- ML models require continuous monitoring to maintain effectiveness post-deployment, as performance can degrade due to distribution shifts.
- Understanding the three types of data distribution shifts—covariate shift, label shift, and concept drift—is essential for anticipating model performance issues.
- Software system failures also affect ML systems, emphasizing the need for traditional engineering practices alongside ML-specific monitoring.
- Robust monitoring frameworks are essential for capturing both operational and ML-specific metrics to preemptively detect performance issues.
ch11Continual Learning and Test in Production
This chapter addresses the critical need for ongoing adaptation of machine learning models to data distribution shifts through continual learning, emphasizing the infrastructure required for efficient updates and the practice of testing models in production.
- Continual learning is fundamentally an infrastructural challenge that can enhance the adaptability of machine learning systems to data distribution shifts.
- Employing micro-batching and stateful training methodologies can yield significant improvements in model performance and resource efficiency.
- The champion-versus-challenger model strategy is crucial for safe deployments, reducing the risk of catastrophic failures in production environments.
- Flexibility in retraining schedules allows organizations to remain agile in response to changing data landscapes.
ch12Continual Learning and Test in Production
This chapter argues for the imperative of continual learning in machine learning systems to ensure adaptability in rapidly shifting data environments, emphasizing practical challenges and strategies for implementation.
ch13Infrastructure and Tooling for MLOps
Navigating the complexities of machine learning (ML) infrastructure is essential for practitioners to effectively implement ML systems and avoid stagnation due to inadequate tooling and support.
- Adequate infrastructure is a prerequisite for successful ML implementation; neglecting this leads to operational bottlenecks.
- Organizations vary in their infrastructure requirements; no one-size-fits-all solution exists.
- Standardized environments enhance productivity and reduce friction in development, facilitating smoother transitions to production.
- The cloud provides scalable solutions but consider workload variations that may prompt a return to on-prem solutions as organizations mature.
ch14The Human Side of Machine Learning
This chapter emphasizes the critical role of human factors—user experience, team structure, and societal impacts—in the design and implementation of machine learning systems.
ch15Epilogue
The epilogue reflects on the journey of learning and applying machine learning principles, emphasizing the potential for innovation while acknowledging the challenges that lie ahead.
Related in the library