A Panorama of ML System Design: Inference, Training, Data, and Deployment

I’ve recently been working through Stanford’s CS 329S: Machine Learning Systems Design, a course dedicated to the overall design of ML systems, covering a fairly broad range of topics. Here are the parts of the course I found worth noting.

Layered view of an ML system: Interface, Data and ML algorithms, Infrastructure, Hardware, from top to bottom — Figure 1: A layered view of an ML system — Interface, Data, ML algorithms, Infrastructure, Hardware — which the course works through layer by layer

A number of noteworthy keywords show up in the course: DevOps, CI/CD, A/B testing, Flink (stream processing), Microservices and REST APIs, Kubernetes, and tinyML (a new book based on TensorFlow Lite). Overall, the course feels quite comprehensive when it comes to production deployment.

Inference, Computation, and Learning Paradigms

The first few slides systematically lay out the different modes of inference, computation, and learning.

On inference modes, it’s not black-and-white — multiple approaches can be mixed:

Comparison table of batch prediction vs online prediction: frequency, use cases, optimization target, input space, examples — Figure 2: Batch prediction vs. online prediction — batch prediction runs periodically, optimizes throughput, and has a finite input space (e.g. TripAdvisor ranking, Netflix recommendations); online prediction fires as requests arrive, optimizes low latency, and can have an infinite input space (e.g. speech recognition, Twitter feed)

Online prediction is the default, with common queries precomputed and cached: DoorDash and Netflix examples — Figure 3: Online prediction is the default, but common queries are precomputed and stored — DoorDash uses batch predictions for restaurant recommendations and online predictions for in-restaurant item recommendations; Netflix uses batch predictions for title recommendations and online predictions for row ordering

On computation modes:

	Cloud computing	Edge computing
Computations	Done on the cloud (servers)	Done on edge devices (browsers, phones, tablets, laptops, smart watches, activity watchers, cars, etc.)
Requirements	Network connections: availability and speed for data transfer	Hardware: memory, compute power, energy for doing computations
Examples	Most queries to Alexa / Siri / Google Assistant; Google Translate for rare language pairs (e.g. English–Yiddish)	Wake words for Alexa / Siri / Google Assistant; Google Translate for popular language pairs (e.g. English–Spanish); predictive text; unlocking with fingerprints or faces

Four-quadrant chart with hardware constraint on the y-axis and model inference latency on the x-axis — Figure 4: A four-quadrant view of hardware constraint (y-axis) vs. model inference latency (x-axis) — as hardware grows more powerful, applications shift overall from “cloud + high latency” toward “edge + low latency”

On learning paradigms:

	Offline learning	Online learning
Iteration cycle	Periodical (months)	Continual (minutes) — note: ≠ continuous
Batch size	batch (thousands → millions of samples); GPT-3 125M params: batch size 0.5M; GPT-3 175B params: batch size 3.2M	microbatch (hundreds of samples)
Data usage	Each sample seen multiple times (epochs)	Each sample seen at most once
Evaluation	Mostly offline evaluation	Offline evaluation as a sanity check; mostly relying on online evaluation (A/B testing)
Examples	Most applications	TikTok recommendation system, Twitter hashtag trending

One point here is worth calling out on its own: when there are multiple optimization objectives, the course recommends splitting them into multiple models, each focused on a single metric — this makes both training and tuning easier.

Three reasons to split multiple objectives into separate models: easier for training, easier to tweak the system, easier for maintenance — Figure 5: Why split multiple objectives into separate models — easier for training (single-objective optimization beats multi-objective), easier to tweak the system (e.g. α% model optimized for quality + β% for engagement), and easier for maintenance (different objectives need different schedules; e.g. spam-filtering systems need updates more frequently than quality-ranking systems)

Data Storage and Feature Management

This part covers two main approaches to data storage.

Row-based storage, organized similarly to numpy, is suited to scenarios with frequent INSERTs:

OLTP (OnLine Transaction Processing) slide: handling many small transactions, ACID requirements, INSERT/UPDATE/DELETE operations, with a row-oriented SQL INSERT example — Figure 6: OLTP (OnLine Transaction Processing) — handles large numbers of small transactions (ordering food, rides, transferring money), requires ACID (Atomicity, Consistency, Isolation, Durability), and its operations are mainly INSERT/UPDATE/DELETE on row-organized data (INSERT INTO RideTable …)

Column-based storage, similar to pandas, is suited to scenarios with frequent SELECTs:

OLAP (OnLine Analytical Processing) slide: aggregating information from large data, complex queries, mostly SELECT operations, with a column-oriented SQL SELECT example — Figure 7: OLAP (OnLine Analytical Processing) — gets aggregated information from large amounts of data (e.g. last month’s average ride price), can handle complex queries on large volumes while tolerating slower responses, and its operations are mostly SELECT on column-organized data (SELECT AVG(Price) FROM RideTable …)

The two storage formats can be converted between via ELT:

ETL (Extract, Transform, Load) diagram: data flows from OLTP through extract-transform-load into OLAP — Figure 8: ETL (Extract, Transform, Load) — data flows from OLTP through “extract → transform → load” into OLAP, where Transform is the meaty part (cleaning, validating, transposing, deriving values, joining multiple sources, deduplicating, splitting, aggregating, etc.)

The course also mentions combining static data with dynamic data for inference:

Static data	Streaming data
CSV, PARQUET, etc.	Kafka, Kinesis, etc.
Bounded: you know when a job finishes	Unbounded: never finishes
Static features: age, gender, job, city, income; when the account was created; rating	Dynamic features: locations in the last 10 minutes; recent activities
Can be processed in batch (e.g. SQL, MapReduce)	Processed as events arrive (e.g. Apache Flink, Samza)

One model, two pipelines: inference uses streaming data through stream processing, training uses static data through batch processing, both feeding the same ML model — Figure 9: One model, two pipelines — at inference, streaming data goes through stream processing to produce features; at training, static data goes through batch processing to produce features; both ultimately feed the same ML model

The remainder of the third slide deck covers some transfer-learning material, which I’m skipping for now.

Sampling and Handling Class Imbalance

The fourth slide deck focuses mainly on sampling and class imbalance; the speaker appears to come from a traditional ML theory background. There are three main ways to address class imbalance: resampling, weight balancing, and ensembles.

Resampling comes in two flavors: downsampling and oversampling.

Undersampling vs oversampling comparison and illustration: undersampling removes majority-class samples (can cause overfitting), oversampling copies minority-class samples (can cause loss of information) — Figure 10: Undersampling vs. oversampling — undersampling removes samples from the majority class and can cause overfitting; oversampling adds examples to the minority class and can cause loss of information

Downsampling can use the Tomek Links method (reference: https://www.kaggle.com/rafjaa/resampling-strategies-for-imbalanced-datasets):

Tomek Links undersampling illustration: find close pairs of opposite-class samples and remove the majority-class one to make the decision boundary clearer — Figure 11: Tomek Links undersampling — find pairs of close samples from opposite classes and remove the majority-class sample in each pair (pro: makes the decision boundary clearer; con: makes the model less robust)

Oversampling can use SMOTE:

SMOTE oversampling illustration: synthesize minority-class samples as convex (roughly linear) combinations of existing points and their same-class nearest neighbors — Figure 12: SMOTE oversampling — synthesizes minority-class samples as convex (roughly linear) combinations of existing points and their same-class nearest neighbors, generating synthetic instances in the minority-class region

For weight balancing, the classic approach is Focal Loss:

p_{\mathrm{t}} = \begin{cases} p & \text{if } y = 1 \\ 1 - p & \text{otherwise,} \end{cases}

\mathrm{CE}(p_{\mathrm{t}}) = -\log(p_{\mathrm{t}})

\mathrm{FL}(p_{\mathrm{t}}) = -(1 - p_{\mathrm{t}})^{\gamma} \log(p_{\mathrm{t}})

Ensemble methods train multiple classifiers and then ensemble all the results:

Bagging ensemble illustration: sample with replacement to create multiple datasets, train a classifier on each, then aggregate all classifiers' predictions — Figure 13: Bagging — sample with replacement to create several different datasets (Bootstrapping), train a classifier on each, then aggregate (Aggregating) all classifiers’ predictions into an ensemble classifier

Boosting ensemble illustration: iteratively train a weak classifier, give misclassified samples more weight, then repeat on the reweighted data — Figure 14: Boosting — train a weak classifier, give the samples it misclassifies more weight, then repeat on the reweighted data, iteratively aggregating into an ensemble classifier

On data augmentation, there’s a 2019 survey worth a look: https://journalofbigdata.springeropen.com/articles/10.1186/s40537-019-0197-0

The latter part of deck 4 mainly covers feature engineering, data leakage, and model selection, still in a fairly traditional ML style. Deck 5 is all PyTorch, with nothing particularly remarkable.

Parallel Training and System Testing

Deck 6 covers parallelism, including data parallelism and model parallelism, but not in much detail. In addition, deck 6 and the first half of deck 7 both spend a fair amount of space on testing ML systems, especially data testing:

Different kinds of testing mapped onto the system layers: data testing targets Data, model evaluation targets ML algorithms, pipeline testing and system benchmarking cover broader scopes — Figure 15: How different tests map onto the ML system layers — data testing targets the Data layer, model evaluation targets the ML algorithms layer, pipeline testing spans Data and Infrastructure, and system benchmarking covers the whole system

Definitions of four kinds of testing — data testing, pipeline testing, system benchmarking, model evaluation — annotated on the system-layers diagram — Figure 16: Definitions of four kinds of testing — data testing (ensuring new data satisfies your assumptions), pipeline testing (ensuring the pipeline is set up correctly), system benchmarking (reporting overall performance on well-defined tasks/metrics/rules, for comparison), and model evaluation (evaluating how good the ML model itself is)

Experiment-Management Tools

The second half of deck 7 highlights two tools.

The first is Weights & Biases (wandb), which offers rich visualization features, a bit like NNI.

The second is DVC, which feels a bit like git — it manages training data the way git manages code:

DVC workflow diagram: code pushed/pulled to a Git server via git push/pull, data pushed/pulled to S3/Azure/GCS via dvc push/pull, with a small local .dvc file pointing to the large model file — Figure 17: The DVC workflow — code is synced to a Git server (GitHub/GitLab, etc.) via git push/pull, while data is synced to remote storage (S3/Azure/GCS/SSH) via dvc push/pull; locally you keep only a 1KB model.pkl.dvc pointer linked to the 500MB actual model file

Beyond managing data, DVC can also manage experiments, similar to NNI, making it easy to compare results across different experiments; finally, it also supports CI/CD pipelines.

Model Deployment

Deck 8 focuses on deployment. The earlier portion touches on model compression and TensorRT, while the later portion introduces two deployment approaches.

Approach one: deploy directly on a cloud platform, for example using GCP (similar to Alibaba Cloud):

Flowchart for deploying a model on the cloud: after train→compress→export, upload the model via 'Create predictor→Hosted model' or 'Create function→Endpoint', or build an app via 'Create app→Sync app→Hosted app' — Figure 18: Deploying a model on the cloud — after training, compressing, and exporting the model, you can upload it and Create a predictor for a Hosted model (GCP AI Platform / AWS SageMaker), Create a function exposing an Endpoint (GCP Cloud Functions / AWS Lambda), or Create an app synced from a git repo for a Hosted app (GCP App Engine / AWS EC2)

Approach two: deploy via Docker containers, which can likewise run on GCP; there are also VM-based and Kubernetes-based options. In practice, the difference between using Docker containers and using VMs is fairly small at the moment:

Line chart: monthly share of Stack Overflow questions for the 'containers' and 'virtual-machine' tags over the years, with containers climbing steadily after 2015 — Figure 19: The monthly share of Stack Overflow questions for the containers and virtual-machine tags over time — questions about containers keep rising after 2015, reflecting the growing adoption of containerization

The slides after tinyML are mostly fairly vague ML industry practices, which didn’t feel worth going through; I’ll come back to them another time.

MLSys
System Design
Deep Learning

2021 · 04 · 18