Week 10 — Nov 16: Machine Learning & Chemometrics
(Lecture 9.) Last week set the rules — what makes a computational result trustworthy, and where an AI system may influence a regulated decision. This week is the method: the mathematics that actually builds those models, and the discipline that keeps them honest. Next week is the deployment.
The one idea
Chemometrics is machine learning — it just had a 40-year head start in regulated analytical science, so it carries things generic ML underplays: interpretability (loadings are chemistry), small-n honesty, and a validation tradition. The whole field, chemometrics and modern ML alike, makes it easy to build a model that fits your data and hard to build one that predicts new data. Almost everything below is a safeguard against mistaking the first for the second.
One continuum
| Linear latent-variable (classic chemometrics) | Nonlinear / modern ML | |
|---|---|---|
| Methods | PCA, PLS, PLS-DA, MCR, PCR | Random forests, gradient boosting, SVM, neural nets, deep learning on raw spectra |
| Best when | Relationships are roughly linear; n is small; you must explain the model | Genuinely nonlinear response; complex matrices; image or high-dimensional data; n is large |
| Cost | Limited flexibility | Less interpretable, more data-hungry, applicability domain harder to see |
| Regulated setting | The default — transparent, established | Used where it clearly wins, with extra credibility evidence |
The regulated setting pushes toward the transparent end, but the discipline — training/test separation, applicability domain, drift monitoring, reproducibility — is identical across the continuum.
Preprocessing — part of the method, not a tidy-up
Spectra carry the signal you want plus physical effects you don’t: scattering, baseline offsets, path-length and density variation, drift. Preprocessing removes those so the model sees chemistry:
| Method | Removes | Note |
|---|---|---|
| Mean-centering | The common offset — always done | |
| Scaling (autoscale, Pareto) | Differences in variable magnitude | Autoscale gives every variable equal weight — powerful and dangerous |
| SNV / MSC | Multiplicative scatter, path-length variation | The default for diffuse-reflectance NIR |
| Savitzky–Golay derivatives (1st, 2nd) | Baseline slope (1st) and offset+slope (2nd) | Amplifies noise — needs smoothing; changes peak appearance |
| Normalisation | Overall intensity differences |
Preprocessing choices are locked with the model and revalidated if changed — as much part of the method as the column and mobile phase in an HPLC method.
PCA — look before you model
Principal component analysis re-expresses many correlated variables as a few uncorrelated components:
- Scores — where each sample sits; reveals clustering by batch, site, operator, season.
- Loadings — how the wavelengths combine into each component; interpretable as chemistry.
- Hotelling’s T² — how far a sample is from the centre within the model; Q-residual (DModX) — how far it is from the model. Together they are the outlier and out-of-domain detectors a running PAT model depends on.
PLS — the calibration workhorse
Partial least squares regresses spectra against a reference value (assay, moisture, particle size) through a few latent variables built to be relevant to y, not just to explain X. The one parameter that matters and the one place people cheat:
- Number of latent variables — too few underfits, too many fits noise. Chosen by cross-validation — but the cross-validation must reflect how the model will be used.
- The error metrics — RMSEC (calibration) always improves with more LVs. RMSECV and especially RMSEP (independent test set) are what count. A large RMSECV–RMSEP gap means overfitting or a genuinely different test set.
- VIP scores and regression coefficients — check they line up with known chemistry; a model leaning on a meaningless wavelength is leaning on an artefact.
MCR — resolving mixtures
Multivariate curve resolution (MCR-ALS) takes a matrix of mixed spectra — a reaction over time, a peak cluster, a Raman map — and resolves it into pure-component spectra and concentration profiles with minimal assumptions. The catch is rotational ambiguity: without constraints (non-negativity, unimodality, known spectra) there is a family of valid solutions, and choosing among them is chemical judgment.
Design of experiments and the design space
Planning data is the flip side of analysing it. Factorial, fractional-factorial, response-surface, and D-optimal designs map how several factors jointly affect a response with far fewer runs than one-factor-at-a-time — and reveal interactions. This is the machinery behind the analytical robustness study and the method operable design region (Q14) and the manufacturing design space (Q8).
Worked case — the model that passed cross-validation and failed in production
A PLS model for tablet assay by NIR: 6 latent variables, RMSECV 0.9%, R² 0.99. In production it was biased 2–3% and trended with batch.
The cause: the calibration set had 3 replicate spectra per tablet, and cross-validation left out random spectra, not whole tablets or batches. Replicates of the same tablet sat on both sides of every split, so the model was scored on data it had effectively seen — information leakage between training and test, the failure the AI week flagged as generic to ML. Re-run with batch-blocked cross-validation, the honest RMSECV was 2.1% and the model needed rebuilding with a wider set.
The lesson is the one idea restated: the model fit beautifully and predicted badly, and only a validation design that mirrored real use revealed the gap.
The notebook thread
This week is the centre of the shared Python/Jupyter thread (Week 9 → here → Week 11): load real spectral data, preprocess it, fit PLS and a tree ensemble, and — the point — build the validation that tells them apart, then show what an out-of-domain sample does to the prediction and to the T²/Q diagnostics that are supposed to catch it.
Where the analyst sits
The software will return a model for any dataset. Deciding whether it has learned chemistry or memorised noise — reading loadings as functional groups, insisting on a test set that reflects real variation, blocking cross-validation by the right factor, knowing when a sample is outside the model’s world — is irreducibly the analyst’s judgment. It is the STEAM “M” and “A” at once, and it is why Week 1 said every scientist is now a data scientist. The refrain: science → evidence → reduced uncertainty → control → regulatory confidence → patient trust.
For discussion
- Why does RMSEC always improve as you add latent variables, while RMSEP eventually gets worse? What is happening to the model?
- You have 40 tablets, 3 NIR spectra each. Describe a cross-validation scheme that will not lie to you, and one that will.
- A PCA of incoming API batches shows one supplier’s material forming a separate cluster, though every batch passed spec. What do you do with that information?
- MCR-ALS gives you a “pure” spectrum for an unknown reaction intermediate. How much do you trust it, and what constraints would you add first?
- When is it worth moving from PLS to a neural network for a quantitation model in a regulated setting? What would you give up?
- Preprocessing is “part of the method.” A colleague wants to change 1st to 2nd derivative because it “looks cleaner.” What has to happen?
Source note. Follows the standard chemometrics literature (Brereton, Chemometrics; Martens & Næs, Multivariate Calibration; Esbensen) and ASTM E1655, with ISO 12099 and the NIR/multivariate regulatory guidance referenced in Week 5. DoE / design space connects to Q8 and Q14; model governance to Week 9. The overfitting/leakage case is a composite of commonly reported failures. (Instructor: confirm the software the class will use; decide how the notebook thread is split across Weeks 9–11.)