Machine learning vocabulary in terms a process engineer already knows

I spent an evening on a course an assistant was giving me, and it went through the scikit-learn workflow, from exploring the data and preprocessing to cross-validation and hyperparameter tuning. scikit-learn is the Python library that holds most of the standard machine learning methods. I am a chemical engineer and I was just starting to learn, so a lot of that evening was new words. A few months later I ran my first modeling script on my own computer, and its first step reported how many samples and how many features it had loaded. Samples are runs and features are input columns. Most of these words are like that, a new name for something you already do with a run table or a calibration curve.

The table

A sample is one row of the data table, and for a pilot unit one row is one run. The scikit-learn glossary says other people call it an instance, a data point or an observation.11scikit-learn glossary, entries “sample”, “feature” and “target”.

A feature is an input column, something you set or know before the run gives a result, like a temperature setpoint or a feed type.11scikit-learn glossary, entries “sample”, “feature” and “target”. In code the block of feature columns is called X.

A target is the output column, the thing you want predicted, like a yield or a conversion. The glossary also lists outcome, response and label for it.11scikit-learn glossary, entries “sample”, “feature” and “target”. In code it is y. You fit a model with fit(X, y) and then ask it for predictions with predict(X).

Holding runs back

Think of a calibration curve. The training set is the runs the model is fitted on, like the standards you use to draw the curve. The test set is the runs you keep out of the fitting and use only to check the model, like a check standard that was not used to draw the curve. The scikit-learn guide gives the reason for it: a model that only repeated the values it had already seen would score perfectly and still fail to predict anything useful on a new run.22scikit-learn user guide, section 3.1: the opening paragraph on testing a prediction function on its training data and on overfitting; the description of k-fold cross-validation; the Leave One Out section; and the passage stating that preprocessing such as standardization should be learnt from a training set and applied to held-out data. The function that splits the table is train_test_split.

Cross-validation is the same check done in rotation. The runs are divided into groups called folds, the model is fitted on all the folds except one and checked on the one left out, then this is repeated until every fold has been the check once, and the scores are averaged.22scikit-learn user guide, section 3.1: the opening paragraph on testing a prediction function on its training data and on overfitting; the description of k-fold cross-validation; the Leave One Out section; and the passage stating that preprocessing such as standardization should be learnt from a training set and applied to held-out data. It is like rotating which standard you hold back. In scikit-learn it is KFold and cross_val_score.

Leave-one-out is cross-validation with one run in each fold. Each run is held back in turn, the model is fitted on all the others and then predicts the held-back one, so for n runs that is n fits.22scikit-learn user guide, section 3.1: the opening paragraph on testing a prediction function on its training data and on overfitting; the description of k-fold cross-validation; the Leave One Out section; and the passage stating that preprocessing such as standardization should be learnt from a training set and applied to held-out data. The name is LeaveOneOut.

Settings

A hyperparameter is a setting you choose before the fit, and a parameter is a number the fit finds. If you fit a polynomial to calibration data, the order of the polynomial is the hyperparameter and the coefficients are the parameters. The depth of a decision tree is another hyperparameter. The guide defines them as parameters that are not learnt directly within the estimator and are passed in when you construct it.33scikit-learn user guide, section 3.2: “Hyper-parameters are parameters that are not directly learnt within estimators”, passed as arguments to the constructor; GridSearchCV considers all combinations of given values and RandomizedSearchCV samples a given number of candidates; the final evaluation is to be done on an evaluation set of samples not seen during the search. GridSearchCV and RandomizedSearchCV are the tools that try many settings and compare them by cross-validation.33scikit-learn user guide, section 3.2: “Hyper-parameters are parameters that are not directly learnt within estimators”, passed as arguments to the constructor; GridSearchCV considers all combinations of given values and RandomizedSearchCV samples a given number of candidates; the final evaluation is to be done on an evaluation set of samples not seen during the search.

Overfitting is the calibration curve with too high an order. It goes through every standard, noise included, and then it is wrong for the next sample.22scikit-learn user guide, section 3.1: the opening paragraph on testing a prediction function on its training data and on overfitting; the description of k-fold cross-validation; the Leave One Out section; and the passage stating that preprocessing such as standardization should be learnt from a training set and applied to held-out data. You see it as a big gap between the score on the training set and the score on the runs held back.

A baseline is the simplest prediction you have to beat. For a yield that is the mean yield of the training runs, given for every run whatever the conditions were. scikit-learn has it as DummyRegressor, which by default always predicts the mean of the training set.44scikit-learn API reference, DummyRegressor: a regressor that makes predictions using simple rules, useful as a simple baseline; the default strategy “mean” always predicts the mean of the training set. If a model cannot beat that, it has learned nothing from the features.

Preparation and what is left over

Scaling puts the feature columns on comparable ranges, because a temperature in the hundreds of degrees and a flag that is 0 or 1 are very different sizes of number. StandardScaler subtracts the mean of each column and divides by its standard deviation. The part I think is easy to miss is that the scaler is fitted too. The guide says it should be learnt from the training set and then applied to the held-out runs.22scikit-learn user guide, section 3.1: the opening paragraph on testing a prediction function on its training data and on overfitting; the description of k-fold cross-validation; the Leave One Out section; and the passage stating that preprocessing such as standardization should be learnt from a training set and applied to held-out data.

A pipeline is a fixed sequence of steps, say scaler then model, handled as one object. I think of it as the written procedure, so the same preparation is done every time in the same order. And when you give the whole pipeline to cross-validation, the scaler is refitted on the training part of every split, so nothing about the held-back runs reaches the model. When information from the test runs does get into the model building it is called data leakage, and the documentation recommends the pipeline to prevent it.55scikit-learn, Common pitfalls and recommended practices, section on data leakage: the definition, the rule never to call fit on the test data, and the recommendation of the pipeline for cross-validation and hyperparameter tuning. The names are Pipeline and make_pipeline.

A residual is measured minus predicted for one run, the same as in any regression, and you read it the same way. Plot it against the predicted value or against a feature and it should show no pattern. scikit-learn has no object for it, it is just y - model.predict(X).

Word	What it is on the plant side	scikit-learn name
Feature	Setpoint or feed property	X
Target	Yield	y
Training set	Calibration standards	first outputs of train_test_split
Test set	Check standard	second outputs of train_test_split
Cross-validation	Rotating the check standard	KFold, cross_val_score
Leave-one-out	One check standard at a time	LeaveOneOut
Hyperparameter	Order of the calibration polynomial	constructor argument, GridSearchCV
Overfitting	Curve through every point	none, seen as a gap between training and test scores
Baseline	Mean of past runs	DummyRegressor
Scaling	Comparable ranges for all columns	StandardScaler
Pipeline	Written procedure	Pipeline, make_pipeline
Residual	Measured minus predicted	none, y - model.predict(X)

Where the calibration comparison stops

The calibration comparison works for what the words mean, and I think it stops working when you ask how much data you need. A calibration has one input and a known form, so a few standards are enough. A model with several features and no assumed form needs many more rows before it can say anything, and a run table from a pilot unit has few.

One thing about the comparison above. I used the check standard for the test set and also for the fold held back inside cross-validation. The user guide keeps a separate evaluation set for the final score, one that was never seen while the settings were being searched,33scikit-learn user guide, section 3.2: “Hyper-parameters are parameters that are not directly learnt within estimators”, passed as arguments to the constructor; GridSearchCV considers all combinations of given values and RandomizedSearchCV samples a given number of candidates; the final evaluation is to be done on an evaluation set of samples not seen during the search. so those two are different things and I have used one picture for both.

Footnotes

1. scikit-learn glossary, entries “sample”, “feature” and “target”. ↩ ↩² ↩³

2. scikit-learn user guide, section 3.1: the opening paragraph on testing a prediction function on its training data and on overfitting; the description of k-fold cross-validation; the Leave One Out section; and the passage stating that preprocessing such as standardization should be learnt from a training set and applied to held-out data. ↩ ↩² ↩³ ↩⁴ ↩⁵

3. scikit-learn user guide, section 3.2: “Hyper-parameters are parameters that are not directly learnt within estimators”, passed as arguments to the constructor; GridSearchCV considers all combinations of given values and RandomizedSearchCV samples a given number of candidates; the final evaluation is to be done on an evaluation set of samples not seen during the search. ↩ ↩² ↩³

4. scikit-learn API reference, DummyRegressor: a regressor that makes predictions using simple rules, useful as a simple baseline; the default strategy “mean” always predicts the mean of the training set. ↩

5. scikit-learn, Common pitfalls and recommended practices, section on data leakage: the definition, the rule never to call fit on the test data, and the recommendation of the pipeline for cross-validation and hyperparameter tuning. ↩