Prediction and interpretation are two different uses of a fitted model

I was trying to get my thoughts straight on what machine learning is for when the data come from experiments, so that I could say it in my own words. I had already asked an assistant what machine learning is used for in experimental research and development like mine, in chemistry and chemical engineering. Then I dictated my own version. Machine learning offers predictive modeling of datasets. On large experimental datasets the models fit well. On pilot data the same models do not, or at least it is not guaranteed. And for pilot-scale or process development work, prediction is not the only thing we want anyway. We want the relationships between the variables, the importance of each one, and the effect of each on the final product.

So there are two uses in that paragraph, and each has its own words. A fitted model is a function learned from a table. Each row is a run, the input columns are called features (temperature, feed type, reactor configuration) and the output column is called the target or the label (a yield). Prediction is the use everybody knows. You give the function a new row of features and read off the target. The other use is called interpretation, or model inspection. You ask the fitted function which features it leans on and in which direction, and you do not predict any new row.

This split is not new. Breiman in 2001 described two cultures of statistical modeling, one that assumes the data come from a stated model and reads its parameters, and one that treats the mechanism as unknown and judges an algorithm by how well it predicts.11Breiman (2001), abstract: the data modeling culture and the algorithmic modeling culture. Shmueli in 2010 argued that explaining and predicting are different goals that people mix up, and that the difference affects every step of the modeling.22Shmueli (2010), abstract: explanation and prediction as distinct purposes, and the conflation of explanatory with predictive power. I think an engineer already knows this split in a simpler form. A calibration curve is for predicting, because you read the signal and get the concentration. A regression of yield on temperature in a design of experiments is for explaining, because you want to know whether the slope is there and how large it is.

Interpretation has its own tools, and these are the ones I met.

Permutation importance takes a model that is already fitted, shuffles one feature column so it no longer lines up with the rows, and measures how much the score drops.33scikit-learn user guide, “Permutation feature importance”: the definition, the warning on low-importance features of a bad model, the statement that importance is specific to a particular model, and the note on correlated features. A big drop means the model was relying on that column.

SHAP values come from a 2017 paper by Lundberg and Lee. The method gives each feature an importance value for one particular prediction, so the prediction for one run is split into a contribution from each input.44Lundberg and Lee (2017), abstract: SHAP assigns each feature an importance value for a particular prediction. Averaged over all the runs that gives a ranking, and plotted against a feature it shows which way the effect goes.

An ablation study I had to ask about directly, what it is and what it tells us. It takes something out, an input or a part of the method, fits again, and compares the result with and without it. I think of it as running the unit with one stream shut off.

The documentation for these tools also says where they stop.

The scikit-learn guide says permutation importance shows how important a feature is for one particular model, and that a feature with low importance in a bad model could be very important in a good one, so the model has to predict well on held-out data before you read its importances.33scikit-learn user guide, “Permutation feature importance”: the definition, the warning on low-importance features of a bad model, the statement that importance is specific to a particular model, and the note on correlated features. So the two uses are tied together. If the model predicts badly, its ranking of the inputs means very little.

Then there are inputs that move together. If two features are correlated and you shuffle one, the model still gets the same information through the other, and both come out lower than they deserve.33scikit-learn user guide, “Permutation feature importance”: the definition, the warning on low-importance features of a bad model, the statement that importance is specific to a particular model, and the note on correlated features. In process data, two settings that were always changed together cannot be pulled apart by any ranking.

And a high importance does not mean cause. scikit-learn has an example called “Failure of Machine Learning to infer causal effects”, where a model scores almost as well with a variable left out and still gives a wrong answer about the effect of the one that stayed in.55scikit-learn example, “Failure of Machine Learning to infer causal effects”, including its lessons learned. The way out it points to is one an experimentalist already has, which is an experiment where you set that variable on purpose.

Where I have got to is that for pilot data I use the model mostly to find which variables matter and how, and much less to forecast. But one question still bothers me. If prediction accuracy is not the main goal of experimental and pilot-scale R&D, then what is?

Footnotes

1. Breiman (2001), abstract: the data modeling culture and the algorithmic modeling culture. ↩

2. Shmueli (2010), abstract: explanation and prediction as distinct purposes, and the conflation of explanatory with predictive power. ↩

3. scikit-learn user guide, “Permutation feature importance”: the definition, the warning on low-importance features of a bad model, the statement that importance is specific to a particular model, and the note on correlated features. ↩ ↩² ↩³

4. Lundberg and Lee (2017), abstract: SHAP assigns each feature an importance value for a particular prediction. ↩

5. scikit-learn example, “Failure of Machine Learning to infer causal effects”, including its lessons learned. ↩