What a SHAP value says and what it does not

I was fitting machine learning models to a small table of runs. Modeling in the browser was taking too long, so I moved the work to my own computer, and the code came from an assistant one step at a time. The first step was an install line, and one of the libraries in it was called shap.

pip install pandas numpy scikit-learn xgboost shap matplotlib seaborn

By the end of that day I had a SHAP bar chart, a SHAP beeswarm and a SHAP plot against temperature, all from models I had fitted earlier the same day. Each figure was a few lines of code. So then I had to find out what those figures let me say about the process behind the runs.

What a SHAP value is

SHAP stands for SHapley Additive exPlanations. It is a method, and a Python library with the same name, that takes one prediction from a fitted model and divides it among the inputs of the model.11Lundberg and Lee (2017), abstract: SHAP assigns each feature an importance value for a particular prediction.

Say the model predicts a yield for one run. The model also has an average prediction over all the runs it was given. The prediction for this run sits above or below that average, and SHAP splits the difference among the inputs of the run, so much to the temperature and so much to the feed. The shares add up exactly to the difference. The idea comes from game theory, where a group of players earns a payout together and you want to divide it fairly when they did not contribute equally. The share of one input is found by adding it to every possible combination of the other inputs and averaging how much the prediction moves each time.22Molnar (2025), chapter on Shapley values: the definition as the average marginal contribution across coalitions, the efficiency property, the statement of what the value is not, and the limits on interpretation, reference data and correlated features. For a process engineer I think the nearest thing is splitting a shared utility cost among the units that use it.

A SHAP value has the units of what the model predicts. If the model predicts a yield in weight percent, then a value of plus 2 for temperature on one run means that, in this model’s accounting, the temperature of that run put the prediction 2 points above the average prediction.

Reading the beeswarm

The plot you see most is the beeswarm, which the library also draws with shap.summary_plot. Every run is one dot on the line of each input. How far the dot sits left or right is the SHAP value of that input for that run, and the color is the value of the input itself, low to high. Where many runs have similar values the dots pile up.33SHAP documentation, beeswarm plot: one dot per instance per feature, x position by SHAP value, color by original feature value, default ordering by mean absolute value and the alternative ordering by maximum absolute value. So you read each line twice. First you look at how far the dots spread from zero, because that is how much the input moves the predictions. Then you look at whether the colors sort along the axis, because if the high values sit on the right and the low ones on the left, the prediction goes up with that input. The inputs are stacked from top to bottom by the mean of their absolute SHAP values, and the bar chart that comes with the beeswarm is only that mean.33SHAP documentation, beeswarm plot: one dot per instance per feature, x position by SHAP value, color by original feature value, default ordering by mean absolute value and the alternative ordering by maximum absolute value. The documentation says that this ordering favors an input with a broad average effect over one that matters a lot for a few runs, and that you can order by the maximum absolute value to find the second kind.33SHAP documentation, beeswarm plot: one dot per instance per feature, x position by SHAP value, color by original feature value, default ordering by mean absolute value and the alternative ordering by maximum absolute value. With a small table where one group has only a handful of runs, I think that second ordering is worth the extra line.

When I redid the figures about two weeks later, the beeswarm cell failed because shap.summary_plot did not accept an ax argument, which is the usual way to tell a plotting function which panel to draw in. It got fixed.

What it does not tell you

All of this is about the fitted model. The shares add up to the model’s prediction, and the measured value does not come into it, so if the model predicts a run badly, SHAP explains the bad number completely. Whether the model predicts well on runs it has not seen has to be shown somewhere else, by validation, because the plot cannot show it.

The other thing is cause. A model reproduces the associations in the data it was trained on, and SHAP displays them. An article in the library’s own documentation, with the first author of the method among its writers, says “making correlations transparent does not make them causal”.44Dillon et al., SHAP documentation: the quoted sentence, and the conditions of independence from other features and from unobserved confounders. They give the condition under which a causal reading is safe, which is that the input has to be independent of the other inputs and of anything unmeasured that also drives the outcome.44Dillon et al., SHAP documentation: the quoted sentence, and the conditions of independence from other features and from unobserved confounders. Data from experiments stand a little better here than plant operating records, because the experimenter sets the factors and they do not drive one another. But anything that moved with a setpoint and was not recorded is carried inside the share of that setpoint. A residence time that shifts with temperature would be one, or a feeder that behaves differently with one material.

There are four common readings of the plot that it does not support.

The first is that the value is what would be lost if the input were removed. Molnar says plainly that the Shapley value is not the difference in prediction when the feature is taken out of the model.22Molnar (2025), chapter on Shapley values: the definition as the average marginal contribution across coalitions, the efficiency property, the statement of what the value is not, and the limits on interpretation, reference data and correlated features. To answer that you refit without the input, and that is a different experiment.

The second is that the value is a slope. It cannot tell you how much the prediction changes for a given change in the input. Molnar suggests pairing it with curves that vary one input and hold the rest.22Molnar (2025), chapter on Shapley values: the definition as the average marginal contribution across coalitions, the efficiency property, the statement of what the value is not, and the limits on interpretation, reference data and correlated features.

The third is that a high rank means the input is important in the process. The values are relative to the data they were computed on and to the particular model.22Molnar (2025), chapter on Shapley values: the definition as the average marginal contribution across coalitions, the efficiency property, the statement of what the value is not, and the limits on interpretation, reference data and correlated features. Another model fitted to the same runs may divide the same prediction differently.

The fourth is that correlated inputs are divided cleanly. When inputs are correlated the calculation evaluates the model at combinations that never occur in reality, and the shares become unreliable.22Molnar (2025), chapter on Shapley values: the definition as the average marginal contribution across coalitions, the efficiency property, the statement of what the value is not, and the limits on interpretation, reference data and correlated features.

Footnotes

1. Lundberg and Lee (2017), abstract: SHAP assigns each feature an importance value for a particular prediction. ↩

2. Molnar (2025), chapter on Shapley values: the definition as the average marginal contribution across coalitions, the efficiency property, the statement of what the value is not, and the limits on interpretation, reference data and correlated features. ↩ ↩² ↩³ ↩⁴ ↩⁵

3. SHAP documentation, beeswarm plot: one dot per instance per feature, x position by SHAP value, color by original feature value, default ordering by mean absolute value and the alternative ordering by maximum absolute value. ↩ ↩² ↩³

4. Dillon et al., SHAP documentation: the quoted sentence, and the conditions of independence from other features and from unobserved confounders. ↩ ↩²