Kernel models in plain terms

I first met support vector regression one evening when I was working through scikit-learn in a course I was taking with an assistant, and it was one item in a list that started at linear regression and went on through random forests and XGBoost. A few months later it was one of the models in a tuning search I ran on my own computer, and it was one of the quick ones to tune.

What a kernel is

A kernel is a function that takes two rows of the data table, two runs, and gives back a number that says how alike their conditions are. The common one is the radial basis function, RBF. In scikit-learn it is exp(-gamma d²), where d is the distance between the two runs in the space of their inputs.11scikit-learn user guide, section 1.4: SVR with C and epsilon, dependence on a subset of the training data, the RBF kernel and gamma, the recommendation to scale the data with a pipeline, the advice on C for noisy observations and on searching C and gamma. Two runs at the same conditions score 1, and the score falls toward 0 as the conditions move apart.

A kernel model predicts a new run from the training runs, each one weighted by how alike it is to the new one. It is like interpolating in a steam table, where the two neighbouring entries do all the work. The kernel does the same with several inputs at once, and the influence of a run fades smoothly with distance.

How fast it fades is the main setting. In support vector regression it is gamma, and the larger it is, the closer a run has to be to count.11scikit-learn user guide, section 1.4: SVR with C and epsilon, dependence on a subset of the training data, the RBF kernel and gamma, the recommendation to scale the data with a pipeline, the advice on C for noisy observations and on searching C and gamma. In a Gaussian process the same thing is given as a length_scale, which Rasmussen and Williams describe as roughly the distance you have to move in input space before the function value can change significantly.22Rasmussen and Williams (2006), chapter 2: the definition of a Gaussian process, the predictive mean and variance, the characteristic length scale, the cost of the Cholesky decomposition, and Figure 2.5 on error bars and length scales. If you put the two formulas side by side, gamma is 1 over twice the square of the length scale.11scikit-learn user guide, section 1.4: SVR with C and epsilon, dependence on a subset of the training data, the RBF kernel and gamma, the recommendation to scale the data with a pipeline, the advice on C for noisy observations and on searching C and gamma. 33scikit-learn user guide, section 1.7: hyperparameters fitted by maximizing the log marginal likelihood, n_restarts_optimizer, alpha and WhiteKernel, the RBF kernel exp(-d²/(2 l²)). So a large gamma and a short length scale say the same thing: only very near runs count, and the fitted surface can wiggle a lot.

Why the inputs are scaled first

The similarity comes from a distance, and a distance adds up the differences across all the input columns in whatever units they are in. If one column is a temperature that spans several hundred degrees and another is a 0 or 1 code, then the temperature decides the distance almost alone. The scikit-learn guide says support vector machines are not scale invariant and that scaling the data is highly recommended, for example standardizing each input to mean 0 and variance 1.11scikit-learn user guide, section 1.4: SVR with C and epsilon, dependence on a subset of the training data, the RBF kernel and gamma, the recommendation to scale the data with a pipeline, the advice on C for noisy observations and on searching C and gamma. In code you tie the scaler to the model in one line, and then every new row gets the same scaling:11scikit-learn user guide, section 1.4: SVR with C and epsilon, dependence on a subset of the training data, the RBF kernel and gamma, the recommendation to scale the data with a pipeline, the advice on C for noisy observations and on searching C and gamma.

model = make_pipeline(StandardScaler(), SVR(kernel="rbf", C=1.0, epsilon=0.1))

Support vector regression

Support vector regression looks for a function that stays within a distance epsilon of the training targets and is otherwise as flat as possible. Smola and Schölkopf say errors smaller than epsilon are not cared about.44Smola and Schölkopf (2004), section 1.2: the goal of epsilon support vector regression and the role of the constant C. I read epsilon as a tolerance band, the plus or minus inside which a yield counts as reproduced. A point inside the band costs nothing.

Real data seldom fit inside a band, so a miss beyond it is allowed at a price, and C sets the price. It is the trade between how flat the function is and how much deviation beyond epsilon is tolerated.44Smola and Schölkopf (2004), section 1.2: the goal of epsilon support vector regression and the role of the constant C. A high C follows the training runs closely. A low C gives a smoother function and accepts some misses, and the scikit-learn guide says to lower it when the observations are noisy.11scikit-learn user guide, section 1.4: SVR with C and epsilon, dependence on a subset of the training data, the RBF kernel and gamma, the recommendation to scale the data with a pipeline, the advice on C for noisy observations and on searching C and gamma.

The name comes from the band. The fitted function depends only on some of the training runs, the ones on or outside the band, because the cost ignores the runs whose prediction is already close to the target.11scikit-learn user guide, section 1.4: SVR with C and epsilon, dependence on a subset of the training data, the RBF kernel and gamma, the recommendation to scale the data with a pipeline, the advice on C for noisy observations and on searching C and gamma. Those runs are the support vectors.

So the settings are C, epsilon and gamma. The guide calls the choice of C and gamma critical and says to search them over values spaced exponentially.11scikit-learn user guide, section 1.4: SVR with C and epsilon, dependence on a subset of the training data, the RBF kernel and gamma, the recommendation to scale the data with a pipeline, the advice on C for noisy observations and on searching C and gamma. What you get back is one number for each prediction, and nothing that says how far to trust it.

Gaussian process regression

A Gaussian process starts from the same kernel. It treats the unknown function as random, and it uses the kernel to say how strongly the outputs at two conditions are correlated, strongly when the conditions are close and weakly when they are far apart.22Rasmussen and Williams (2006), chapter 2: the definition of a Gaussian process, the predictive mean and variance, the characteristic length scale, the cost of the Cholesky decomposition, and Figure 2.5 on error bars and length scales. From the training runs it then gives, for any new condition, a predicted mean and a predicted variance.22Rasmussen and Williams (2006), chapter 2: the definition of a Gaussian process, the predictive mean and variance, the characteristic length scale, the cost of the Cholesky decomposition, and Figure 2.5 on error bars and length scales. In scikit-learn you ask for it in the call:

mean, std = gpr.predict(X_new, return_std=True)

The standard deviation is small near the training runs and grows away from them.22Rasmussen and Williams (2006), chapter 2: the definition of a Gaussian process, the predictive mean and variance, the characteristic length scale, the cost of the Cholesky decomposition, and Figure 2.5 on error bars and length scales. For someone who plans experiments I think this is the useful part, because the model marks where in the operating window it is guessing.

The settings are handled differently too. The length scale and the noise level are fitted to the data by maximizing a quantity called the log marginal likelihood. That surface can have several local optima, so scikit-learn has n_restarts_optimizer to repeat the fit from different starting values.33scikit-learn user guide, section 1.7: hyperparameters fitted by maximizing the log marginal likelihood, n_restarts_optimizer, alpha and WhiteKernel, the RBF kernel exp(-d²/(2 l²)). Measurement noise goes in through alpha or through a WhiteKernel term.33scikit-learn user guide, section 1.7: hyperparameters fitted by maximizing the log marginal likelihood, n_restarts_optimizer, alpha and WhiteKernel, the RBF kernel exp(-d²/(2 l²)).

It has a cost in arithmetic. The basic algorithm works on a matrix with one row and one column for each training run, and the work grows with the cube of the number of runs.22Rasmussen and Williams (2006), chapter 2: the definition of a Gaussian process, the predictive mean and variance, the characteristic length scale, the cost of the Cholesky decomposition, and Figure 2.5 on error bars and length scales. At a few tens of runs that is nothing.

The other cost is that the standard deviation is computed from the model’s own assumptions, which are the kernel you chose, the smoothness it implies and noise of constant size. Rasmussen and Williams show the same data explained by a short length scale with almost no noise and by a long length scale with a lot of noise, and the error bars are very different.22Rasmussen and Williams (2006), chapter 2: the definition of a Gaussian process, the predictive mean and variance, the characteristic length scale, the cost of the Cholesky decomposition, and Figure 2.5 on error bars and length scales. So the interval depends on the kernel, and it does not measure experimental error. For that you still need replicate runs.

Both models fall back to something flat outside the training range. Far from every training run the RBF similarity is zero for all of them, so the SVR prediction goes to a constant and the Gaussian process goes back to its prior mean, with the standard deviation at its largest.

Footnotes

1. scikit-learn user guide, section 1.4: SVR with C and epsilon, dependence on a subset of the training data, the RBF kernel and gamma, the recommendation to scale the data with a pipeline, the advice on C for noisy observations and on searching C and gamma. ↩ ↩² ↩³ ↩⁴ ↩⁵ ↩⁶ ↩⁷ ↩⁸

2. Rasmussen and Williams (2006), chapter 2: the definition of a Gaussian process, the predictive mean and variance, the characteristic length scale, the cost of the Cholesky decomposition, and Figure 2.5 on error bars and length scales. ↩ ↩² ↩³ ↩⁴ ↩⁵ ↩⁶

3. scikit-learn user guide, section 1.7: hyperparameters fitted by maximizing the log marginal likelihood, n_restarts_optimizer, alpha and WhiteKernel, the RBF kernel exp(-d²/(2 l²)). ↩ ↩² ↩³

4. Smola and Schölkopf (2004), section 1.2: the goal of epsilon support vector regression and the role of the constant C. ↩ ↩²