A small neural network in plain terms

When I started on machine learning I had no understanding of neural networks, and I told the assistant so. About three months later a small neural network was one of the models I was fitting on my own computer.

The network I mean is a multilayer perceptron, MLP for short. It is the plainest kind of neural network, and it is the one scikit-learn gives you for a table of numbers, under the name MLPRegressor.

Units and layers

Start with one unit. A unit takes the numbers handed to it, multiplies each one by a weight, adds them up, and then passes the sum through a fixed curve called the activation function.11scikit-learn user guide, section 1.17: the weighted sum followed by an activation, the layers, the identity output and squared error for regression, training by backpropagation, the listed disadvantages, and the tips on scaling, pipelines and alpha. A weight is like a fitted coefficient in a regression. It is a number the fitting adjusts.

Then the units are arranged in layers. The input layer has one unit for each input column, so if your table has a temperature and a reactor type, that is two units. The output layer gives the prediction, a yield for example. In between are the hidden layers, and each unit in a hidden layer takes its inputs from every unit in the layer before it.11scikit-learn user guide, section 1.17: the weighted sum followed by an activation, the layers, the identity output and squared error for regression, training by backpropagation, the listed disadvantages, and the tips on scaling, pipelines and alpha. For regression the output unit is a plain weighted sum with no curve, and the fit minimizes the squared error, which is the same quantity a least squares line minimizes.11scikit-learn user guide, section 1.17: the weighted sum followed by an activation, the layers, the identity output and squared error for regression, training by backpropagation, the listed disadvantages, and the tips on scaling, pipelines and alpha.

The setting that fixes the shape is hidden_layer_sizes. The value (10, 10) means two hidden layers of ten units each. If you set nothing you get one layer of 100.22MLPRegressor reference: hidden_layer_sizes default (100,), activation options and default relu, the solver note on small datasets and lbfgs, alpha default 0.0001, max_iter default 200, tol, random_state. The count of adjustable numbers grows fast. Say there are three inputs, two hidden layers of ten units and one output. Then I count 40 weights and offsets for the first hidden layer, 110 for the second and 11 for the output, so 161 in all. On a table of a few dozen runs, even a small network like this one has several times more adjustable numbers than there are rows.

This setting also gave me a bug. I had saved the layer sizes of tuned networks in a CSV as text like “10_10”. When the file was loaded again, the loader read that column as integers, so “10_10” became 1010 and “20_10” became 2010, and no error was thrown. I found it later, fixed the values and ran the predictions again.

The activation

Without the curve the layers would do nothing, because a weighted sum of weighted sums is still a weighted sum, so the whole network would be a linear model however many layers it had. The activation bends the output of each unit, and many bent pieces stacked together can follow a curved response, like a yield that rises with temperature and then levels off. In scikit-learn the choices are identity, logistic, tanh and relu, and relu is the default.22MLPRegressor reference: hidden_layer_sizes default (100,), activation options and default relu, the solver note on small datasets and lbfgs, alpha default 0.0001, max_iter default 200, tol, random_state. Relu passes a positive sum through unchanged and sets a negative one to zero.

Training and epochs

The weights start as random numbers. Training shows the network the rows, measures the error, and works out for every weight which direction would reduce the error. Then the weights move a small step and the cycle repeats. The procedure that passes the error back through the layers is called backpropagation.11scikit-learn user guide, section 1.17: the weighted sum followed by an activation, the layers, the identity output and squared error for regression, training by backpropagation, the listed disadvantages, and the tips on scaling, pipelines and alpha.

An epoch is one pass through all the training rows. In scikit-learn the setting max_iter caps the number of passes at 200 by default.22MLPRegressor reference: hidden_layer_sizes default (100,), activation options and default relu, the solver note on small datasets and lbfgs, alpha default 0.0001, max_iter default 200, tol, random_state. Training stops earlier if the error stops improving by more than a tolerance. If it reaches the cap first, the library prints a convergence warning. On one long job on my PC the terminal printed those warnings the whole time. As I understand it, the warning means the fit was stopped before it settled, and the usual answer is more iterations or a different solver. The documentation says the default solver, adam, works well on datasets with thousands of rows, and that for small datasets lbfgs can converge faster and perform better.22MLPRegressor reference: hidden_layer_sizes default (100,), activation options and default relu, the solver note on small datasets and lbfgs, alpha default 0.0001, max_iter default 200, tol, random_state.

Tuning a network is also slow, and the size of the table does not tell you that in advance. In my tuning search the MLP was one of the slow models.

Scaling the inputs

A furnace temperature runs to several hundred degrees. A reactor type coded as 0 or 1 never goes above one. If both go raw into a weighted sum, the temperature swamps the code. The user guide says the model is sensitive to feature scaling, so each input should be brought to a common range or standardized to mean 0 and variance 1, and the same scaling has to be applied to the rows held out for testing.11scikit-learn user guide, section 1.17: the weighted sum followed by an activation, the layers, the identity output and squared error for regression, training by backpropagation, the listed disadvantages, and the tips on scaling, pipelines and alpha.

The part that is easy to get wrong is where the scaling is learned. If the mean and the spread are computed on all the rows before validation, then the held-out row has already had a say in the model. So the guide recommends putting the scaler and the network in one pipeline,11scikit-learn user guide, section 1.17: the weighted sum followed by an activation, the layers, the identity output and squared error for regression, training by backpropagation, the listed disadvantages, and the tips on scaling, pipelines and alpha. and then each validation split learns its scaling from its own training rows.

from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.neural_network import MLPRegressor

model = make_pipeline(
    StandardScaler(),
    MLPRegressor(hidden_layer_sizes=(10, 10), max_iter=2000, random_state=0),
)

Why the fit changes from one start to the next

The user guide lists this as a disadvantage. A network with hidden layers has an error surface with more than one minimum, so different random starting weights can end in different fits with different validation scores.11scikit-learn user guide, section 1.17: the weighted sum followed by an activation, the layers, the identity output and squared error for regression, training by backpropagation, the listed disadvantages, and the tips on scaling, pipelines and alpha. On a small table I think it is easy to see why. There are more adjustable numbers than rows, so many different sets of weights reproduce the training rows about equally well, and they do not agree with each other in between those rows. Which set you get depends on where the training started.

Setting random_state to a fixed integer makes one fit repeatable.22MLPRegressor reference: hidden_layer_sizes default (100,), activation options and default relu, the solver note on small datasets and lbfgs, alpha default 0.0001, max_iter default 200, tol, random_state. That gives you the same start every time, and it says nothing about whether another start would have done better. The other handle is alpha, a penalty on large weights that defaults to 0.0001, and the guide suggests searching it over several orders of magnitude.11scikit-learn user guide, section 1.17: the weighted sum followed by an activation, the layers, the identity output and squared error for regression, training by backpropagation, the listed disadvantages, and the tips on scaling, pipelines and alpha. 22MLPRegressor reference: hidden_layer_sizes default (100,), activation options and default relu, the solver note on small datasets and lbfgs, alpha default 0.0001, max_iter default 200, tol, random_state.

Footnotes

1. scikit-learn user guide, section 1.17: the weighted sum followed by an activation, the layers, the identity output and squared error for regression, training by backpropagation, the listed disadvantages, and the tips on scaling, pipelines and alpha. ↩ ↩² ↩³ ↩⁴ ↩⁵ ↩⁶ ↩⁷ ↩⁸

2. MLPRegressor reference: hidden_layer_sizes default (100,), activation options and default relu, the solver note on small datasets and lbfgs, alpha default 0.0001, max_iter default 200, tol, random_state. ↩ ↩² ↩³ ↩⁴ ↩⁵ ↩⁶