Misc edits to keras lecture

Author: jstac

.gitignore

@@ -1,5 +1,6 @@
 .DS_Store
 _build/
 lectures/_build/
+.ipynb_checkpoints/
 .ipython_checkpoints
-.virtual_documents
+.virtual_documents

lectures/keras.md

@@ -4,7 +4,7 @@ jupytext:
     extension: .md
     format_name: myst
     format_version: 0.13
-    jupytext_version: 1.16.4
+    jupytext_version: 1.16.7
 kernelspec:
   display_name: Python 3 (ipykernel)
   language: python
@@ -58,7 +58,7 @@ Next we import some tools from Keras.
 
 ```{code-cell} ipython3
 import keras
-from keras.models import Sequential
+from keras import Sequential
 from keras.layers import Dense
 ```
 
@@ -70,22 +70,30 @@ The data has the form
 
 $$
     y_i = f(x_i) + \epsilon_i,
-    \qquad i=1, \ldots, n
+    \qquad i=1, \ldots, n,
 $$
 
-The map $f$ is specified inside the function and $\epsilon_i$ is an independent
-draw from a fixed normal distribution.
+where 
+
+* the input sequence $(x_i)$ is an evenly-spaced grid,
+* $f$ is a nonlinear transformation, and
+* each $\epsilon_i$ is independent white noise.
 
 Here's the function that creates vectors `x` and `y` according to the rule
 above.
 
 ```{code-cell} ipython3
-def generate_data(x_min=0, x_max=5, data_size=400):
+def generate_data(x_min=0,           # Minimum x value
+                  x_max=5,           # Max x value
+                  data_size=400,     # Default size for dataset
+                  seed=1234):
+    np.random.seed(seed)
     x = np.linspace(x_min, x_max, num=data_size)
-    x = x.reshape(data_size, 1)
-    ϵ = 0.2 * np.random.randn(*x.shape)
+    
+    ϵ = 0.2 * np.random.randn(data_size)
     y = x**0.5 + np.sin(x) + ϵ
-    x, y = [z.astype('float32') for z in (x, y)]
+    # Keras expects two dimensions, not flat arrays
+    x, y = [np.reshape(z, (data_size, 1)) for z in (x, y)]
     return x, y
 ```
 
@@ -115,43 +123,61 @@ x_validate, y_validate = generate_data()
 
 We supply functions to build two types of models.
 
+## Regression model
+
 The first implements linear regression.
 
 This is achieved by constructing a neural network with just one layer, that maps
 to a single dimension (since the prediction is real-valued).
 
-The input `model` will be an instance of `keras.Sequential`, which is used to
+The object `model` will be an instance of `keras.Sequential`, which is used to
 group a stack of layers into a single prediction model.
 
 ```{code-cell} ipython3
-def build_regression_model(model):
-    model.add(Dense(units=1))
+def build_regression_model():
+    # Generate an instance of Sequential, to store layers and training attributes
+    model = Sequential()
+    # Add a single layer with scalar output
+    model.add(Dense(units=1))  
+    # Configure the model for training
     model.compile(optimizer=keras.optimizers.SGD(), 
                   loss='mean_squared_error')
     return model
 ```
 
-In the function above you can see that we use stochastic gradient descent to
-train the model, and that the loss is mean squared error (MSE).
+In the function above you can see that 
+
+* we use stochastic gradient descent to train the model, and
+* the loss is mean squared error (MSE).
+
+The call `model.add` adds a single layer the activation function equal to the identity map.
 
 MSE is the standard loss function for ordinary least squares regression.
 
+### Deep Network
+
 The second function creates a dense (i.e., fully connected) neural network with
 3 hidden layers, where each hidden layer maps to a k-dimensional output space.
 
 ```{code-cell} ipython3
-def build_nn_model(model, k=10, activation_function='tanh'):
-    # Construct network
-    model.add(Dense(units=k, activation=activation_function))
-    model.add(Dense(units=k, activation=activation_function))
-    model.add(Dense(units=k, activation=activation_function))
-    model.add(Dense(1))
+def build_nn_model(output_dim=10, num_layers=3, activation_function='tanh'):
+    # Create a Keras Model instance using Sequential()
+    model = Sequential()
+    # Add layers to the network sequentially, from inputs towards outputs
+    for i in range(num_layers):
+        model.add(Dense(units=output_dim, activation=activation_function))
+    # Add a final layer that maps to a scalar value, for regression.
+    model.add(Dense(units=1))
     # Embed training configurations
     model.compile(optimizer=keras.optimizers.SGD(), 
                   loss='mean_squared_error')
     return model
 ```
 
+### Tracking errors
+
++++
+
 The following function will be used to plot the MSE of the model during the
 training process.
 
@@ -160,12 +186,16 @@ as the parameters are adjusted to better fit the data.
 
 ```{code-cell} ipython3
 def plot_loss_history(training_history, ax):
-    ax.plot(training_history.epoch, 
+    # Plot MSE of training data against epoch
+    epochs = training_history.epoch
+    ax.plot(epochs, 
             np.array(training_history.history['loss']), 
             label='training loss')
-    ax.plot(training_history.epoch, 
+    # Plot MSE of validation data against epoch
+    ax.plot(epochs, 
             np.array(training_history.history['val_loss']),
             label='validation loss')
+    # Add labels
     ax.set_xlabel('Epoch')
     ax.set_ylabel('Loss (Mean squared error)')
     ax.legend()
@@ -180,19 +210,16 @@ Now let's go ahead and train our  models.
 
 We'll start with linear regression.
 
-First we create a `Model` instance using `Sequential()`.
-
 ```{code-cell} ipython3
-model = Sequential()
-regression_model = build_regression_model(model)
+regression_model = build_regression_model()
 ```
 
 Now we train the model using the training data.
 
 ```{code-cell} ipython3
 training_history = regression_model.fit(
     x, y, batch_size=x.shape[0], verbose=0,
-    epochs=4000, validation_data=(x_validate, y_validate))
+    epochs=2000, validation_data=(x_validate, y_validate))
 ```
 
 Let's have a look at the evolution of MSE as the model is trained.
@@ -241,14 +268,13 @@ Now let's switch to a neural network with multiple layers.
 We implement the same steps as before.
 
 ```{code-cell} ipython3
-model = Sequential()
-nn_model = build_nn_model(model)
+nn_model = build_nn_model()
 ```
 
 ```{code-cell} ipython3
 training_history = nn_model.fit(
     x, y, batch_size=x.shape[0], verbose=0,
-    epochs=4000, validation_data=(x_validate, y_validate))
+    epochs=2000, validation_data=(x_validate, y_validate))
 ```
 
 ```{code-cell} ipython3
@@ -286,3 +312,9 @@ fig, ax = plt.subplots()
 plot_results(x_validate, y_validate, y_predict, ax)
 plt.show()
 ```
+
+Not surprisingly, the multilayer neural network does a much better job of fitting the data.
+
+```{code-cell} ipython3
+
+```