Homework #5: Logistic Regression and MLP Inference

EE 541: Fall 2026

Assignment Details

Assigned: 28 September
Due: Sunday, 4 October at 23:59

Gradescope: Homework 5 | Setup | How to Submit

Getting Started Guide: View Guide
Data: External Link

Requirements

Use only Python standard library modules, numpy, h5py, and matplotlib for this assignment. Do not use PyTorch, tf.keras, scikit-learn, etc.

Overview

A binary logistic “2” detector trained with gradient descent on MNIST, then the forward pass of a pre-trained MLP implemented in NumPy.

Getting Started

Download the starter code: hw5-starter.zip

unzip hw5-starter.zip
cd hw5-starter
python generate_datasets.py --seed 541

This creates data/ with placeholder copies of mnist_traindata.hdf5, mnist_testdata.hdf5, and mnist_network_params.hdf5 that have the right layout but random content. Download the real files from the Data link into data/ to replace them.

Each problem directory contains the script named in the problem, with a docstring stating how it is run and what it prints or writes, and a test_interfaces.py that runs the script and checks its output files. Run it from inside the problem directory:

python -m pytest test_interfaces.py

Problem 1: Logistic Regression

The MNIST dataset of handwritten digits is one of the earliest and most used datasets to benchmark machine learning classifiers. Each datapoint contains 784 input features – the pixel values from a \(28 \times 28\) image – and belongs to one of 10 output classes – represented by the numbers 0-9.

In this problem you will use numpy to classify input images using a logistic-regression. Implement the functions in logistic.py; python logistic.py then trains the detector and writes weights.hdf5, learning_curve.pdf, and accuracy.pdf.

Use the provided MNIST handwritten-digit data to build and train a logistic “2” detector:

\[ y = \begin{cases} 1 & \mathbf{x} \textrm{ is a "2"} \\ 0 & \textrm{else}. \end{cases} \]

A logistic classifier takes learned weight vector \(\mathbf{w} = [w_1, w_2, \ldots w_L]^T\) and the unregularized offset bias \(b \triangleq w_0\) to estimate a probability that an input vector \(\mathbf{x} = [x_1, x_2, \ldots, x_L]^T\) is “2”:

\[ p(\mathbf{x}) = P[Y = 1 | \mathbf{x}, \mathbf{w}] = \frac{1}{1 + \exp\left(-\left(\sum_{k=1}^{L} w_k \cdot x_k + w_0\right)\right)} = \frac{1}{1 + \exp\left(-\left(w^T x + w_0\right)\right)}. \]

Train a logistic classifier to find weights that minimize the binary log-loss (also called the binary cross entropy loss):

\[ l(w) = - \frac{1}{N} \sum_{i=1}^N \left(y_i \log p(x) + \left(1 - y_i\right) \log\left(1 - p(x)\right)\right) \]

where the sum is over the \(N\) samples in the training set.

Train your model until convergence according to some metric you choose. Experiment with variations of \(\ell_1\)- and/or \(\ell_2\)-regularization to stabilize training and improve generalization.

Answer the following:

How did you determine a learning rate? What values did you try? What was your final value?
Describe the method you used to establish model convergence.
What regularizers did you try? Specifically, how did each impact your model or improve its performance?
Plot log-loss (i.e., learning curve) of the training set and test set on the same figure. On a separate figure plot the accuracy against iteration number of your model on the training set and test set. Plot each as a function of the iteration number.
Classify each input to the binary output “digit is a 2” using a 0.5 threshold. Compute the final loss and final accuracy for both your training set and test set.

Part B: Save the Weights

Save your weights and bias to weights.hdf5. Use keys w and b for the weights and bias, respectively. w should be a length-784 numpy vector/array and b should be a numpy scalar. Use the following as guidance:

with h5py.File('weights.hdf5', 'w') as hf:
    hf.create_dataset('w', data = np.asarray(weights))
    hf.create_dataset('b', data = np.asarray(bias))

Grading

You will not be scored on your model’s overall accuracy. But a low score may indicate errors in training or poor optimization.

Deliverables

See Submission. logistic.py is your implementation and weights.hdf5 holds the trained weights. q1.pdf contains your answers to items 1–5, including the learning curves, accuracy plots, and final loss and accuracy values.

Problem 2: MLP Inference with NumPy

In this problem you will use numpy to program the forward (inference) pass of a multilayer perceptron (MLP). We provide a pre-trained MLP for MNIST classification. The network has a 784-neuron input layer, two hidden layers of 200 and 100 neurons, and a 10-neuron output layer. The hidden layers use ReLU activation and the output layer uses softmax. Implement the functions in mlp_inference.py; python mlp_inference.py then runs the network on the test set and writes predictions.hdf5 and example figures.

Download mnist_network_params.hdf5 and mnist_testdata.hdf5 from the course data server.

Part A: Load the Parameters and Data

Extract the weights and biases from mnist_network_params.hdf5. The file has 6 keys: W1, b1, W2, b2, W3, b3. Verify the dimensions of each numpy array with the shape property. Note the orientation: W1 is \(200 \times 784\), so the first layer computes \(\mathbf{s}^{(1)} = \mathbf{W}^{(1)} \mathbf{x} + \mathbf{b}^{(1)}\).
Extract the test images and labels from mnist_testdata.hdf5. The file has 2 keys: xdata holds 10,000 image vectors, each of length 784 with pixel intensities in \([0, 1]\), and ydata holds the corresponding one-hot labels, e.g., [0,0,0,1,0,0,0,0,0,0] means the image is a “3”.

Part B: Forward Pass

Write functions for ReLU and softmax: \[ \textrm{ReLU}(x) = \max(0, x) \] \[ \textrm{Softmax}(\mathbf{x}) = \left[ \frac{e^{x_1}}{\sum_{i=1}^{n} e^{x_i}}, \frac{e^{x_2}}{\sum_{i=1}^{n} e^{x_i}}, \ldots, \frac{e^{x_n}}{\sum_{i=1}^{n} e^{x_i}} \right] \] Softmax takes a vector of length \(n\) and returns a vector of length \(n\) that you can interpret as a probability distribution. For example, \(\textrm{Softmax}([0, 1, 2]) = [0.09, 0.24, 0.67]\) so the third element is the most likely outcome. Subtract \(\max_i x_i\) from every element before exponentiating – softmax is invariant to this shift and it prevents overflow.
Implement the forward pass. Follow the notation used in the slides: for \(l = 1, 2, 3\) compute the linear activation \(\mathbf{s}^{(l)} = \mathbf{W}^{(l)} \mathbf{a}^{(l-1)} + \mathbf{b}^{(l)}\) and the output \(\mathbf{a}^{(l)} = \underline{h}(\mathbf{s}^{(l)})\), with \(\mathbf{a}^{(0)} = \mathbf{x}\), ReLU for \(l = 1, 2\), and softmax for \(l = 3\). The output \(\mathbf{a}^{(3)}\) is the vector of 10 class probabilities. Classify each image as \(\hat{y} = \arg \max_l a^{(3)}_l\).
Compare your prediction to the true label. Count the classification as correct if the position of the maximum element in \(\mathbf{a}^{(3)}\) matches the position of the 1 in ydata. Tally the number of correctly classified images from the whole set of 10,000 [hint: 9790 correct].

Part C: Inspect the Errors

Identify several images your MLP classified correctly and several it classified incorrectly. Inspect them visually. Is the correct class obvious to you in the incorrect cases? Use matplotlib to visualize:

import matplotlib.pyplot as plt
plt.imshow(xdata[i].reshape(28, 28), cmap='gray')
plt.show()
# the index i selects which image to visualize
# xdata[i] is a length-784 numpy array

Part D: Save the Predictions

Save your output activations and predictions to predictions.hdf5. Use keys activations and yhat for the softmax outputs and predicted classes, respectively. activations should be a \(10000 \times 10\) numpy array and yhat should be a length-10000 integer numpy array. The code to save the file is the same as Problem 1 – substituting the key names.

Deliverables

See Submission. mlp_inference.py is your implementation and predictions.hdf5 holds the output activations and predictions. q2.pdf contains the number of correctly classified images and several correctly and incorrectly classified images with brief commentary.

Submission

README.md
.gitignore
requirements.txt
generate_datasets.py
q1/
├── logistic.py
├── test_interfaces.py
├── weights.hdf5
└── q1.pdf
q2/
├── mlp_inference.py
├── test_interfaces.py
├── predictions.hdf5
└── q2.pdf

Do not commit data/ — the starter’s .gitignore excludes it.