Homework #7: MLP Training with NumPy

EE 541: Fall 2026

ImportantAssignment Details

Assigned: 19 October
Due: Sunday, 25 October at 23:59

Gradescope: Homework 7 | Setup | How to Submit

Getting Started Guide: View Guide
Data: External Link

WarningRequirements

Use only numpy for all computation in this assignment, with h5py to load data and matplotlib to plot. Do not use PyTorch, tf.keras, scikit-learn, etc.

Overview

A fully connected MLP for MNIST trained end to end in NumPy: your own activations and derivatives, minibatch SGD with learning-rate decay, and a six-configuration comparison on a held-out validation set.

Getting Started

Download the starter code: hw7-starter.zip

unzip hw7-starter.zip
cd hw7-starter
python generate_datasets.py --seed 541

This creates data/ with placeholder copies of mnist_traindata.hdf5 and mnist_testdata.hdf5 that have the right layout but random content. Download the real files from the Data link into data/ to replace them (they are the Homework 5 files).

q1/ contains mlp_train.py, with a docstring stating how it is run and what it prints or writes, and a test_interfaces.py that runs the script and checks its output files. Run it from inside q1/:

python -m pytest test_interfaces.py

Problem 1: MLP Training with NumPy

In this problem you manually train a multilayer perceptron (MLP) deep neural network using only numpy. The purpose of this problem is to understand backprop and its implementation before we hand the calculation to PyTorch Autograd.

This problem uses the same MNIST image classification problem as Homework 5 and Homework 6. So you should build upon the Python code you wrote for the feedforward (inference) processing.

The file mnist_traindata.hdf5 contains 60,000 images in the key xdata and the corresponding labels in the key ydata. Split into 50,000 training images and 10,000 validation images.

Part A: Network Specification

Create a neural network for classification. It must have 784 neurons in the input layer and 10 neurons in the output layer. You may use any number of intermediate (hidden) layers with any number of neurons. It must be a fully connected MLP (don’t use convolutional or other layers). Construct your MLP with:

  1. Hidden layer activations: Try tanh and ReLU. Write your own functions for these, and their derivatives. Note: derivative of ReLU at 0 can be any number in [0,1].

  2. Cost function: Cross entropy.

  3. Optimizer: Stochastic gradient descent. Try 3 different values for learning rate. Momentum is optional. Remember that you can only use numpy, so momentum complicates the implementation.

  4. Minibatch size: Your choice. Pick a number between 50 and 1000 that divides by 50,000 (i.e. don’t pick 123). Average the gradients for weight and bias updates over each minibatch – e.g. minibatch size of 100 means 50,000/100 = 500 updates per epoch.

  5. Regularizer: Your choice. If you choose to use it be mindful when calculating cost gradient.

  6. Parameter initialization: Your choice. Don’t initialize weights with zeroes. You can use random values like \(w_{ij} \sim U[0,1]\) or \(w_{ij} \sim \mathcal{N}(0,1)\) or use a standard algorithm like He or Xavier normal.

  7. Input preprocessing, batch normalization: None

Part B: Training and Validation

Train the network for 50 epochs using the 50,000 training images. At the end of each epoch complete a forward pass with the validation set and record the accuracy (e.g. 9630 correct out of 10000 means accuracy is 96.3%). DO NOT use the validation set for training. Perform only forward inference with the validation set.

  1. Learning rate decay: Divide the initial learning rate by 2 twice during training. You can do this after any epoch, for example, after epochs 20 and 40. (Do not do this in the middle of an epoch). The final learning rate should be 1/4th of the initial value.

  2. Repeat training and validation for each of the 6 configurations — using 2 different activation functions and 3 initial learning rates. Make 6 plots each with 2 curves – training accuracy (y-axis) and validation accuracy (y-axis) vs. epoch number (x-axis). Mark the epochs where you divided learning rate.

  3. Choose the configuration (activation and initial learning rate) with the best validation accuracy. Train this for all 60,000 images = training + validation. Remember to apply a learning rate decay as before. After all 50 epochs are done test with the data in mnist_testdata.hdf5 from Homework 5. Record the final test accuracy.

Remember: Do not touch the test set from mnist_testdata.hdf5 until the very end after you decide which configuration is best.

You have a lot of freedom in this problem — welcome to deep learning! This problem does not constrain the number of layers or the number of neurons in each layer. Experiment and don’t worry about low accuracy. A reasonable number for final test accuracy is 95%-99%.

Note: Researchers constructed the Fashion-MNIST dataset for classification because MNIST became “too easy” with modern deep learning techniques. Use the in-lecture experiments on Fashion-MNIST to provide guidance for this problem.

Deliverables

See Submission. mlp_train.py is your implementation. q1.pdf contains:

  • Network configuration – how many layers and how many neurons.
  • Batch size
  • 3 different values of initial learning rate
  • How you initialized parameters
  • 6 curves, as described
  • Final test accuracy for the best network
TipSubmission 
README.md
.gitignore
requirements.txt
generate_datasets.py
q1/
├── mlp_train.py
├── test_interfaces.py
└── q1.pdf

Do not commit data/ — the starter’s .gitignore excludes it.