Lec 02. How to Train a Neural Net

Overview

This lecture walks through the end‑to‑end process of training neural networks, beginning with loss definition and stochastic gradient descent, then delving into computational graphs, forward and backward passes, and back‑propagation mechanics. It covers linear layer mathematics, handling of DAG structures, parameter sharing, and the modern differential programming paradigm. The session concludes with practical guidance on integrating custom operations in PyTorch, highlighting when and how to manage non‑differentiable steps.

Chapter breakdown

Key points

• 00:14Introduction to lecture on how to train a neural network.
• 04:07Explanation of gradient descent: move parameters opposite the gradient direction using a learning rate η.
• 07:01Using the full dataset as a batch yields standard gradient descent, but may be memory‑intensive or computationally intractable.
• 10:04Adam is a widely‑used optimizer that combines momentum with adaptive learning‑rate scaling.
• 15:05Vanishing gradients produce very slow progress; stochastic noise can dominate the learning signal, making optimization difficult.
• 19:25Gradient clipping caps large gradients to a preset maximum, preventing overshoot and oscillations during training.
• 21:46Evolution strategies sample random perturbations around parameters and move in directions that improve the loss, helping with issues like vanishing gradients and acting like regularization.
• 27:24A computational graph consists of nodes representing functional transformations and directed edges representing data flow.
• 32:00Matrix calculus basics: derivatives of scalar‑by‑vector, vector‑by‑vector (Jacobian), and scalar‑by‑matrix functions are introduced.
• 36:46Concrete example: Z is scalar, U has length 2, X has length 4, leading to a product of a 1×2 vector and a 2×4 matrix.
• 42:21Forward pass through a hidden layer computes the layer output from its input using the layer's function.
• 45:12Backward propagation to the input is the output‑gradient multiplied by the transposed weight matrix.
• 50:12The backward pass for a linear layer uses the transposed weight matrix.
• 55:27Arbitrary DAGs require only merging and branching operations to correctly propagate gradients.
• 58:20Deep networks are popular because they are easy to optimize (fully differentiable) and compositional (block‑based).
• 02:58Bottleneck constraints limit model performance; end‑to‑end larger models often outperform hand‑engineered pipelines.
• 05:01Feature‑visualization generates synthetic images that maximize a specific neuron’s activation, revealing what the model has learned.
• 09:48An encoder neural network maps high‑dimensional inputs (e.g., images) to low‑dimensional embeddings.
• 12:31An attention block is more than cosine similarity; it also includes projections into and out of a shared space.
• 16:50When an operation lacks a simple gradient, PyTorch supplies approximations, but highly complex cases may need a custom torch operation.

Key terms

• Gradient descent — An iterative first‑order optimization algorithm that updates parameters by moving opposite to the gradient of the loss function.
• Stochastic Gradient Descent (SGD) — An optimization method that updates model parameters using the gradient computed on a randomly selected mini‑batch rather than the whole dataset.
• loss function — A mathematical expression that quantifies the difference between predicted and true values; the quantity minimized during training.
• Learning rate (η) — A scalar hyper‑parameter that controls the step size taken along the gradient direction during optimization.
• Parameter (θ) — Trainable values (weights and biases) inside a neural network that are adjusted to minimize the loss.
• Hessian — The matrix of second‑order partial derivatives of a function; used in second‑order optimization methods.
• Computational graph — A directed acyclic graph representing the sequence of operations used to compute a function, enabling automatic differentiation.
• Backpropagation — An algorithm that computes gradients of a loss function with respect to all parameters in a neural network by recursively applying the chain rule backward through the computation graph.
• Non‑convex — A property of a loss landscape where multiple local minima may exist, making global optimization difficult.
• Batch — A collection of training examples processed together to compute a single gradient estimate.
• Batch size — The number of training examples used to compute a single gradient update.
• Momentum — A technique that incorporates a fraction of the previous update into the current one, smoothing the optimization trajectory.
• Adam optimizer — An adaptive learning‑rate method that combines momentum and per‑parameter scaling of gradients.
• Hyper‑parameter — A configuration setting (e.g., learning rate, activation type) that is set before training and influences model performance.
• Variance (in SGD updates) — The degree of fluctuation in gradient estimates across different mini‑batches.
• Implicit regularizer — A property of an algorithm (here, SGD noise) that discourages overfitting without an explicit penalty term.
• Local minima — Points in the loss landscape that are lower than surrounding points but not the absolute lowest (global minimum).
• Exploding gradients — When gradient magnitudes become excessively large, causing unstable parameter updates.
• Autograd — PyTorch’s automatic differentiation engine that computes gradients for tensor operations.
• Convexity — A property of a function where any line segment connecting two points on the graph lies above the graph, guaranteeing a single global minimum.
• Gradient — A vector of partial derivatives indicating the direction of steepest increase of a function; its negative points toward steepest descent.
• Vanishing gradient — A situation where gradient magnitudes become extremely small, slowing learning and allowing noise to dominate the update signal.
• Exploding gradient — A condition where gradients grow without bound, leading to unstable parameter updates and possible divergence.
• Local minimum — A point in parameter space where the loss is lower than in its immediate neighborhood but not necessarily the lowest possible (global) loss.
• Loss landscape — The high‑dimensional surface defined by the loss value as a function of model parameters.
• Evolution strategy — An optimization technique that iteratively samples random perturbations of parameters and moves toward those that improve the objective.
• Gradient clipping — A technique that limits the magnitude of gradients to a predefined threshold to improve training stability.
• Random seed — An initial value for a pseudorandom number generator that determines reproducible random sequences, influencing model initialization and training outcomes.
• ReLU — Rectified Linear Unit, an activation function defined as f(x)=max(0,x).
• GELU — Gaussian Error Linear Unit, an activation function defined as f(x)=x·Φ(x) where Φ is the Gaussian cumulative distribution function.
• Gaussian error linear unit — Full name of GELU, highlighting its use of the Gaussian CDF.
• Evolution strategies — Optimization algorithms that sample random perturbations of parameters and move toward directions that improve the objective, rather than following the gradient.
• Smoothness — A property of a function where it has continuous derivatives of all orders.
• Continuity — A function is continuous if small changes in input produce small changes in output.
• differentiable — A property of a function that has a well‑defined derivative everywhere in its domain.
• Monotonic activation function — An activation function that never decreases (or never increases) as its input grows.
• Regularization — Techniques that add constraints or penalties to the loss to prevent overfitting and improve generalization.
• directed acyclic graph (DAG) — A graph with directed edges and no cycles, commonly used to represent computation graphs in neural networks.
• multi‑layer perceptron (MLP) — A feed‑forward neural network composed of one or more hidden layers of linear transformations followed by non‑linear activations.
• Forward pass — The phase in neural‑network computation where inputs are transformed layer‑by‑layer to produce the network output.
• Jacobian — A matrix of all first‑order partial derivatives of a vector‑valued function with respect to a vector of inputs.
• Chain rule — A calculus rule that allows differentiation of composite functions by multiplying the derivative of the outer function evaluated at the inner function by the derivative of the inner function.
• Partial derivative — Derivative of a multivariable function with respect to one variable while holding the others constant.
• Jacobian matrix — A matrix of all first‑order partial derivatives of a vector‑valued function.
• Backward pass — The phase where gradients of the loss with respect to parameters are propagated from the output back toward the input.
• Linear layer — A neural‑network layer that performs a linear transformation (matrix multiplication) of its input.
• Activation function — A non‑linear function applied element‑wise after a linear transformation; e.g., ReLU.
• transpose — An operation that flips a matrix over its diagonal, swapping rows with columns.
• gating matrix — A diagonal matrix that masks or passes through selected components of a vector, often built from ReLU activations.
• diagonal matrix — A square matrix where all off‑diagonal entries are zero; only the main diagonal may contain non‑zero values.
• bias term — An additional parameter added to a linear transformation to allow shifting the output.
• merge operation — An operation that combines multiple tensor streams (e.g., addition or concatenation) into a single stream.
• branch operation — An operation that splits a tensor into multiple downstream paths, often duplicating data.
• Parameter sharing — Reusing the same set of weights across multiple parts of a neural network, requiring gradient accumulation during backpropagation.
• Differential programming — A paradigm where models are written as differentiable programs, allowing automatic gradient‑based optimization of any component.
• DAG — Directed Acyclic Graph, a computation structure without cycles, used to describe complex model architectures.
• Software 2.0 — Concept that modern software is defined by an optimizable, differentiable objective rather than fixed code, blending learned and hand‑coded modules.
• Neural Module Networks — Architectural framework where a network is assembled from reusable modules, some of which may be learned and others fixed.
• bottleneck — A restrictive design constraint that limits the flow of information or performance of a system.
• end‑to‑end learning — Training a model where raw inputs are mapped directly to outputs, allowing the entire system to be optimized jointly.
• softmax — A function that converts a vector of raw scores (logits) into probabilities that sum to one.
• logits — The raw, unnormalized scores output by a neural network before applying a softmax.
• feature visualization — Technique that synthesizes inputs which maximally activate a chosen neuron or layer to interpret model behavior.
• DeepDream — An algorithm that iteratively modifies an image to amplify patterns recognized by a deep network, creating hallucinogenic visuals.
• CLIP — Contrastive Language‑Image Pre‑training; a model that aligns text and image embeddings in a shared space.
• GAN — Generative Adversarial Network, a framework with a generator and discriminator trained in opposition to produce realistic data.
• embedding — A dense vector representation of categorical data learned by the model, often used as input to downstream layers.
• Encoder — A neural network that transforms high‑dimensional data into a compact latent representation.
• Loss — A scalar measuring error for a single training example.
• Cost — The aggregated loss over a batch or the entire dataset.
• Gradient freezing — Preventing gradients from being computed for selected parameters during back‑propagation.
• Modularity — Design principle where a model consists of interchangeable components, some hand‑crafted, some learned.
• Attention block — A transformer sub‑module that computes similarity scores (often via cosine similarity) and mixes information via learned projections.
• Cosine similarity — A measure of similarity between two vectors based on the cosine of the angle between them.
• Dynamic computational graph — A graph that is built on‑the‑fly during the forward pass, as in PyTorch.
• UDF (user‑defined function) — A custom function written by the user that can be inserted into a data pipeline or model graph.
• Clip operation — A function that limits values of a tensor to a specified range, often used to bound gradients.
• torch operation — A function implemented in PyTorch that supports automatic differentiation via a defined backward (gradient) computation.
• occupancy model — A statistical model that estimates the probability that a species occupies a site, often involving complex likelihoods.
• data loader — PyTorch utility that iterates over a dataset, optionally applying preprocessing or augmentation before feeding data to the model.
• preprocessing — Operations applied to raw data (e.g., scaling, cleaning) before it enters the learning model; need not be differentiable.
• intractable — Describes a computation that is practically impossible to solve due to excessive time or memory requirements.
• branching operation — A network operation that splits a tensor into multiple paths, each of which may be processed separately.
• augmentation — Techniques that artificially increase the diversity of training data (e.g., rotations, flips) without changing labels.

← All MIT OpenCourseWare course notes