Lec 01. Introduction to Deep Learning

Overview

The instructor introduces the rapid growth of deep learning, defines it as neural networks plus differential programming, and outlines the course philosophy of theory‑practice balance. Coursework consists of problem sets and a final blog‑style research project with limited compute, and a detailed schedule of upcoming topics is previewed. The instructor outlines the upcoming deep‑learning topics—including generative modeling, OOD generalization, LLMs, scaling laws, and PyTorch tutorials—while emphasizing academic integrity policies for problem sets, collaborations, and AI assistant usage. The segment walks through the rise, fall, and resurgence of neural‑network research, highlighting key milestones such as the perceptron, backpropagation, CNNs, the AI winter, and the transformative impact of AlexNet and large datasets.

Chapter breakdown

Key points

• 00:12Instructor asks why everyone is present, framing the lecture.
• 02:12Breakthroughs arise from a mix of theory and practice; the course will balance both.
• 05:44High‑level schedule preview: training basics, approximation theory, various architectures, scaling rules, generalization, transformers, and representation learning.
• 07:10Outline upcoming material on scaling laws and automatic gradient descent, plus a possible lecture on the past and future of deep learning.
• 10:04Apply the same collaboration policy to AI assistants—use them as discussion partners, not as solution providers, and disclose their use.
• 15:01Minsky and Papert’s 1972 book *Perceptrons* critically exposed the model’s limits, causing a dip in AI enthusiasm.
• 20:02The instructor asks where the field will be in 2028, prompting students to consider the next hype cycle.
• 22:46Environmental concerns: carbon cost of training large models versus lightweight models like Stable Diffusion.
• 25:15Preview of deeper topics: differential programming and the mechanics of backpropagation.
• 28:04A neural net requires a nonlinear function (activation) applied to the linear output to be more than a simple linear model.
• 31:00Training aims to minimize a loss function that penalizes misclassifications; gradient descent updates the weights and bias toward a better fit.
• 34:01Introduces the Rectified Linear Unit (ReLU) as max(0, Z), an unbounded positive activation that is computationally cheap.
• 38:38Mentions that sinusoidal activations can be useful when the data have inherent Fourier‑type structure.
• 40:48With at least one non‑linear activation between layers, a network can in theory represent any function.
• 45:12Capacity refers to the total number of trainable parameters, determined by both width and depth of the model.
• 47:31The over‑parameterized paradigm replaces the under/over‑fitting language with under‑/over‑parameterized regimes.
• 50:28Explains softmax‑cross‑entropy loss as the standard objective for multi‑class classification.
• 53:31Training consists of predicting, calculating loss, and updating weights repeatedly over many data samples.
• 56:26Early CNN layers learn low‑level features; later layers combine them into higher‑level concepts.
• 58:49Scaling is introduced by comparing neuron counts across species—from C. elegans (302 neurons) to humans (≈100 billion) and elephants (≈250 billion).

Key terms

• Neural network — A machine‑learning model composed of layers that apply linear transformations followed by pointwise nonlinearities.
• differential programming — Programming paradigm where programs are written to be differentiable, allowing gradient‑based optimization of arbitrary code.
• Gradient‑based optimization — Algorithms (e.g., SGD, Adam) that adjust parameters by following the gradient of a loss function.
• Problem set — Weekly or bi‑weekly assignments combining pen‑and‑paper/Overleaf work with coding tasks.
• Final project — A research‑style assignment culminating in a blog post that showcases original experimentation and visualization.
• Transformer — A deep‑learning architecture based on self‑attention mechanisms, widely used for sequence modeling.
• Generalization theory — The study of how well a model trained on finite data performs on unseen data.
• Scaling rules — Empirical or theoretical guidelines describing how model performance changes with data, compute, or model size.
• representation learning — A set of techniques that enable a model to automatically discover useful features or representations from raw data.
• generative models — Models that learn the joint probability distribution of data and can generate new samples resembling the training data.
• conditional models — Generative or predictive models that produce outputs conditioned on some input variable or context.
• out-of-distribution generalization — The ability of a model to perform well on data that comes from a different distribution than the training set.
• transfer learning — A machine‑learning technique where a model trained on one task is adapted to perform another, leveraging previously learned representations.
• large language models — Neural networks with billions of parameters trained on massive text corpora to perform a wide range of language tasks.
• scaling laws — Empirical relationships that describe how model performance scales with compute, data, and parameter count.
• automatic gradient descent — Optimizers that adapt learning rates and other hyper‑parameters automatically during training, e.g., Adam, RMSprop.
• PyTorch — An open‑source deep learning framework that provides tensor computation and automatic differentiation with a Pythonic interface.
• honor code — A set of academic integrity rules requiring students to produce their own work and properly credit collaborators, including AI tools.
• Perceptron — A single‑layer artificial neuron that computes a weighted sum of inputs, applies a non‑linear activation, and outputs a binary decision.
• Backpropagation — An algorithm that computes gradients of a loss function with respect to network weights by propagating errors backward through the layers.
• Convolutional Neural Network (CNN) — A deep learning architecture that uses convolutional layers to automatically learn spatial hierarchies of features from grid‑like data such as images.
• AI Winter — A period of reduced funding and interest in artificial intelligence research, often triggered by unmet expectations.
• GPU — Graphics Processing Unit; a hardware accelerator originally for rendering graphics that can perform massive parallel numeric operations, ideal for training deep networks.
• ImageNet — A large, publicly‑available, labeled image dataset that became the benchmark for visual recognition tasks.
• Self‑supervised learning — A training paradigm where the model generates its own supervisory signal from raw data, reducing reliance on manually labeled datasets.
• autograd — A set of software tools that automatically compute gradients of arbitrary functions, enabling efficient backpropagation.
• chain rule — A calculus principle used to compute the derivative of a composite function; essential for backpropagation in neural networks.
• TensorFlow — A widely used open‑source platform for building and deploying machine‑learning models, offering both static and eager execution modes.
• LAION — A large, openly available dataset of image‑text pairs used for training vision‑language models.
• gradient descent — An optimization algorithm that iteratively updates model parameters in the direction opposite to the gradient of the loss function.
• modular reuse — The practice of reusing pretrained model weights or components as building blocks in new systems.
• API — Application Programming Interface; a set of rules that allows software components to communicate, often used to expose model functionality without releasing weights.
• carbon footprint — The total greenhouse‑gas emissions associated with an activity, here referring to the energy consumption of training large models.
• stable diffusion — A relatively lightweight diffusion‑based generative model that produces high‑quality images with far lower computational cost than massive transformer models.
• Linear layer — A computation that forms a weighted sum of inputs plus a bias term.
• Bias term — An additive constant vector (b) applied after the linear transformation to shift the activation.
• θ (theta) — The collection of all learnable parameters in a model, i.e., all weights and biases.
• Pointwise nonlinearity — An activation function applied independently to each component of a vector.
• ReLU (Rectified Linear Unit) — An activation function defined as f(z)=max(0, z); it outputs zero for negative inputs and a linear identity for positive inputs.
• Linearly separable — A dataset where a single hyperplane can separate classes without error.
• Loss function — A metric that quantifies the difference between the model's prediction and the true target.
• tanh — Hyperbolic tangent activation, ranging from –1 to 1, defined as (e^{Z}–e^{‑Z})/(e^{Z}+e^{‑Z}).
• Sigmoid — Logistic activation σ(Z)=1/(1+e^{‑Z}) that maps real numbers to the interval (0,1).
• Gradient saturation — A condition where the derivative of an activation (e.g., sigmoid) becomes near zero for very large or very small inputs, hindering learning.
• Dead neuron — A ReLU unit that permanently outputs zero because its input stays negative, receiving no gradient updates.
• Hidden layer — A layer of units whose outputs are not directly observed as input or final output but feed into subsequent layers.
• Matrix multiplication (in NN) — The operation W·x that combines weights (matrix W) with the input vector x to produce a pre‑activation vector.
• Non‑linearity — An activation function that introduces a non‑linear transformation between linear layers, enabling the network to model complex relationships.
• Convergence speed — The rate at which a training algorithm reduces loss; ReLU was reported to give ~6× faster convergence than tanh in the Krizhevsky paper.
• Sinusoidal activation — An activation function based on sine or cosine, useful for data with periodic or Fourier‑like characteristics.
• Deep net — A neural network with multiple hidden layers, allowing hierarchical representation learning.
• Threshold — A decision rule that classifies inputs based on whether a computed value exceeds a certain constant.
• Representational power — The ability of a model class (e.g., neural network depth/width) to express a wide variety of functions.
• Approximation theory — The study of how well functions can be approximated by simpler, parameterized families such as neural networks.
• Depth vs. Width — A trade‑off where deeper, narrower networks can achieve the same expressive power as very wide shallow networks with fewer parameters.
• Double descent — A risk curve where error first rises after the classical over‑fitting point, then falls again as model size continues to increase.
• Capacity — The total number of learnable parameters in a network, influencing its ability to fit data.
• Interpolation — A regime where a model fits the training data exactly (zero training error) while still maintaining good test performance.
• overfitting — When a model fits the training data too closely, capturing noise and failing to generalize to new data.
• over‑parameterized — A regime where a model has more parameters than strictly needed for the training set, yet can still generalize.
• under‑parameterized — A model with insufficient capacity to capture the underlying structure of the data.
• generalization — The ability of a model to perform well on unseen data outside the training distribution.
• interpretability — How easily a human can understand the internal workings or decisions of a model.
• simplicity hypothesis — The theory that deep neural networks, despite being large, tend to converge to simple functions that generalize.
• cross‑entropy loss — A loss function measuring the difference between the predicted probability distribution and the true distribution.
• softmax — A function that converts raw logits into a normalized probability distribution over classes.
• one‑hot encoding — A binary vector representation where only the index of the true class is 1 and all others are 0.
• Score (probability score) — A raw output value from a model that is later normalised; not an actual probability.
• Gradient — The vector of partial derivatives indicating the direction of steepest increase of the loss; used to update weights.
• Batch — A collection of multiple training examples processed together in one forward and backward pass.
• Tensor — A multi‑dimensional array (matrix, vector, scalar) that represents data, parameters, or activations.
• scaling — Increasing the size or capacity of a model (e.g., number of parameters, data) and studying how performance changes.
• neuron — A computational unit in a neural network that processes inputs and produces an output; analogous to biological neurons.
• optimization — The process of adjusting model parameters to minimize a loss function during training.
• hype cycle — A model describing the phases of public expectation for emerging technologies: innovation trigger, peak of inflated expectations, trough of disillusionment, slope of enlightenment, and plateau of productivity.
• deep learning — A subfield of machine learning that uses multi‑layered neural networks to learn hierarchical representations from data.
• logistics — Practical details and administrative tasks required to run a course or project.

← All MIT OpenCourseWare course notes