MIT 6.7960 Deep Learning — Lec 04. Architectures: Grids

Overview

The lecture reviews the multilayer perceptron, highlighting its theoretical strengths and practical drawbacks such as weak inductive bias and high computational cost. It emphasizes the importance of adding structural assumptions—hypothesis spaces—to improve sample efficiency and guide learning toward the true function. The speaker discusses how both the amount of training data and the architectural inductive bias of a model influence generalization. He compares simple ReLU nets to sinusoid‑based SIREN networks, showing that a strong prior can lead to efficient learning even with limited data. The talk also highlights the trade‑off between generality and specificity in image classification, stressing the need for thoughtful architecture design. This section explains how to tackle complex images by dividing them into overlapping patches, classifying each, and using convolutional kernels for translation-invariant semantic segmentation. It highlights the importance of local context and the transition from image-level to pixel-level classification.

Chapter breakdown

Key points

• 00:01The lecture introduces moving beyond basic multilayer perceptrons to explore richer machine learning architectures.
• 03:28Weak inductive biases make MLPs sample‑inefficient; they require a large amount of data to learn complex functions.
• 07:36More data and a more constrained architecture together improve learning; both are essential for better models.
• 10:18SIREN, a sinusoidal neural network, uses sine activations and approximates image functions more efficiently than ReLU or Tanh nets.
• 15:29A strategy to handle multiple objects is to subdivide the image into patches and classify each patch independently.
• 21:32The weights learned for each patch form a convolutional kernel that can be applied across the entire image.
• 23:34The illustrated filter detects horizontal edges between dark and light regions.
• 29:03A convolutional filter can be seen as a linear layer with only three weights (w1, w0, w-1) that slide along the diagonal of a sparse matrix.
• 32:24Receptive field can be enlarged by larger kernels or by adding more layers (spatial pyramid effect).
• 36:41Multi‑output convolutions use a filter bank: each filter produces its own output channel.
• 41:01Parameter count in a convolution is (#output filters) × (#input channels) × (kernel height × kernel width).
• 44:05Typical CNN filters are square, though non‑square filters are possible.
• 47:27Pooling across feature channels can yield invariance to edge orientation.
• 52:08Over the past decade, the trend has moved from large, shallow filters (e.g., AlexNet) to smaller, deeper filters that achieve large receptive fields.
• 56:16When data is scarce, handcrafted filters or stronger inductive biases can improve performance; with abundant data, end‑to‑end learning often outperforms hand‑crafted designs.
• 01:04:09U‑Net solves this by adding skip connections that carry high-resolution spatial information from encoder to decoder.
• 05:22ResNet introduces skip (identity) connections that allow each layer to either transform input or simply pass it through, giving the network flexibility to learn an optimal depth.
• 01:12:04A neural field is a physical quantity varying over space and time, parameterized by a neural network.
• 01:16:51Pooling can be seen as a non‑learned convolution that reduces spatial resolution using max or average operations.
• 20:07Side information (metadata such as GPS) can be incorporated via positional encoding or separate convolutions.

Key terms

• Multilayer Perceptron — A fully connected neural network comprising alternating linear layers and point‑wise non‑linearities.
• Universal Approximator — A property stating that a neural network can approximate any continuous function on a compact domain given sufficient width or depth.
• Inductive Bias — Prior assumptions encoded in the model architecture that guide learning and help generalize beyond training data.
• Embarrassingly Parallel — An algorithmic structure where computations can be performed independently without inter‑process communication.
• Hypothesis Space — The set of all models that a learning algorithm can consider; larger spaces allow more flexibility but can hurt generalization.
• Generalization — The ability of a model to perform well on unseen data outside the training distribution.
• Sinusoid — A sine function used in activations (as in SIREN) to capture periodic patterns.
• ReLU — Rectified Linear Unit activation; commonly used in deep networks.
• Convolution — A linear, shift-invariant operation that applies a kernel across a grid-structured input.
• Fourier Basis — Set of sine and cosine functions used to represent signals efficiently.
• Fast Fourier Transform — Algorithm to compute Fourier transforms quickly, enabling efficient image compression.
• semantic segmentation — Assigning a class label to each pixel in an image, producing a spatially structured output.
• Instance segmentation — Extending semantic segmentation by identifying individual instances of objects.
• Equivariance — Property where applying a transformation to the input produces a predictable transformation of the output.
• Translation invariance — The ability of a model to produce the same prediction regardless of an object's position in the image.
• Patch — A sub-region of an image used as input to a classifier.
• Weighted sum — Linear combination of pixel values in a patch using learned weights.
• Kernel — The spatial window (e.g., 3×3) of weights used in a convolution operation.
• Filter — A set of weights applied to a local region of the input to detect patterns.
• Toeplitz matrix — A matrix with constant diagonals, representing the sparse structure of convolution weights.
• Bias — A constant added to the weighted sum before activation.
• Fully connected layer — A neural network layer where each output neuron connects to every input neuron.
• Cross-correlation — An operation similar to convolution but without flipping the filter.
• Convolutional Layer — A neural network layer that applies a set of learnable filters (kernels) across an input, performing weighted sums over local patches.
• Patch Processing — Processing local regions (patches) of an input independently, enabling parallel computation.
• Image Filter — A small kernel that highlights specific features (edges, textures) when convolved with an image.
• Parameter Sharing — Using the same set of weights across all spatial locations within a layer.
• receptive field — The spatial extent of the input image that influences a particular neuron in a convolutional layer.
• Spatial Pyramid — Technique of increasing receptive field by using progressively larger filters or deeper layers.
• Multi‑Channel Input — Input data with an additional depth dimension (e.g., RGB color channels).
• Filter Bank — A set of multiple filters in a convolutional layer, each producing a separate output channel.
• Feature Map — The output produced by applying a filter over an input; represents learned features.
• Channel — A dimension representing different feature maps or input data layers (e.g., RGB channels).
• Regularization — Techniques (e.g., dropout) that constrain model complexity to prevent overfitting.
• Dropout — A regularization method that randomly drops units during training to reduce co‑adaptation.
• Feature Collapse — A scenario where multiple output channels become redundant or identical, reducing representational diversity.
• AlexNet — A pioneering deep convolutional neural network architecture known for its layered feature extraction.
• Pooling — A down‑sampling operation that aggregates information over a local region, e.g., max or mean.
• Stride — The number of pixels a convolutional filter moves across the input image; larger strides reduce spatial resolution.
• Dilated Filter — A filter with zeros inserted between weights, increasing the receptive field without increasing parameter count.
• downsampling — Reducing the spatial or temporal resolution of a feature map, often via pooling or strided convolutions.
• Neural Architecture Search — An automated process that explores a space of possible network architectures to identify high‑performing designs.
• encoder-decoder architecture — A neural network design that compresses input into a latent space via an encoder and then reconstructs output via a decoder.
• bottleneck — A narrow intermediate layer in an encoder-decoder that forces compression of information.
• U‑Net — An encoder-decoder architecture with skip connections that preserves high-resolution spatial details for tasks like segmentation.
• skip connection — An additional pathway that forwards the input of a layer unchanged (identity) to its output, allowing the layer to be bypassed.
• latent space — The compressed representation (often a vector) produced by the encoder.
• softmax — An activation function that turns raw logits into probability distributions over classes.
• convolutional neural network — A class of deep neural networks that process data with convolutional layers, widely used for image tasks.
• variational autoencoder — A generative model that learns a probabilistic latent space and decodes samples to reconstruct inputs.
• masked autoencoder — An autoencoder variant that predicts missing input patches, often using attention mechanisms.
• spatial extent — The physical size (width x height) of feature maps within a network layer.
• upsampling — Increasing spatial resolution, often through transposed convolutions or interpolation.
• ResNet — A convolutional neural network architecture that employs residual (skip) connections to mitigate vanishing gradients and enable training of very deep models.
• identity mapping — A function that returns its input unchanged, used in skip connections to preserve information.
• 3D convolution — A convolution operation applied over three dimensions (e.g., height, width, time) to capture spatiotemporal patterns.
• shift invariance — The property of a network where its output does not change when the input is shifted, useful for spatial features but sometimes undesirable temporally.
• Positional encoding — A technique that transforms coordinate information into a higher‑dimensional space, often using sinusoidal functions.
• MLP — Multi‑layer perceptron, a feed‑forward neural network of fully connected layers that preserves positional information.
• self‑attention — Mechanism in transformers that allows each element of a sequence to weigh all others, enabling dynamic feature combination.
• learned weight — Parameters of a neural network that are updated during training through gradient descent.
• temporal dimension — The axis representing time in a data tensor, e.g., frames in a video sequence.
• Neural field — A spatial‑temporal field whose values are represented by a neural network.
• SIREN — A neural network architecture that uses sine activation functions for high‑frequency representation.
• NeRF — Neural Radiance Field, a neural representation that models 3D scenes by predicting color and density for any 3D point.
• Volumetric density — A scalar field indicating matter density at each point in 3‑D space.
• Hyperspectral — Data capturing many narrow spectral bands across the electromagnetic spectrum.
• Upscaling — Increasing the resolution or detail of data, often using learned models.
• 3D modeling — Constructing a three‑dimensional representation of a scene or object.
• Metadata — Supplementary data that accompanies an image or video, such as GPS coordinates or timestamps.
• Fast & Slow Pathway — Dual‑stream architecture where one stream processes low‑resolution, high‑frequency information and the other processes high‑resolution, low‑frequency information.
• Action Recognition — Task of identifying human or object actions within video content.
• Subsampling — Reducing the temporal or spatial resolution of data to lower computational load.

← All MIT OpenCourseWare course notes