Understanding Deep Learning

Model Capacitydesign lever

The total expressive power of a neural network, determined by the number of parameters, hidden layers, and hidden units per layer. Higher capacity means the model can represent a larger family of input-output functions and more linear regions in the piecewise linear mapping.

Architectural Inductive Biasdesign lever

The implicit preference a neural network architecture encodes for certain families of input-output mappings, arising from structural choices such as weight sharing in convolutional layers, equivariance to permutations in GNNs, or masked self-attention in transformers. Determines which solutions are reachable and which are implicitly penalized.

Loss Function Designdesign lever

The choice of mathematical objective used during training, ideally derived as the negative log-likelihood of a probability distribution appropriate to the output domain. Determines what signal guides parameter updates and what the model is implicitly optimizing.

Parameter Initialization Qualitydesign lever

The degree to which the initial parameter values prevent vanishing or exploding gradients in both the forward pass (activations) and backward pass (gradients) during training. He initialization and related methods set weight variance as a function of layer width to maintain stable signal magnitudes throughout the network.

Optimization Algorithm Choicedesign lever

The specific iterative method used to minimize the loss function, including the choice between full-batch gradient descent, SGD, SGD with momentum, and Adam, as well as associated hyperparameters such as learning rate, batch size, and momentum coefficients.

Explicit Regularizationdesign lever

Deliberate additions to the loss function or training procedure designed to favor certain parameter configurations, including L2 weight decay, L1 regularization, dropout, label smoothing, data augmentation, and early stopping.

Training Data Quantitycontextual condition

The number of labeled input-output pairs available for supervised training. Larger datasets reduce the variance component of test error by ensuring better coverage of the input space and averaging out noise in parameter estimates.

Data Quality and Noise Levelcontextual condition

The degree to which training labels are accurate and inputs are clean. Includes label noise (mislabeled examples), measurement noise in inputs, and distributional mismatch between training data and the true data-generating process.

Gradient Stabilitypsychological state

The degree to which gradient magnitudes remain well-conditioned throughout the network during backpropagation—neither vanishing to zero (preventing parameter updates in early layers) nor exploding to very large values (causing unstable updates). Determined by initialization, activation functions, normalization layers, and architectural choices.

Loss Surface Geometrypsychological state

The shape of the loss function as a function of model parameters, including the presence and depth of local minima, saddle points, curvature (sharpness vs. flatness of minima), and the degree to which the surface is smooth and predictable relative to the optimization step size.

Implicit Regularization Effectpsychological state

The tendency of gradient descent and stochastic gradient descent with finite step sizes to prefer certain solutions over others even without explicit regularization terms in the loss. SGD implicitly penalizes solutions where batch gradients disagree, and gradient descent implicitly penalizes steep regions of the loss surface.

Model Biaspsychological state

The systematic deviation of the model from the true underlying function, arising when the model family is insufficiently flexible to represent the true input-output mapping even with optimal parameters and infinite training data.

Model Variancepsychological state

The variability in fitted model parameters and predictions that arises from the particular finite noisy training dataset used. High variance means different training sets produce substantially different models; this is the primary driver of overfitting in the classical regime.

Generalization Performanceoutcome metric

The degree to which a trained model makes accurate predictions on data not seen during training, quantified by test set loss or error rate. The primary outcome of the supervised learning pipeline; summarizes the combined effect of bias, variance, and irreducible noise.

Training Convergencebehavioral pattern

The ability of the optimization algorithm to reliably find a low-loss solution in a reasonable number of iterations, without divergence (exploding gradients), stagnation (vanishing gradients or saddle points), or oscillation. A prerequisite for achieving good generalization performance.

Overparameterization Regimecontextual condition

The condition in which the number of model parameters substantially exceeds the number of training examples, allowing the model to interpolate the training data exactly. In this regime, generalization depends on the model's inductive bias and implicit regularization rather than classical capacity control.

Generative Sample Quality and Coverageoutcome metric

For generative models, the joint property that synthesized samples are individually realistic (quality/precision) and that the full diversity of the training distribution is represented in the generated set (coverage/recall). These two properties can trade off against each other.

Ethical Deployment Riskoutcome metric

The degree to which a deployed AI system poses risks of harm including perpetuation of historical biases, lack of explainability, weaponization, concentration of economic or political power, and potential for large-scale societal harm or existential risk.

Transfer Learning Leveragedesign lever

The degree to which pre-training on a related or self-supervised task provides useful initial parameter values or representations that improve final task performance, especially when labeled data for the primary task is scarce.

Normalization Schemedesign lever

The use of batch normalization, layer normalization, group normalization, or related techniques that re-center and rescale activations during training. These stabilize forward propagation, smooth the loss surface, allow higher learning rates, and add a regularizing noise source.

Residual Connectionsdesign lever

Skip connections that add the input of each layer directly to its output, allowing layers to learn additive corrections rather than full transformations. This creates multiple paths of different lengths through the network, reduces shattered gradients, smooths the loss surface, and enables training of very deep networks.