Foundations
37 concepts, 37 reviewed and merged.
Architecture
- Activation FunctionThe small nonlinear step between one layer and the next.verified
- Long Short-Term MemoryA recurrent cell with a deliberate path for information to travel a long way unchanged.verified
- Recurrent Neural NetworkProcess a sequence one element at a time, carrying a hidden state forward.verified
- Sequence to SequenceRead a whole input sequence, then produce a whole output sequence of a different length.verified
Ensembles
Functions
Generalization
- Bias-Variance TradeoffA model can be wrong because it is too simple to capture the pattern, or because it is so flexible it fits the noise…verified
- Cross-ValidationOne held-out test set gives you one number, and that number has error bars nobody prints.verified
- Inductive BiasThe assumptions a model makes before it sees any data.verified
Instance-Based
Optimization
- Batch NormalizationNormalise each feature across the examples in the batch, so every layer sees inputs with a stable scale however the…verified
- ConvexityA bowl has one bottom.verified
- Vanishing GradientTrain a deep network and the early layers barely move.verified
- Weight InitializationThe numbers a network starts with, before it has seen anything.verified
Representation
- Byte-Pair EncodingByte-pair encoding builds a vocabulary by repetition.verified
- Token EmbeddingA token id is just a number with no meaning — id 400 is not twice id 200.verified
- TokenizationA model does not see text, it sees numbers.verified
- Word EmbeddingA word becomes a vector, and words used in similar contexts end up near each other — not because anyone described…verified
Statistics
- Bayesian InferenceStart with what you believed before, weigh it by how well each possibility explains what you saw, and end with what…verified
- Expectation-MaximizationFitting a model when some of what you need is missing.verified
- Hidden Markov ModelA sequence of states you cannot see, each producing an observation you can.verified
- Maximum LikelihoodPick the parameters that make the data you actually observed as probable as possible.verified
- N-Gram ModelPredict the next word by counting how often it followed the previous few in a large pile of text.verified
Supervised
- Active LearningIf labels are expensive, do not label at random — let the model choose what to ask about.verified
- Data AugmentationMake more training data out of the data you have, by changing it in ways that do not change the answer.verified
- Decision TreeAsk a series of yes-or-no questions about a data point until you reach an answer.verified
- Early StoppingWatch performance on data the model is not training on, and stop when it stops improving.verified
- Ensemble MethodsCombine several models instead of trusting one.verified
- Linear RegressionFit a straight line through points, choosing the one whose distances to them are smallest overall.verified
- Logistic RegressionThe same idea as fitting a line, except the answer has to be a probability.verified
- Semi-Supervised LearningA few labelled examples and an enormous pile of unlabelled ones — which is the situation almost everybody is actually…verified
- Support Vector MachineOf all the lines separating two classes, take the one with the widest empty corridor around it.verified
Training
- Exposure BiasThe consequence of `teacher-forcing`: a model has only ever seen correct prefixes, so when its own output drifts it…verified
- Label SmoothingInstead of training toward a target of exactly one for the correct class and zero for everything else, aim slightly…verified
- Teacher ForcingDuring training, feed the model the correct previous tokens rather than the ones it just produced.verified
- Transfer LearningLearn something on one problem and reuse it on another.verified