Explaining and Harnessing Adversarial Examples
The 2014 paper behind the famous panda example: it explained adversarial examples as a result of neural networks' linearity and introduced FGSM.
Landmark Papers
What the papers actually said - linked to the originals.
691 entries, all primary-sourced
Adam: A Method for Stochastic Optimization
The 2014 paper introducing Adam, the adaptive optimizer that became the default for training deep neural networks.
Image Super-Resolution Using Deep Convolutional Networks (SRCNN)
SRCNN, the 2014 paper that first applied a deep convolutional network end-to-end to single-image super-resolution.
Show, Attend and Tell: Neural Image Caption Generation with Visual Attention
Show, Attend and Tell added visual attention to captioning, letting the model fix its gaze on the object it was naming word by word.
Batch Normalization: Accelerating Deep Network Training
The 2015 paper introducing batch normalization, which let much deeper networks train far faster and more reliably.
Trust Region Policy Optimization (TRPO)
The 2015 TRPO paper made policy-gradient training stable with a trust region, paving the way for PPO and modern deep RL.
Distilling the Knowledge in a Neural Network
The 2015 Hinton-Vinyals-Dean paper that formalized knowledge distillation, training a small student model to mimic a large model's soft outputs.
Rotation-invariant CNNs for galaxy morphology prediction
A 2015 paper showed a rotation-invariant CNN could classify galaxy shapes from Galaxy Zoo crowdsourced labels at over 99% agreement.
Fast R-CNN
The 2015 paper that sped up region-based object detection by running the convolutional network once per image and sharing features across regions.
Highway Networks
The 2015 Schmidhuber-lab paper that used learned gates to let gradients flow through very deep networks, a direct precursor to ResNet's skip connections.
VQA: Visual Question Answering
The 2015 VQA paper defined free-form visual question answering and shipped a dataset of 0.25M images with 0.76M questions.
U-Net: Convolutional Networks for Biomedical Image Segmentation
The 2015 U-Net paper introduced an encoder-decoder with skip connections for image segmentation, now a backbone of diffusion image generators.
Faster R-CNN: Towards Real-Time Object Detection
The 2015 paper that built region proposals into the network itself, making accurate object detection nearly real-time and fully end-to-end.
Generalized Advantage Estimation (GAE)
The 2015 GAE paper introduced a variance-reduction technique for policy gradients that underpins methods like PPO.
You Only Look Once (YOLO)
The 2015 YOLO paper reframed object detection as a single regression pass, hitting 45 frames per second and making real-time detection mainstream.
FaceNet: A Unified Embedding for Face Recognition and Clustering
Google's 2015 FaceNet learned a face embedding with a triplet loss and reached 99.63% on the LFW benchmark.
BookCorpus and Aligning Books and Movies
The 2015 paper that introduced BookCorpus, a collection of free e-books that later quietly trained BERT, GPT, and many early language models.
Why Are There Still So Many Jobs? (Autor)
David Autor's argument that automation both substitutes for and complements labor, which is why employment persists despite automation.
Listen, Attend and Spell
A 2015 Google paper that transcribed speech to characters with an attention-based encoder-decoder, no separate phoneme or HMM stage.
Effective Approaches to Attention-based NMT (Luong Attention)
Introduces global and local attention for translation, with simpler scoring that became widely adopted.
Byte Pair Encoding for Neural Machine Translation
Adapts a compression algorithm to split rare words into subword units, solving open-vocabulary translation.
chrF: Character n-gram F-score for MT Evaluation
chrF is a 2015 translation metric that compares text at the character level, working better than BLEU for morphologically rich languages.
Continuous control with deep reinforcement learning (DDPG)
DeepMind's 2015 DDPG paper extended deep Q-learning to continuous action spaces, enabling RL control of simulated robots from pixels.
Double DQN
The 2015 Double DQN paper showed standard deep Q-learning overestimates values and fixed it with a simple decoupling trick.
Model Inversion Attacks that Exploit Confidence Information
A 2015 paper showing that an ML model's confidence scores can be used to reconstruct sensitive training inputs, even faces.
Prioritized Experience Replay
The 2015 prioritized replay paper showed RL agents learn faster by replaying important past experiences more often.
Unsupervised Representation Learning with Deep Convolutional GANs (DCGAN)
DCGAN gave GANs a stable convolutional architecture, making adversarial image generation practical and reproducible.
Dueling Network Architectures for Deep RL
The 2015 dueling network paper split a Q-network into separate value and advantage streams, improving Atari performance.
Session-based Recommendations with Recurrent Neural Networks (GRU4Rec)
Used recurrent neural networks to recommend the next item from a short browsing session alone.
Efficient and Robust Automated Machine Learning (auto-sklearn)
Feurer et al.'s 2015 NeurIPS paper introducing auto-sklearn, which automatically selects, configures, and ensembles ML pipelines.
Hidden Technical Debt in Machine Learning Systems
Google's influential 2015 NeurIPS paper arguing that the model is a small part of a real ML system and most cost is hidden maintenance debt.
SSD: Single Shot MultiBox Detector
The 2015 paper that detected objects in one network pass using default boxes across multiple feature scales, trading a little accuracy for big speed.
Deep Residual Learning for Image Recognition (ResNet)
The 2015 Microsoft paper that introduced residual connections, letting networks grow to hundreds of layers and winning that year's ImageNet contest.
Asynchronous Methods for Deep Reinforcement Learning (A3C)
DeepMind's 2016 A3C paper trained deep RL agents in parallel on CPUs, beating the state of the art on Atari in half the time.
Practical Black-Box Attacks against Machine Learning
A 2016 paper showing adversarial examples transfer: a local substitute model can fool a remote classifier you cannot see inside.
LIME (Local Interpretable Model-agnostic Explanations)
The 2016 Ribeiro paper that explains any classifier's individual predictions by fitting a simple model in a local neighborhood.
Communication-Efficient Learning of Deep Networks from Decentralized Data (FedAvg)
The original federated learning paper, introducing the FedAvg algorithm that trains a shared model without moving raw data off devices.
Group Equivariant Convolutional Networks (G-CNN)
Cohen and Welling extended convolution to respect rotations and reflections, not just translations, improving sample efficiency.
XGBoost: A Scalable Tree Boosting System
Chen and Guestrin's 2016 paper introducing XGBoost, the scalable gradient-boosting system that dominated tabular machine learning and Kaggle.
Hyperband: A Novel Bandit-Based Approach to Hyperparameter Optimization
Li et al.'s 2016 method that speeds up tuning by giving more compute to promising configurations and killing weak ones early.
HNSW: Hierarchical Navigable Small World Graphs
The graph-based nearest-neighbor algorithm that powers fast vector search in most vector databases.
Theano: A Python framework for fast computation of mathematical expressions
The 2016 paper documenting Theano, the pioneering Python math compiler from MILA that seeded the modern deep learning framework era.
TensorFlow: A System for Large-Scale Machine Learning
Google's 2016 systems paper describing TensorFlow's dataflow-graph architecture for running machine learning across CPUs, GPUs, and TPUs at cluster scale.
Computational Imaging for VLBI Image Reconstruction (CHIRP)
The 2016 CHIRP paper introduced a computational-imaging method for reconstructing black hole images from sparse VLBI data.
The Age of Em (Robin Hanson, 2016)
Robin Hanson's 2016 book forecasts an economy run by emulated human brains, copied and sped up at will.
CryptoNets: Applying Neural Networks to Encrypted Data
Showed a neural network can run on homomorphically encrypted inputs, making predictions a cloud server cannot read.
Concrete Problems in AI Safety
A 2016 paper that reframed AI safety around five concrete engineering problems in present-day machine learning systems rather than far-off speculation.
Wide & Deep Learning for Recommender Systems
Google's recommender that combines a memorizing linear model with a generalizing neural network.