All sources linked here (books, papers, courses, etc.) are freely accessible.
It could be that some sources are only accessible upon sign up (Coursera courses, for example)
but I'll avoid those if possible. However, none will require a registration fee.
Also notice, many articles are hosted in Medium, which has a monthly limit of articles that can
be accesses. However, this limitation can be easily bypassed by using *Private Browsing*,
such as *Incognito Mode* in Google Chrome or *Private Window* in Firefox.
Basic concepts to start with Deep Learning, before delving into the state-of-the-art
- Neural Network [Video]
- Gradient Descent [Video] [Understanding the Mathematics behind Gradient Descent] [ML Cheatseet Documentation]
- Backpropagation [Video]
- Calculus of backpropagation [Video]
- ReLU: Rectified Linear Units Improve Restricted Boltzmann Machines (2010) [paper]
[...]After some ReLU layers, programmers may choose to apply a pooling layer.
It is also referred to as a downsampling layer. In this category, there are also several
layer options, with maxpooling being the most popular. This basically takes a filter
(normally of size 2x2) and a stride of the same length. It then applies it to the input
volume and outputs the maximum number in every subregion that the filter convolves around.
(https://adeshpande3.github.io/assets/MaxPool.png "Example of MaxPool") [source]
The intuitive reasoning behind this layer is that once we know that a specific feature
is in the original input volume (there will be a high activation value), its exact location
is not as important as its relative location to the other features. As you can imagine,
this layer drastically reduces the spatial dimension (the length and the width change
but not the depth) of the input volume. This serves two main purposes. The first is
that the amount of parameters or weights is reduced by 75%, thus lessening the computation cost.
The second is that it will control overfitting.
The idea of dropout is simplistic in nature. This layer “drops out” a random set of activations
in that layer by setting them to zero. Simple as that. Now, what are the benefits of such a simple
and seemingly unnecessary and counterintuitive process? Well, in a way, it forces the network
to be redundant. By that I mean the network should be able to provide the right classification
or output for a specific example even if some of the activations are dropped out. It makes sure
that the network isn’t getting too “fitted” to the training data and thus helps alleviate the
overfitting problem. An important note is that this layer is only used during training, and not
during test time.
See also: Dropout: A Simple Way to Prevent Neural Networks from Overfitting (2014) [paper])
A network in network layer refers to a conv layer where a 1 x 1 size filter is used. Now,
at first look, you might wonder why this type of layer would even be helpful since receptive
fields are normally larger than the space they map to. However, we must remember that these
1x1 convolutions span a certain depth, so we can think of it as a 1 x 1 x N convolution where
N is the number of filters applied in the layer. Effectively, this layer is performing a N-D
element-wise multiplication where N is the depth of the input volume into the layer.
See also: Network In Network by Min Li. [arxiv.org]
- Learning Neural Network Architectures. [1]
[Here] you can see a summary of some of the most important neural networks of the last years. [Article] (Mar 2017) and [Update] (Sep 2018).
The same authors revise [here] the different architectures for practical applications.
Within each category, models are ordered chronologically.
Review of Deep Learning Algorithms for Image Classification. [1]
- AlexNet (2012) [Paper]
- VGGNet (2015) [Paper] Review
- ResNet (2015) [Paper]
- Squeezenet (2016) [Paper] [github]
Review of Deep Learning Algorithms for Object Detection. [1]
Performance analysis (speed and accuracy) comparison between different object detection architectures. [1]
-
YOLO (2016) [Paper]
-
YOLO9000, a.k.a. YOLOv2 (2016) [Paper]
-
Single Shot Detector (2016). Also known as SSD. [paper] [review]
-
YOLOv3 (2018) [Paper]
-
Real-time Object Detection with YOLO, YOLOv2 and now YOLOv3 [link]
-
SSD object detection: Single Shot MultiBox Detector for real-time processing [link]
What do we learn from region-based object detectors (Faster R-CNN, R-FCN, FPN)? [Part 1]
What do we learn from single shot object detectors (SSD, YOLOv3), FPN & Focal loss (RetinaNet)? [Part 2]
- FCN - Fully Convolutional Network (2014). [paper] [review]
- How to do Semantic Segmentation using Deep learning [link]
- PASCAL Visual Object Classification (PASCAL VOC) [link]
- Common Objects in COntext (COCO) [link]
- Cityscapes Dataset - Semantic Understanding of Urban Street Scenes. [link]
- Labeled Faces in the Wild. Dataset for face detection. [link]
- Network In Network (2013) [arxiv.org]
- Inception (2014) [arxiv.org]
-
Introduction [link]
-
First paper "Generative Adversarial Networks" in 2014 [Paper]
-
How deep learning fakes videos (Deepfake) and how to detect it? [link]
-
Gradient Descent [Video] [Understanding the Mathematics behind Gradient Descent] [ML Cheatseet Documentation]
-
Adaptive Gradient Algorithm (AdaGrad): maintains a per-parameter learning rate that improves performance on problems with sparse gradients (e.g. natural language and computer vision problems).
-
Root Mean Square Propagation (RMSProp): also maintains per-parameter learning rates that are adapted based on the average of recent magnitudes of the gradients for the weight (e.g. how quickly it is changing). This means the algorithm does well on online and non-stationary problems (e.g. noisy).
-
Adaptive Moment Estimation (Adam): A learning rate is maintained for each network weight (parameter) and separately adapted as learning unfolds. "The method computes individual adaptive learning rates for different parameters from estimates of first and second moments of the gradients." The authors describe Adam as combining the advantages of AdaGrad and RMSProp. [deeplearning.ai lecture on Adam] [Gentle Introduction to the Adam Optimization Algorithm for Deep Learning]
-
The Evolution of Gradient Descent by Siraj Raval [video] "Adam usually outperforms the rest [...] followed very closely by AdaGrad and AdaDelta [...] [other optimization algorithms] when data is sparse, they don't perform well"
-
An overview of gradient descent optimization algorithms (2016) [paper]
-
Visualizing and Understanding Convolutional Networks (2013) [arxiv.org]
-
Deep Neural Networks are Easily Fooled: High Confidence Predictions for Unrecognizable Images (2015) [CV Foundation]
-
Understanding Neural Networks Through Deep Visualization (2015) [arxiv.org] [website] [code] [video]
-
Inceptionism: Going Deeper into Neural Networks [Google AI Blog]
-
How computers are learning to be creative, by Blaise Agüera y Arcas [video]
-
Tensorflow (Google)
-
PyTorch (Facebook)
-
OpenVINO (Intel, only inference)
-
OpenAI Gym. A toolkit for developing and comparing reinforcement learning algorithms.
-
OpenAI Universe. "With Universe, any program can be turned into a Gym environment."
- Getting Started with TensorFlow and Deep Learning, from ScyPy 2018 by Josh Gordon (Google) [link]
- fast.ai - Course Overview
- deeplearning.ai by Andrew Ng. Offers a 5-course Deep Learning Specialization (plus a Tensorflow Specialization and an AI For Everyone course). [Thoughts after taking the Deeplearning.ai courses]
- Machine Learning Crash Course - Google
-
Two Minute Papers. [link] Short videos (not exactly two minutes, but 2 to 5 minutes regularly) presenting interesting and novel papers in the field of computer science. Most of them are related to Deep Learning but different (related) topics are also presented.
-
Siraj Raval. [link] Siraj regularly presents technical concepts in an easy way with some examples. Videos are usually 5 to 15min long, but he talks really fast, which sometimes requires you to pause or rewind the video to process all he just said. Topics are presented in a very friendly and "Booliwood-ish" way (whatever that means).
-
3Blue1Brown [link] "3blue1brown, by Grant Sanderson, is some combination of math and entertainment, depending on your disposition. The goal is for explanations to be driven by animations and for difficult problems to be made simple with changes in perspective." The author presents, exceptionaly understandable, mathematical concepts (such as Calculus, Fourier Transorm, etc.) but he also has a series of videos about Deep Learning and delves into various mathematical concepts behind AI and Deep Learning.
-
DotCSV (Carlos Santana Vega)) [link] In Spanish but with English captions. The author has different types of videos: Data Coffees, BITS, and AI news (Noticias IA), where he discusses the latest milestones, curiosities, and news in the field of AI, respectively; Learning AI (Aprendiendo Inteligencia Artificial), presents basic AI concepts; AI Notebooks (IA Notebooks), teaches while programming "live" some of the previously presented AI concepts. Highly recommended even if you don't speak Spanish (just use captions).
- How computers are learning to be creative by Blaise Agüera y Arcas. [video]
- What happens when our computers get smarter than we are? by Nick Bostrom. [video]
- Can we build AI without losing control over it? by Sam Harris. [video]
- How computers learn to recognize objects instantly by Joseph Redmon (YOLO Team). [video]
- The incredible inventions of intuitive AI by Maurice Conti. [video]
- What is a model and what is an architecture?
This terminology is often abused, but the pedantic view is:
A model would be a network architecture with all it's weights viewed as free parameters.
A fit model is a network with fixed weights determined by running a fitting algorithm with
some training data.
Parameters map out the various specific shapes that the model can obtain, fitting chooses
specific values of the weights that best reflect the training data.
Hyperparameters control the behaviour of the fitting algorithm, they are often set to find
the parameters that offer the best performance according to some estimate of hold-out error.
I settled on this terminology after reading Wasserman's All of Statistics.
It's very common to call the fit model just a model. I try to use my words precisely and
consistently, especially when talking to students, but it is hard to avoid sometimes!
[link to stackoverflow answer]
- IMPORTANT: Use proper citation style.