Historical Dates & Timeline for Deep Learning

deep learning timeline

This post is a quick check on the timeline including historical dates in relation to the evolution of deep learningWithout further ado, let’s get to the important dates and what happened on those dates in relation to deep learning:

Year Details/Paper Information Who’s who

An artificial neuron was proposed as a computational model of the “nerve net” in the brain.

Paper: “A logical calculus of the ideas immanent in nervous activity,” Bulletin of Mathematical Biophysics, volume 5, 1943

Warren McCulloch, Walter Pitts
Late 1950s

A neural network application by reducing noise in phone lines was developed

Paper: Andrew Goldstein, “Bernard Widrow oral history,” IEEE Global History Network, 1997

Bernard Widrow, Ted Hoff
Late 1950s

The Perceptron was introduced. It mimicked the neural structure of the brain and showed an ability to learn

Paper: Frank Rosenblatt, “The Perceptron: A probabilistic model for information storage and organization in the brain,” Psychological Review, volume 65, number 6, 1958.

Frank Rosenblatt

Proved mathematically that the Perceptron could only perform very basic tasks. It was published in their book, “Perceptrons”. They also discussed the challenges in relation to training multi-layer neural networks.

Marvin Minsky and Seymour A. Papert, Perceptrons: An introduction to computational geometry, MIT Press, January 1969.

Marvin Minsky, Seymour

Solved the training challenges of multi-layer neural network using back propagation training algorithm

David E. Rumelhart, Geoffrey E. Hinton, and Ronald J. Williams, “Learning representations by back-propagating errors,” Nature, volume 323, October 1986; for a discussion of Linnainmaa’s role see Juergen Schmidhuber, Who
invented backpropagation?, Blog post

Geoffrey Hinton, David Rumelhart, Ronald Williams

Made the Use of neural networks on image recognition tasks. Defined the concept of
convolutional neural networks (CNN), in his paper, which mimic the human visual cortex.

Yann LeCun, Patrick Haffner, Leon Botton, and Yoshua Bengio, Object recognition with gradient-based learning, Proceedings of the IEEE, November 1998.

Yann LeCun

Popularized the “Hopfield” network which was the first recurrent neural network (RNN)

John Hopfield, Neural networks and physical systems with emergent collective computational abilities, PNAS, April 1982.

John Hopfield
1997 – 1998

Introduced long short-term memory (LSTM), greatly improving the efficiency and
the practicality of recurrent neural networks (RNN)

Sepp Hochreiter and Juergen Schmidhuber, “Long short-term memory,” Neural Computation, volume 9, number 8, December 1997.

Jurgen Schmidhuber and Sepp Hochreiter

Highlighted the power of deep learning by showing significant results in the well-known ImageNet competition

Alex Krizhevsky, Ilya Sutskever, and Geoffrey E. Hinton, ImageNet classification with deep convolutional neural networks, NIPS 12 proceedings of the 25th International Conference on Neural Information Processing Systems,
December 2012.

Geoffrey Hinton and two of his students
2012 – 2103

Breakthrough work on large scale image recognition at Google Brain

Jeffrey Dean et al., Large scale distributed deep networks, NIPS 2012.

Jeffrey Dean, Andrew Ng

Published his paper on generative
adversarial networks (GAN)

Ian J. Goodfellow, Generative adversarial networks, ArXiv, June 2014.

Ian Goodfellow

Reinforcement learning: An Introduction

Richard S. Sutton and Andrew G. Barto, Reinforcement learning: An introduction, MIT Press, 2014.

Richard S. Sutton, Andrew G. Barto 
Ajitesh Kumar
Follow me

Ajitesh Kumar

I have been recently working in the area of Data analytics including Data Science and Machine Learning / Deep Learning. I am also passionate about different technologies including programming languages such as Java/JEE, Javascript, Python, R, Julia, etc, and technologies such as Blockchain, mobile computing, cloud-native technologies, application security, cloud computing platforms, big data, etc. For latest updates and blogs, follow us on Twitter. I would love to connect with you on Linkedin. Check out my latest book titled as First Principles Thinking: Building winning products using first principles thinking. Check out my other blog, Revive-n-Thrive.com
Posted in Data Science, Deep Learning, Machine Learning. Tagged with , .

Leave a Reply

Your email address will not be published. Required fields are marked *