Historical Dates & Timeline for Deep Learning

This post is a quick check on the timeline including historical dates in relation to the evolution of deep learning. Without further ado, let’s get to the important dates and what happened on those dates in relation to deep learning:

Year	Details/Paper Information	Who’s who
1943	An artificial neuron was proposed as a computational model of the “nerve net” in the brain. Paper: “A logical calculus of the ideas immanent in nervous activity,” Bulletin of Mathematical Biophysics, volume 5, 1943	Warren McCulloch, Walter Pitts
Late 1950s	A neural network application by reducing noise in phone lines was developed Paper: Andrew Goldstein, “Bernard Widrow oral history,” IEEE Global History Network, 1997	Bernard Widrow, Ted Hoff
Late 1950s	The Perceptron was introduced. It mimicked the neural structure of the brain and showed an ability to learn Paper: Frank Rosenblatt, “The Perceptron: A probabilistic model for information storage and organization in the brain,” Psychological Review, volume 65, number 6, 1958.	Frank Rosenblatt
1969	Proved mathematically that the Perceptron could only perform very basic tasks. It was published in their book, “Perceptrons”. They also discussed the challenges in relation to training multi-layer neural networks. Marvin Minsky and Seymour A. Papert, Perceptrons: An introduction to computational geometry, MIT Press, January 1969.	Marvin Minsky, Seymour Papert
1986	Solved the training challenges of multi-layer neural network using back propagation training algorithm David E. Rumelhart, Geoffrey E. Hinton, and Ronald J. Williams, “Learning representations by back-propagating errors,” Nature, volume 323, October 1986; for a discussion of Linnainmaa’s role see Juergen Schmidhuber, Who invented backpropagation?, Blog post	Geoffrey Hinton, David Rumelhart, Ronald Williams
1998	Made the Use of neural networks on image recognition tasks. Defined the concept of convolutional neural networks (CNN), in his paper, which mimic the human visual cortex. Yann LeCun, Patrick Haffner, Leon Botton, and Yoshua Bengio, Object recognition with gradient-based learning, Proceedings of the IEEE, November 1998.	Yann LeCun
1998	Popularized the “Hopfield” network which was the first recurrent neural network (RNN) John Hopfield, Neural networks and physical systems with emergent collective computational abilities, PNAS, April 1982.	John Hopfield
1997 – 1998	Introduced long short-term memory (LSTM), greatly improving the efficiency and the practicality of recurrent neural networks (RNN) Sepp Hochreiter and Juergen Schmidhuber, “Long short-term memory,” Neural Computation, volume 9, number 8, December 1997.	Jurgen Schmidhuber and Sepp Hochreiter
2012	Highlighted the power of deep learning by showing significant results in the well-known ImageNet competition Alex Krizhevsky, Ilya Sutskever, and Geoffrey E. Hinton, ImageNet classification with deep convolutional neural networks, NIPS 12 proceedings of the 25th International Conference on Neural Information Processing Systems, December 2012.	Geoffrey Hinton and two of his students
2012 – 2103	Breakthrough work on large scale image recognition at Google Brain Jeffrey Dean et al., Large scale distributed deep networks, NIPS 2012.	Jeffrey Dean, Andrew Ng
2014	Published his paper on generative adversarial networks (GAN) Ian J. Goodfellow, Generative adversarial networks, ArXiv, June 2014.	Ian Goodfellow
2014	Reinforcement learning: An Introduction Richard S. Sutton and Andrew G. Barto, Reinforcement learning: An introduction, MIT Press, 2014.	Richard S. Sutton, Andrew G. Barto

Author
Recent Posts

Ajitesh Kumar

I have been recently working in the area of Data analytics including Data Science and Machine Learning / Deep Learning. I am also passionate about different technologies including programming languages such as Java/JEE, Javascript, Python, R, Julia, etc, and technologies such as Blockchain, mobile computing, cloud-native technologies, application security, cloud computing platforms, big data, etc. I would love to connect with you on Linkedin.
Check out my latest book titled as First Principles Thinking: Building winning products using first principles thinking.