From Perceptron to Deep Neural Networks.
Welcome to my debut blog on artificial intelligence, I like to connect dots instead of learning the entire trajectory and that is what has ensured to sow the seeds of curiosity in my mind. Without wasting a second more, let’s get started!.
What do you think made us as human invent a new domain of science known as artificial intelligence? May be, the fundamental reason for it was to automate the process of decision making,Isn’t it?. The hello world to the process of automating decision making must have been as simple as predicting a yes/no label against a question which must have eventually aided the invention of perceptron. For those, who are yet unaware of what a perceptron is!, Let’s get an understanding about it. Perceptron’s main aim is to learn a binary classifier algorithm which can be regarded as a threshold function that maps its real valued input to an output value. A perceptron can be regarded as a simplified model of a biological neuron. The very first step towards the perceptron model was taken by McCulloch and Pitts which expects a boolean input and outputs a boolean decided primarily by a thresholding function. The McCulloch and Pitts model further got evolved into a perceptron model by thresholding the inputs through their weighted sum to boolean output values. However, due to limitations of perceptron learning to achieve convergence only for linearly seperable inputs, the need for two-layers and eventually multi layer perceptrons rose. The multilayer perceptron model primarily got its breakthrough by formulating the concept of hidden layers and backpropogation. The concept of hidden layers was introduced to allow neural networks to learn non-linearity mappings between the input and output through a non-linear activation function whereas backpropogation ensured that weights were optimized over a number of iterations to minimize the difference between predicted and actual values. Inorder to get the maximum advantage from Multilayer perceptron, it was suggested to use an anti-symmetric activation function which aided towards faster training of a multi-layer perceptron with backpropogation. It was also suggested that the output values and their dynamic range should overlap to the maximum extent with the dynamic range of activation function. The input values to the multi-layer perceptron was suggested to be normalized to ensure that they lie nearest to the unit variance and zero mean curve. Moreover, in order to facilitate better training, the weights were initialized randomly to significantly small values to avoid neurons reaching the state of saturation. Multilayer perceptron are
often reffered as vanilla neural networks. However, Multilayer perceptrons has its own limitation by not being able to utilize the spatial information along with suffering from redundancy of parameters due to accumulation of parameters over each hidden layer units.
These limitations got removed with the introduction of convolutional neural networks which due to its ability to break down the input image into feature maps over various stages each of which capable of detecting a significant distinguishing feature in the image. Moreover, it ensured that the previously disregarded spatial information was now utilized using the CNNs. Unlike the MLPs, The weights became smaller and got shared with sparse connection between layers which eventually lead towards building deeper neural networks. Amongst the aforementioned advantages of CNNs over MLPs, the location invariance of CNNs due to its innate ability to process over a patch of image makes it more reliable. However, CNNs are also limited by the inability to preprocess timeseries and leverage the temporal variation in data.
Hence RNNs(Reccurent Neural Network) were introduced which has a Hidden State being capable of retaining information about a sequence. Being able to retain the information of previous state of the data sequence has revolutionised the entire natural language processing domain drastically. Altogether, evolution of deep neural networks has facilitated the process of automating the decision making ever its inception to an altogether different level and with current trends in the research of reinforcement learning who knows the future of decision making might completely become autonomous, may be with least possible human intervention. No doubt, over significant amount of time this breed of artificial intelligence might actually prove to be the most cognitive species on the earth or who knows it might surpass human intelligence to be termed as extra-terrestial intelligence !?.
