The artificial neural network (ANN) or simply neural network is a form of machine learning (ML) that utilises interconnected nodes or neurons arranged in layers, resembling the structure of the human brain. It is a subset of artificial intelligence (AI) and a fundamental technique in deep learning which allow computers to identify patterns, make decisions, and solve complex problems.
Early Foundations and Developments
The journey of ANN has evolved through groundbreaking discoveries, from early mathematical models to cutting-edge AI applications. The following timeline highlights some major milestones in the advancement of ANNs:
|
Year |
Development |
|
1943 |
McCulloch-Pitts Model Warren McCulloch and Walter Pitts proposed a mathematical model of artificial neurons, laying the foundation for ANN. |
|
1949 |
Hebbian Learning Donald Hebb introduced the principle that neural pathways strengthen with repeated activation, influencing future learning models. |
|
1958 |
Perceptron Development Frank Rosenblatt developed the perceptron, an early ANN capable of learning from data but limited to solving linearly separable problems. |
|
1965 |
Adaptive Linear Neuron (ADALINE) Bernard Widrow and Marcian Hoff developed ADALINE and Multiple Adaptive Linear Neuron (MADALINE), early ANNs used for signal processing. |
|
1969 |
Limitations of Perceptrons Marvin Minsky and Seymour Papert published Perceptrons, demonstrating the model’s inability to handle nonlinear problems, leading to a decline in research interest. |
|
1970s–80s |
AI Winter Due to perceptrons’ limitations and lack of progress in ANN research, funding and academic interest declined. |
|
1982 |
Hopfield Networks John Hopfield introduced Hopfield networks, a form of recurrent ANN used for associative memory and optimisation problems. |
|
1986 |
Backpropagation Rediscovery Researchers such as Paul Werbos, David Rumelhart, and Geoffrey Hinton introduced backpropagation, enabling efficient training of multi-layer neural networks and reigniting interest in the field. |
|
1989 |
Convolutional Neural Networks (CNNs) Yann LeCun pioneered CNNs, demonstrating their effectiveness in image recognition tasks. |
|
1995 |
Support Vector Machines (SVMs) Vladimir Vapnik introduced SVMs, which outperform early ANN in certain classification tasks. |
|
1997 |
Long Short-Term Memory (LSTM) Sepp Hochreiter and Jurgen Schmidhuber introduced LSTM networks, solving long-term dependency issues in recurrent neural networks (RNNs). |
|
2006 |
Deep Learning Resurgence Geoffrey Hinton and colleagues developed deep belief networks (DBNs), marking the revival of deep learning research. |
|
2009 |
CUDA Acceleration for Neural Networks Researchers began using GPUs with NVIDIA CUDA to accelerate deep learning computations, leading to significant performance improvements. |
|
2014 |
Generative Adversarial Networks (GANs) Ian Goodfellow and his team introduced GANs, enabling realistic image and data generation through adversarial learning. |
|
2018 |
BERT and Transfer Learning in NLP Google introduced BERT (Bidirectional Encoder Representations from Transformers), improving contextual understanding in NLP tasks. |
|
2020 |
GPT-3 and Large Language Models OpenAI released GPT-3, a 175-billion-parameter language model, setting new benchmarks in AI-powered text generation and NLP tasks. |
|
2021 |
AlphaFold Revolutionises Biology DeepMind’s AlphaFold solved the protein- folding problem, accelerating drug discovery and biological research. |
|
2023 |
Multimodal AI and GPT-4 OpenAI released GPT-4, featuring improved reasoning, multimodal capabilities (processing text and images), and greater adaptability for real-world applications. |
|
2025 |
Frontiers in Neuromorphic Computing and Quantum AI Scientists are exploring neuromorphic computing (brain-inspired chips) and quantum AI to push the boundaries of ANN efficiency and intelligence. |
The 2024 Nobel Prize in Physics was awarded to two scientists, namely, John Hopfield and Geoffrey Hinton, for their “foundational discoveries and inventions that made machine learning with artificial neural networks possible.”
Working of ANNs
Neural networks are just the extension of perceptrons. ANNs consist of artificial neurons, called perceptions units. A typical ANN consists of three layers of artificial neurons. The number of nodes in a layer could range from a few to millions, depending on the complexity of the network required to identify hidden patterns in the data.
ANN includes an input layer, hidden layers, and an output layer. The input layer receives external data for analysis, which then passes through one or more hidden layers that process the data. The output layer then generates a response based on the processed data. These weighted connections mean that the impact of inputs from the previous layer is optimised by assigning different weights to each input. During the training process, these weights are adjusted to improve the model’s performance.
Similar to the human brain, each node in a neural network is connected to others and has its own associated weight and threshold. If the output of a node exceeds the specified threshold, the node is activated and passes data to the next layer. If the output is below the threshold, no data is transmitted to the next layer.
ANNs are trained using a training set. For instance, to teach an ANN to recognise a cat, it is shown thousands of cat images to help the network learn how to identify one. After sufficient training, the network’s ability to correctly identify cat images are tested. This is done by having the ANN classify new images as either cat images or not, with the results compared to a human-provided label. If the ANN makes an incorrect identification, back propagation is used to adjust its learning. This involves fine-tuning the connection weights in the network based on the error rate. The process repeats until the ANN could accurately identify a cat image with minimal error.
Artificial Neurons vs Biological Neurons
Artificial neural networks are inspired by biological neurons or nerve cells which transmit and receive signals that enable humans to perform functions like moving muscles, breathing, speaking, eating, and thinking, among other activities. Artificial and biological neurons share similarities in both structure and function.
Structure Biological neurons have a soma (cell body) to process signals, dendrites to receive them, and an axon to transmit them. Similarly, artificial neurons have input nodes (receiving signals), hidden layer nodes (processing signals), and output layer nodes (producing the final output via activation functions).
Synapses In biological neurons, synapses connect neurons and transmit impulses. In artificial neurons, synapses are represented by weights connecting nodes between layers, determining the strength of the connections.
Learning Biological neurons learn through synaptic plasticity, adjusting the strength of synapses based on activity. In artificial neural networks, learning happens via backpropagation, adjusting the weights based on the error between predicted and actual outcomes.
Activation In biological neurons, activation occurs when impulses reach a threshold, triggering the neuron to fire. In artificial neurons, activation is performed using a mathematical function (activation function) that maps inputs to outputs.
Types of Neural Networks
Neural networks are classified based on their architecture, training methods, and applications. Some common types are as follows:
Feedforward neural networks (FNNs) A simple type that processes data from input to output in a linear manner (input—hidden layers—output) commonly used for tasks like classification and regression. For example, Multi-layer perception (MLP)
RNNs Designed for sequential data (e.g., time series or speech), RNNs retain memory of previous inputs, making them ideal for time-based data. For example, LSTM and GRU (Gated Recurrent Unit)
LSTM is a type of RNN, designed to retain long-term dependencies in sequential data. It uses input, forget, and output gates to manage information flow. For example, Google Translate and Siri
CNNs Specialised for image tasks, CNNs use convolutional layers to extract spatial features like edges and shapes from images. It includes pooling layers to reduce dimensions. For example, AlexNet, ResNet (Residual Network), and VGG (Visual Geometry Group).
GANs Consisting of a generator and a discriminator, it creates and assesses data, leading to the generation of increasingly realistic content such as images and videos. For example, Deepfake and Style GAN
Applications of ANNs
Neural networks have various applications across industries. Some of them are as follows:
Social Media ANNs are widely used in social media. For example, Facebook’s ‘People You May Know’ feature suggests potential friends by analysing a person’s profile, interest, current friends, etc., to determine the people the person might know. Another common application of ANNs in social media is facial recognition, where around 100 reference points on a person’s face are identified and matched with existing database entries using convolutional neural networks.
Marketing and Sales E-commerce sites like Amazon, Flipkart, etc., recommend products based on the customer’s previous browsing history. Similarly, online food delivery platforms such as Swiggy, Zomato, etc., suggest restaurants based on customers’ tastes and past orders. This personalised approach is common across various industries like book sites, movie services, and hospitality. This is achieved through personalised marketing. ANNs are used to analyse customer preferences, dislikes, and past shopping behaviour, allowing these platforms to customise marketing campaigns accordingly.
Health Care ANNs are applied in oncology to train algorithms capable of detecting cancerous tissue at a microscopic level with the same accuracy as trained physicians. Additionally, rare diseases that manifest through physical characteristics could be identified at early stages using facial analysis of patient’s photos. The widespread use of ANN in health care has the potential to enhance the diagnostic capabilities of medical professionals, ultimately improving the quality of medical care, worldwide.
Personal Assistants Virtual assistants like Siri, Alexa, and Cortana use speech recognition powered by natural language processing (NLP) to interact with users and generate appropriate responses. NLP relies on ANNs to manage various tasks for these assistants, such as handling language syntax, semantics, speech accuracy, and ongoing conversations.
Finance It is used in finance for predicting stock prices, fraud detection, and risk management. It helps in financial predictions by processing historical data of financial instruments. ANNs help in process and quality control. Besides, it is also used in chemical compound identification. Furthermore, ANNs enable real-time decision making, enhancing user experience in gaming and enabling autonomous systems like self-driving cars.
Advantages of ANNs
- ANNs are valuable for tasks where the relationship between inputs and outputs is complex or unclear, as they could adjust to new situations and learn from the data.
- Their expertise in pattern recognition makes them highly effective in tasks, such as audio and image recognition, NLP, and other complex data patterns.
- Due to their ability to perform parallel processing, ANNs could handle multiple tasks simultaneously, enhancing the speed and efficiency of computations.
- ANNs could model and understand complex relationships in data due to the nonlinear activation functions in neurons, which address the limitations of linear models.
Disadvantages of ANNs
- Training large neural networks could be a time-consuming and resource-intensive process requiring significant computing power.
- As ‘black box’ models, ANNs present a challenge in critical applications because it is hard to interpret how they make decisions.
- Overfitting occurs when ANNs memorise the training data instead of recognising underlying patterns. While regularisation techniques help reduce this issue, the problem still exists.
- For effective training, neural networks often require large and labelled datasets; otherwise, their performance may degrade due to incomplete or biased data.
Conclusion
ANNs play a pivotal role in advancing AI by enabling systems to make decisions with minimal human intervention. Their ability to learn and model complex as well as nonlinear relationships between input and output data is key to the success of AI applications. However, the effectiveness of ANNs in AI depends on careful training and fine-tuning to ensure they operate as intended, contributing to the growing capabilities of the AI across various fields.
© Spectrum Books Pvt. Ltd.
