Synetic College Logo
Unit 3 of 4 · M.Sc IT Sem 4

Unit 3: Neural networks

Artificial Intelligence & Soft Computing notes · PTU syllabus (PGCA1926)

3 min read9 topics10 exam questions
On this page
  1. Unit summary
  2. Soft computing vs hard computing
  3. Areas and applications of soft computing
  4. The artificial neuron and activation functions
  5. Learning rules
  6. Single-layer perceptrons
  7. Back-propagation networks
  8. Associative memory
  9. Adaptive resonance theory
  10. Self-organising maps and unsupervised networks
  11. Key terms
  12. Quick revision
  13. Important questions

Unit summary

Soft computing tolerates imprecision to solve hard real-world problems, and neural networks learn from data. This unit covers soft versus hard computing, the areas and applications of soft computing, learning rules and activation functions, single-layer perceptrons, back-propagation networks, associative memory, adaptive resonance theory, self-organising maps and unsupervised learning networks.

After this unit you can

  • Compare soft and hard computing and list applications
  • Explain learning rules and activation functions
  • Train perceptrons and back-propagation networks
  • Explain associative memory, ART and self-organising maps

PTU syllabus topics

  • Soft computing vs hard computing
  • major areas and applications of soft computing
  • neural network learning rules and activation functions
  • single-layer perceptrons
  • backpropagation networks and architecture
  • associative memory
  • adaptive resonance theory
  • self-organizing maps
  • unsupervised learning networks
ProcessBackpropagation
  1. 1Forward pass

    Compute outputs layer by layer

  2. 2Compute error

    Target − output

  3. 3Backward pass

    Propagate error gradients

  4. 4Update weights

    Δw = η × δ × input

  5. 5Repeat for each epoch
1

Topic 1

Soft computing vs hard computing

ComparisonHard and soft computing
Hard computing
Soft computing

Basis

Precise models, binary logic, exact answers

Approximate reasoning tolerant of imprecision, uncertainty and partial truth

Techniques

Conventional algorithms, numerical analysis

Neural networks, fuzzy logic, genetic algorithms, probabilistic reasoning

Programs

Written explicitly

Often learn or evolve

Example

Payroll calculation

Handwriting recognition, washing machine control

2

Topic 2

Areas and applications of soft computing

ClassificationSoft computing
Soft computing

  • Neural networks

    Learning from examples — image and speech recognition

  • Fuzzy logic

    Reasoning with vague concepts — appliance and traffic control

  • Evolutionary computation

    Optimisation — scheduling, design

  • Probabilistic reasoning

    Uncertainty — Bayesian networks

  • Hybrid systems

    Neuro-fuzzy, genetic-fuzzy controllers

3

Topic 3

The artificial neuron and activation functions

Key formulasNeuron model

  • Net input

    net = Σ wᵢxᵢ + b

  • Output

    y = f(net)

ComparisonActivation functions
Formula
Use

Binary step

1 if net ≥ θ else 0

Perceptrons

Linear

f(net) = net

ADALINE, regression outputs

Sigmoid (logistic)

1 ÷ (1 + e^(−net)); range 0–1

Back-propagation, probabilities

Tanh (bipolar sigmoid)

Range −1 to 1

Hidden layers

ReLU

max(0, net)

Deep networks

4

Topic 4

Learning rules

Key termsNeural learning rules
Hebbian rule
Δw = η x y — strengthen weights between co-active neurons
Perceptron rule
Δw = η (t − y) x — update only on errors
Delta (Widrow–Hoff, LMS) rule
Δw = η (t − net) x — minimise squared error; used by ADALINE
Competitive (winner-take-all)
Only the winning neuron updates — used in SOMs
Outer product rule
W = Σ s tᵀ — stores pattern pairs in associative memory
5

Topic 5

Single-layer perceptrons

An artificial neuron computes a weighted sum of inputs plus a bias and applies an activation function. The perceptron learns with the rule w ← w + η (t − y) x.

  • A single-layer perceptron can learn linearly separable functions like AND and OR, but not XOR — which needs a multi-layer network.
  • A support vector machine (SVM) finds the separating line (hyperplane) with the maximum margin between classes; the closest points are support vectors.

Example

AND with bipolar inputs and targets, η = 1, θ = 0, starting weights 0: after one epoch of perceptron updates, w1 = 1, w2 = 1, b = −1 classifies all four patterns correctly.

  • ADALINE (adaptive linear neuron) uses the delta rule on the net input; MADALINE combines several ADALINEs to solve non-linear problems such as XOR.
6

Topic 6

Back-propagation networks

ProcessBack-propagation training
  1. 1

    Initialise small random weights

  2. 2

    Forward pass: compute outputs layer by layer

  3. 3

    Compute output error (target − output)

  4. 4

    Backward pass: compute error terms δ for output and hidden layers

  5. 5

    Update weights Δw = η δ x (optionally with momentum)

  6. 6

    Repeat for all patterns over many epochs until error is small

  • Architecture: input layer, one or more hidden layers with sigmoid or ReLU units, output layer; fully connected feed-forward. Issues: local minima, choice of learning rate, overfitting, vanishing gradients in deep nets.
7

Topic 7

Associative memory

ComparisonAssociative memories
Stores
Recall

Auto-associative

Patterns associated with themselves

Recovers a complete pattern from a noisy or partial version

Hetero-associative

Pairs of different patterns (s, t)

Gives t when presented s

Hopfield network

Recurrent auto-associative network with symmetric weights

Converges to stored patterns (energy minima)

  • Weights are set by the Hebb or outer product rule: W = Σ sᵀt over the stored pairs.
8

Topic 8

Adaptive resonance theory

  • ART (Grossberg and Carpenter) solves the stability–plasticity dilemma: learn new patterns without forgetting old ones. An input is compared with stored category prototypes; if the match exceeds the vigilance parameter, the prototype is updated (resonance); otherwise a new category is created.
  • ART1 handles binary inputs; ART2 handles continuous inputs.
9

Topic 9

Self-organising maps and unsupervised networks

ProcessKohonen self-organising map
  1. 1Initialise weight vectors of a grid of neurons
  2. 2Present an input
  3. 3Find the winner (closest weight vector)
  4. 4Update the winner and its neighbours towards the input
  5. 5Shrink the learning rate and neighbourhood over time
  • SOMs map high-dimensional data to a 2D grid preserving topology — for clustering and visualisation. Other unsupervised networks: competitive learning (Maxnet), learning vector quantisation (supervised variant), counter-propagation networks.

Key terms

Soft computing
Computing tolerant of imprecision and uncertainty
Activation function
Function producing a neuron's output from its net input
Perceptron
Single-layer network trained with the perceptron rule
Back-propagation
Algorithm propagating errors backwards to update weights
Vigilance parameter
ART threshold deciding whether input matches a category

Quick revision

  • Hard vs soft computing; components and applications.
  • Neuron model; step, linear, sigmoid, tanh, ReLU.
  • Hebb, perceptron, delta, competitive, outer product rules.
  • Perceptron, ADALINE, MADALINE; XOR needs hidden layers; back-propagation steps.
  • Auto- and hetero-associative memory, Hopfield; ART; SOMs.

Important exam questions

Practice questions written to the PTU exam pattern for this unit's syllabus: short answers (Section A style) and long answers (Sections B and C style).

Short-answer questions

  1. Q1.Distinguish hard and soft computing.
  2. Q2.Write the sigmoid function.
  3. Q3.State the Hebbian learning rule.
  4. Q4.Why can't a single-layer perceptron learn XOR?
  5. Q5.What is the stability–plasticity dilemma?
  6. Q6.What is a self-organising map used for?

Long-answer questions

  1. Q1.Explain soft computing and its applications.
  2. Q2.Explain learning rules and activation functions.
  3. Q3.Explain perceptron training and the back-propagation algorithm.
  4. Q4.Explain associative memory, ART and self-organising maps.

Stuck on this unit?

Message SBS on WhatsApp for help with Artificial Intelligence & Soft Computing, or to ask about studying M.Sc IT at Synetic.

WhatsApp us