Architecture of a Simple Artificial Neural Network and the Role of Activation Functions

Verified Sources

Nov 3, 2026

A simple Artificial Neural Network (ANN) is composed of layered computational units (neurons) connected by weighted edges. Each neuron forms a linear combination of its inputs and then applies a nonlinear activation function to produce its output. In the most common “feedforward” setup, information flows from the input layer through one or more hidden layers to an output layer, using the forward propagation equations: $z^{(l)} = W^{(l)} a^{(l-1)} + b^{(l)}$ $a^{(l)} = f^{(l)}(z^{(l)})$ where $W^{(l)}$ and $b^{(l)}$ are the weight matrix and bias vector of layer $l$ , $a^{(l-1)}$ is the previous layer’s activations, $z^{(l)}$ is the pre-activation (linear score), and $f^{(l)}$ is the activation function.

A concise architecture view is:

Neuron
Layer
Weights
Bias
Forward propagation

Neural Networks: Architecture + Activation Functions (Conceptual)

Key components of a simple ANN architecture

Consider a network with:

input dimension $d$
one hidden layer of width $h$
output dimension $k$

Let the input vector be $a^{(0)} = x \in \mathbb{R}^d$ . Then:

Hidden layer: $z^{(1)} = W^{(1)}x + b^{(1)},\quad W^{(1)}\in\mathbb{R}^{h\times d},\; b^{(1)}\in\mathbb{R}^{h}$ $a^{(1)} = f(z^{(1)})$
Output layer: $z^{(2)} = W^{(2)}a^{(1)} + b^{(2)},\quad W^{(2)}\in\mathbb{R}^{k\times h},\; b^{(2)}\in\mathbb{R}^{k}$ $a^{(2)} = g(z^{(2)})$

The choice of output activation $g$ depends on the task:

for regression (often), output may be linear: $g(z)=z$
for multi-class classification, commonly a softmax is used to produce a probability distribution

Forward pass of a simple 1-hidden-layer ANN

1
Step 1
Set $a^{(0)}=x$ (the feature vector).
2
Step 2
Compute $z^{(1)} = W^{(1)}a^{(0)} + b^{(1)}$ .
3
Step 3
Compute $a^{(1)} = f(z^{(1)})$ where $f$ is the hidden activation function.
4
Step 4
Compute $z^{(2)} = W^{(2)}a^{(1)} + b^{(2)}$ .
5
Step 5
Apply output activation $g$ : $a^{(2)} = g(z^{(2)})$ to get predictions.

Role of activation functions: why nonlinearity matters

If you removed activation functions and used only linear transformations, the entire network would collapse to a single linear mapping (composition of linear maps is linear). Activation functions enable the network to model nonlinear relationships by introducing a nonlinear step between layers.

Core concepts:

Nonlinearity
Activation function
Expressivity
Universal approximation
Decision boundary

Common activation functions (and typical behavior)

Sigmoid: $f(z)=\dfrac{1}{1+e^{-z}}$

outputs values in $(0,1)$
historically popular for binary classification but can suffer from gradient issues

Hyperbolic tangent (tanh): $f(z)=\tanh(z)$

outputs values in $(-1,1)$
often zero-centered compared to sigmoid

ReLU: $f(z)=\max(0,z)$

outputs $0$ for negative inputs, identity for positive
typically easier to optimize in deep networks

Activation-function intuition

"Think of activation functions as “learnable feature gates”: they decide how strongly each neuron responds to its input score $z$ . Without this gating, the model cannot form complex nonlinear patterns."

Training stability: gradients

"Some activations can lead to small gradients over many layers (e.g., sigmoid/tanh in deep nets), making learning slow. ReLU-like functions are often preferred for their gradient behavior."

How activation functions connect to learning (backprop intuition)

Training adjusts $W$ and $b$ to reduce a loss function. In backpropagation, gradients must flow through the activation function, which depends on the derivative $f'(z)$ :

the chain rule multiplies by $f'(z)$ at each layer
therefore, activations with derivatives that saturate (become very small) can hinder gradient flow

A simplified gradient idea for a neuron output $a=f(z)$ is:

if $z$ lies in a region where $f'(z)\approx 0$ , then the gradient signal becomes weak
this reduces how much upstream weights can be updated

Key term:

Backpropagation

Qualitative comparison of activation functions

General behavior used in practice (not a guarantee in every model).

From architecture to trained network (conceptual lifecycle)

Choose layers and width

1) Design architecture

Select input size, number of hidden layers, and neurons per layer."

Pick $f^{(l)}$ and output $g$

2) Choose activations

Use ReLU-like functions for hidden layers; choose output activation based on task."

Run forward + compute loss

3) Train parameters

Compute predictions via forward propagation and evaluate a loss function."

Backpropagate gradients

4) Learn from error

Update weights and biases using activation derivatives in the chain rule."

Generalization check

5) Validate performance

Compare train vs. validation behavior to detect under/overfitting."

Common questions and edge cases

ANN Architecture + Activation Functions: Quick Review

1 / 5

Question · Term

What is the role of a neuron in an ANN?

Click to reveal

Answer · Definition

A neuron computes $z=Wx+b$ and then outputs $a=f(z)$ using an activation function.

Knowledge Check

Question 1 of 3

Q1Single choice

In a feedforward ANN layer, what are $z^{(l)}$ and $a^{(l)}$ ?

Option A: $z^{(l)}$ is the activation and $a^{(l)}$ is the weighted sum

Option B: $z^{(l)}$ is the weighted sum $W^{(l)}a^{(l-1)}+b^{(l)}$ and $a^{(l)}$ is the activated output

Option C: Both are identical and equal to the loss

Option D: $z^{(l)}$ depends only on the bias and $a^{(l)}$ depends only on the weights

Explore Related Topics

Understanding Activation Records and Variable Resolution in Compiler Design

Activation records (stack frames) are the runtime structures that store a function’s context, support calls—including recursion—and enable variable access through fixed offsets and scope links.

Local variables are accessed in O(1) time by adding a compile‑time constant offset to the frame pointer (FP).
Non‑local variables in lexically scoped languages use a static (access) link to the activation record of the enclosing block, while the dynamic (control) link points to the caller’s record.
The function call lifecycle involves argument evaluation, saving return address, establishing FP, allocating locals, executing, and tearing down the frame.
Optimizations such as display registers provide constant‑time access to any outer scope, avoiding long static‑link traversals.

Agents in AI: Definitions, Mechanisms, and Taxonomy

Neural Networks are Inspired By (i) Genetic evolution (ii) Human brain (iii) Particle movement (iv) Graph theory

Browse all research articles