Neural Networks

06-11-2026

The Foundations of Neural Networks

The first principle of a neural network is a Perceptron. This is essentially an artificial neuron meant to mimic how the human brain worksInterestingly, the development of the perceptron dates all the way back to the 1950s and 60s. The idea is simple, you take several binary inputs $x_{1},x_{2},\dots,$ and produce a single binary output. This output, either 1 or 0, is determined by whether the sum of binary inputs multiplied against their corresponding weights at least meets a given threshold. Formally:

output = \begin{cases} 0 & \text{if } \sum_j w_j x_j < \text{threshold} \ \\ 1 & \text{if } \sum_j w_j x_j \geq \text{threshold} \end{cases}

Or, a more contemporary way to write this would be

output = \begin{cases} 0 & \text{if } w \cdot x + b \leq 0 \ \\ 1 & \text{if } w \cdot x + b > 0 \end{cases} \tag{2}

In the example above, we simply take the dot product of the vectors and move the threshold over to the other side where it is now called bias. Bias, in simpler terms, is how easy it is to get a perceptron to fire. If you're familiar with computer architecture, it should be clear by now that this structure can resemble the simple logic functions of AND, OR, and NAND Of course, the nuance lies in producing weights and altering thresholds; both of which can radically change the decision making process. Perceptrons once established are able to form layers and networks, with weights being updated through Backpropagation.

Sigmoid Neurons

Given the ability to have logic operations and to devise large layers of neurons, how can we actually have a learning algorithm? The idea is that, when you change a weight, the output should change as well. Simply:

w + \Delta w \text{ causes } o + \Delta o

This, if able to be controlled, would work. However, with the current structure of perceptrons thats just not possible. A singular shift could flip a bit and wreak havoc on an almost functioning system. Thus, the idea of a Sigmoid Neuron is introduced, allowing small changes in weights and biases to only cause a small change in their output. The big difference mathematically speaking is that inputs and outputs no longer take on a binary value and instead are represented anywhere in the range $[0,1]$ . The output is calculated via the Sigmoid function:

\sigma(z) \equiv \frac{1}{1 + e^{-z}}

Or more explicitly:

\frac{1}{1 + \exp\left(-\sum_j w_j x_j - b\right)}

This will, if mapped out on a curve, leave us with a smoothed out step functionThe smoothness of the function is the point of the sigmoid function. A true step function is modeled by the perceptron. Now that we can make small changes to weights and biases, we can do a better job modeling changes to output with the help of partial derivatives:

\Delta \text{output} \approx \sum_j \frac{\partial \text{ output}}{\partial w_j} \Delta w_j + \frac{\partial \text{ output}}{\partial b} \Delta b

Essentially, this equation answers: How does output change in total when we measure a weight change and a bias change distinctly of each other.

The Architecture

We are, at this point, already aware two layers of neurons, that being the input neurons and the output neuron(s). The middle layers are deemed hidden layers. Neural nets that have multiple layers in the network are denoted as Multilayer Perceptrons or MLPDespite the fact that we have moved well past perceptrons by this point. Another piece of important terminology is that neural nets that use sigmoid neurons to carry output from one layer into input to the next layer are known as feedforward neural networks. In this type of network, the information flows like a river: in one directionThis is not a property of every neural network framework. To train these networks in a cheap and efficient manner across all layers, researchers have unearthed the art of Gradient Descent.