Most of the popular news about Artificial Intelligence isn’t about what it looks like on the inside, but about what it can do. That can make it really hard to find the small first step if you want to work on this, or if you’re simply curious about how it works.
So in this article I want to show you how you can understand it at a fundamental level. No calculus or neuroscience here (or at least not in practice, and not as examples). We’re going to understand neural networks with no prior knowledge.
Let’s start with something from everyday life.
A calculator, a traffic light, or a clock
When you want to do some math, you press the numbers in order, and they show up in order on the calculator. These are simple instructions, both yours (pressing each key) and the calculator’s (showing the numbers in order).
We call this an algorithm, maybe the most technical-looking word for a very simple concept. You’ve created millions of algorithms throughout your life, and you’re even following one while reading this. Ordered, step-by-step instructions to achieve something.
Take a second to remember that a traffic light changes color every so often, following instructions, following an algorithm. And a clock does the same: every second, it moves a hand.
Here is a classic algorithm, drawn as a flowchart, about what to do when a lamp doesn’t work:
flowchart TD
A[Lamp doesn't work] --> B{Lamp plugged in?}
B -- No --> C[Plug in lamp]
B -- Yes --> D{Bulb burned out?}
D -- Yes --> E[Replace bulb]
D -- No --> F[Repair lamp]
Notice how it guides you with questions. Yes or No will be the answers, and depending on which answers come up, it will lead you to do one thing or another.
Artificial Intelligence is not an algorithm
At least not like the ones I’ve shown you so far. That might sound annoying or confusing. But the first people in the field of Artificial Intelligence started from the same place. They began to imagine:
If you, the person reading this, are built to follow instructions (algorithms) like a calculator, could we design calculators as complex as you?
What’s interesting here is that a calculator wouldn’t take even a second to multiply two four-digit numbers, and most people take much longer to do the same. A traffic light or a clock can change state with incredible precision. For us, time slips away faster.
A program can take a 100-page document and find a word in it in a fraction of a second. We have to read the document.
Of course, you could push back on this: a human being, unlike a machine, makes complex decisions. Even something like understanding handwritten numbers can be pretty hard for a machine.
Sometimes those decisions end in catastrophe, and other times they create something brilliant.
The brain makes such unique decisions that this is where we start to shift the meaning of “intelligence”: from multiplying three-digit numbers to knowing when you recognize someone in a picture, when you can imagine a cube in your mind, when you break the rules.
And what was our answer?
To build a machine capable of responding to the unpredictable, just like we would.
And off we go…
How do you create an algorithm that responds to the unpredictable?
If you’ve made it this far and didn’t close the tab: thank you, this already exceeded my expectations.
Let’s go back to the lamp algorithm, because there’s something to break here.
I want you to notice that this problem lives in a pretty simple world:
First, because there are no ambiguities, nothing happens outside of it. For example, maybe the lamp doesn’t work… because there’s no electricity in the house. So you go and buy a new bulb, and another one, and another one. And suddenly there’s no solution.
Second, this algorithm doesn’t allow anything new, there’s no experience behind it. Sure, you could add new arrows to the diagram and allow new options, right? Yes, but we’re looking for something that does it automatically.
Welcome to Machine Learning
The term “machine learning” was coined in 1959 by Arthur Samuel, who built a checkers program that got better by playing. I’ll explain what we mean by it a bit later.
Let’s take a second to remember the problem that brought us here:
We want an algorithm capable of responding to unexpected and, of course, complex situations. For example, it could have the ability to go through an experience and learn from it.
In other words: Artificial Intelligence.
Yes, yes. And I’ll tell you something: many people who study Computer Science think the following at this point.
If what we need is an algorithm that can consider new options and possibilities, we just need to make a bigger algorithm. Imagine the same lamp diagram, but now with A HUNDRED OR A THOUSAND ARROWS COVERING EVERYTHING WE COULD IMAGINE. Every option, every situation we could cover!
This is an exaggerated, dramatized example of what that would look like, at a much smaller scale than the real thing:
flowchart TD
A[Lamp doesn't work] --> B{Lamp plugged in?}
B -- No --> C[Plug in lamp]
B -- Yes --> P{Power in the house?}
P -- No --> Q{Whole street dark?}
Q -- Yes --> R[Call the power company]
Q -- No --> S{Breaker tripped?}
S -- Yes --> T[Reset breaker]
S -- No --> U[Call an electrician]
P -- Yes --> D{Bulb burned out?}
D -- Yes --> E{Spare bulb at home?}
E -- Yes --> F[Replace bulb]
E -- No --> G{Store open?}
G -- Yes --> H[Buy a bulb]
G -- No --> I[Use another lamp]
D -- No --> J{Switch broken?}
J -- Yes --> K[Replace switch]
J -- No --> L{Cable damaged?}
L -- Yes --> M[Replace cable]
L -- No --> N[...and so on, forever]
People did go down this road. In the 1970s and 1980s, expert systems tried to capture human knowledge as huge collections of hand-written if-then rules, and they worked well in narrow domains. And to some extent the idea has merit, but it doesn’t give us everything we want. Trying to solve something with tons of variables, options, and situations using an algorithm we wrote exhaustively by hand is not what we’re after here. Every new situation needs a new rule, written by a person.
Let’s leave the lamp and move on to a more interesting example.
We’re going to a very common starting problem in Machine Learning: character recognition. Or rather, understanding other people’s handwriting.
Back to machines making complex decisions
Let’s say I’m about to build an optical calculator, a kind of machine that:
- Receives numbers written on paper.
- Analyzes them with a Neural Network.
- Recognizes the number we wrote.
- Finally gives us the result on screen.
flowchart LR
A[Handwritten number on paper] --> B[Camera]
B --> C[Neural network]
C --> D[Recognized number]
D --> E[Result on screen]
With a simple calculator, it’s very easy for the machine to understand the electrical signals it receives when we press one of its buttons with a finger. There’s a clear, limited set of options: 1, square root, clear, percent, division, and so on.
But now we’re about to build an optical calculator that can analyze data it doesn’t expect: numbers, but numbers written by us. A little tail, a bigger circle, a stranger stroke… all of these are variables that tell us we’re facing a problem that’s hard even for some of us humans.
But let’s keep going with this premise:
Our handwritten numbers differ a lot from each other, and almost never come out as the exact same stroke.
They’re so different that even if a robot (or a smartphone) memorized every number it had ever seen, our unpredictable and involuntary nature would make us write the same numbers differently next time. And our robot doesn’t like that at all, used as it is to receiving precise, exact, expected requests.
Imagine what all of these options would look like inside a flowchart… No matter how good an algorithm is, there will always be something that doesn’t fit what was planned.
A new number, uglier and weirder, but whose value is obvious to us.
And that’s when we decide to go for another option.
Neural networks
The neuron has been known for a long time. In the late 1800s, Santiago Ramón y Cajal showed that the nervous system is made of individual cells that connect to each other (the neuron doctrine), and he shared the 1906 Nobel Prize for it. We still don’t fully understand how the brain works, but the neuron is one of its most important ingredients.
The main function of neurons is to receive, process, and transmit information through chemical and electrical signals.
They’re like small links, and they act like a mesh. They carry everything from outside your brain, control your body, and let you think. It’s in there that these very words you’re reading are processed and given meaning. And that’s where we’ll take inspiration for the algorithm we’re looking for.
Computer scientists, inspired by how a brain works on the inside, came up with a new way to approach problems. In 1943, Warren McCulloch and Walter Pitts described a simple mathematical model of a neuron (the artificial neuron), and in 1958 Frank Rosenblatt built the perceptron, one of the first networks that could actually learn from examples.
The idea: a network of artificial neurons could store what it learns from data, and get better with it.
A network of neurons like this one has its own structure. Take it, of course, as a computer science example more than a biological one. Artificial neurons are only loosely inspired by real ones.
Going back to our robot with Artificial Intelligence whose job is to recognize our numbers, we can now take the following into account.
Input and output: In this case, the input is the number we want to evaluate. In practice, the image is turned into numbers: a 28×28 pixel image becomes 784 values, one per pixel, from white to black. The output is the robot’s answer: one output neuron for each digit, from 0 to 9.
Connections: And what do these lines and circles mean for the result? Imagine they’re people who break the input into smaller pieces and analyze them.
In other words, they ask questions about the sheet we gave them. Does it have a circle? Maybe it’s an 8 or a 6. Does it have a kind of curvy stroke? Maybe it’s a 3. The neurons share this information with each other (that’s the processing) to find the answer together:
flowchart LR
A[Handwritten digit] --> B{Has a closed loop?}
A --> C{Has a curvy stroke?}
A --> D{Has a straight vertical line?}
B --> E[Maybe 0, 6, 8 or 9]
C --> F[Maybe 2, 3 or 5]
D --> G[Maybe 1, 4 or 7]
E & F & G --> H[Combined answer]
To be honest, this is the hopeful version of the story. The questions that real networks end up “asking” are usually much messier and harder to interpret than “does it have a loop?”. 3Blue1Brown’s video on neural networks shows this beautifully with this exact digit example.
But of course, the auto-magic isn’t us asking those questions. We don’t set up the questions or the scheme each neuron should follow. That’s out of our hands. What we actually do is tell it when it’s right and when it’s wrong.
I gave it a sheet that says 71, but the neural network read it as 97. Well, that’s wrong. And if I give it 12 and it reads 12, I tell it that it’s right. We’re close to understanding how this can work, but we’ve already reached the foundation of Machine Learning.
We clearly don’t learn this way ourselves, but with machines, giving them thousands or millions of examples labeled right or wrong works VERY well.
Now, why is this possible? It must sound strange that telling a machine when something is right or wrong can improve its answers. Remember, we’re dealing with machines, and their answers are typical robot answers.
Something has to change for a machine we’d consider Not Intelligent to become Intelligent.
Weights: how much a neuron trusts another
Neurons are key to solving this. It turns out that between neurons there’s a level of trust, called a weight. Think of it as neuron B paying more attention to neuron A, so when neuron A gives it valuable information, neuron B goes: “I trust you, and I’ll give you much more importance than any other neuron.”
What does more importance mean? It means that if A gives B the number 2, and X also gives B the number 2, B might multiply A’s value by 10 and X’s value by only 2. You can see the preference.
flowchart LR
A["Neuron A (value 2)"] -- "weight ×10 = 20" --> B((Neuron B))
X["Neuron X (value 2)"] -- "weight ×2 = 4" --> B
B --> R["B adds them up: 24"]
This trust is random when we start, set by chance. And here comes the auto-magic: when we tell it “the result the neural network gave me is wrong!”, every weight gets nudged a little in the direction that would have made the answer less wrong. The connections that pushed toward the mistake lose some trust, and the ones that pushed toward the right answer gain some.
Imagine someone you trusted a lot starts lying to you. Then you stop believing them: you trust them less. And so, you no longer rely on them. That same behavior is what we give the neurons, on their own.
The math that figures out how much each weight contributed to the mistake is called backpropagation, popularized in 1986 by David Rumelhart, Geoffrey Hinton, and Ronald Williams. The small nudges themselves are gradient descent. This is the one place where calculus really shows up, and you don’t need it to get the idea.
flowchart LR
A[Show an example] --> B[Network makes a guess]
B --> C{Compare with the right answer}
C -- Wrong --> D[Nudge every weight a little]
D --> A
C -- Right --> E[Small or no change]
E --> A
A neuron can then give you more interesting and valuable things, like the most important strokes on a sheet, and another neuron can give you the circles that best match certain numbers.
Some neurons will be listened to more than others. When they communicate, they’ll carry a strong weight and count for a lot in the final result we want them to produce.
For example, a neuron might infer, based on what it has learned, that when a number has a circular stroke, it could be a six (6), an eight (8), a nine (9), or a zero (0).
And from all that neural intertwining, the network’s output will be a definite result. But from which neuron? Who gives the final answer?
It’s not a vote among all the neurons. Each of the ten output neurons gives a score for its digit, and those scores are turned into probabilities (usually with a function called softmax). Something like:
---
config:
themeVariables:
xyChart:
plotColorPalette: "#2563EB"
---
xychart-beta
title "How sure the network is, for each digit (%)"
x-axis ["0", "1", "2", "3", "4", "5", "6", "7", "8", "9"]
y-axis 0 --> 100
bar [0, 0, 0, 0, 89, 0, 0, 9, 0, 2]
- 89% sure the number is a 4
- 9% sure the number is a 7
- 2% sure the number is a 9
- Almost 0% for every other digit
So the Neural Network answers 4.
And what if it gives a wrong answer? No problem.
We keep training the network with lots of numbers until it can give correct answers almost all the time. The classic dataset for this is MNIST: 70,000 handwritten digits, and well-trained networks get more than 99% of them right.
Up to this point, it’s quite similar to how a child learns, though of course not exactly the same. And this field, within Computer Science, is called Machine Learning: developing techniques that let a computer learn new concepts without us programming how it learns. We only need to create a model, like the Neural Network, which, through probability and lots of computation, lets the Artificial Intelligence learn to recognize numbers.
If you want to play with one yourself, the TensorFlow Playground lets you train a small neural network right in your browser.
The final answer
A neural network is a computational model (a way of doing something) loosely inspired by the nature of neurons, which processes information through them in a way that no hand-written classic algorithm could!
With neural networks we’re finally seeing how small systems working together can achieve big results, just like the world does, or like the tiny connections in our brain do.
YouTube’s recommendations run on deep neural networks, the classic “OK Google” relies on neural networks for speech recognition, Spotify has experimented with them to understand the audio of songs, and the face effects on Instagram use them to track your face. These are just some of the advances made with Neural Networks that you’ve probably already tried.
We started from the foundations of Artificial Intelligence and got to understand (at least superficially, but for real) how a neural network works. We still have a lot to learn.
Throughout history, much of what I’ve explained here started as an “ideal”, a goal that people chased for years, until certain algorithms (like the one for training when there’s an error) finally solved it.
I’ve been writing a few articles about Artificial Intelligence for a while. But this is the first big one I’m publishing. If you have any questions, feel free to reach out 😄 I hope nothing was confusing 😬
Thanks for reading! <3