Killing the LSTM: Why DANI is Pivoting to a Spiking Neural Network

After some extensive research and a few hard encounters with hardware reality, I’ve made a major architectural decision regarding DANI's brain: I am completely ripping out the LSTM neural network and replacing it with a Recurrent Spiking Neural Network (RSNN).

Given my recent enthusiasm for the LSTM architecture, this will probably come as a surprise to anyone following along. However, I have my reasons, and as usual, they come down to math, metal, and thermal throttling.

What is wrong with LSTMs and the Traditional ANN?

Traditional Artificial Neural Networks (ANNs) are fantastic at what they were designed for: probability scoring. They allow us to simulate decision-making using massive matrix calculations mixed with a healthy dose of calculus.

But beneath the hood, they are brute-force mechanisms.

When a traditional ANN is fed data, that data cascades through every single neuron in the network. Even when a neuron has a zero value, the CPU still dutifully performs the mathematical operation of multiplying by zero. If a network has a shape of 10 inputs, 10 outputs, and 5 layers of 12 neurons each, that is 816 floating-point calculations just for one single feed-forward pass. If you start adding in simple recurrence, we are immediately up to 1,488 calculations.

Now, scale this up to something that would actually be useful for DANI. We need at least 100 inputs and about 64 outputs. With hidden layers of around 125 neurons each, we jump to 150,375 calculations per pass. To achieve an LSTM architecture, we can estimate the computation to be roughly four times that: over 600,000 calculations per tick.

As you can see, this grows exponentially. More importantly, it keeps the CPU running permanently hot. Don’t forget, DANI is powered by a Raspberry Pi 5. There is no GPU offloading here. I did manage to get DANI’s LSTM operating at 10Hz with 3.5 million parameters, but it required the Pi 5 to do absolutely nothing else, running flat-out at 100% utilization. DANI's brain was essentially doubling as a space heater, even when he was just "dreaming."

Once you add my simulated hormone system into the mix, the compute overhead goes up even further. I was basically following the same brute-force model that the big AI companies use, just without the luxury of a multi-million dollar server farm.

And then there is the training. To train an ANN of this style requires layer-by-layer calculus (backpropagation). This is heavy, blocking work. It takes a considerable amount of time, meaning we simply cannot train the network in real-time on anything substantial.

The Elegance of the Recurrent Spiking Neural Network

The architecture of an RSNN is topographically similar to other neural networks, but the execution is fundamentally different. It mimics biological reality much closer.

Instead of passing continuous floating-point numbers, each neuron has a membrane potential. As it receives signals from upstream neurons, that potential increases (or decreases, if the synapse is inhibitory). The neuron does absolutely nothing until that potential reaches a specific threshold. Once it hits the limit, it fires—or "spikes." And it spikes at a binary full power. The neuron then resets, either dropping to zero or subtracting the threshold from its current potential.

If a neuron doesn’t reach the threshold, it doesn’t spike. Period.

This means we only need to calculate the pathways for neurons that actually receive a spike. The computational reduction is staggering. The general consensus in the field is that for any given signal, only 5% to 10% of a spiking network is active, compared to the 100% density of a matrix-style network.

Training also becomes radically simpler. We only need to reinforce or weaken the synapses of the pathways that actually fired (a process akin to Spike-Timing-Dependent Plasticity). We apply a localized effector to those specific connections—no massive, network-wide calculus required.

But here is the real kicker for a systems engineer: an RSNN allows us to completely ditch floating-point math. By using integer mathematics and clever bit-shift operations (which are essentially free in terms of CPU cycles), we can completely bypass the messy, cycle-heavy floating-point multiplications.

How does this help DANI?

Obviously, DANI will not have to run so hot. If he has a quiet mind, the network will physically go quiet. When DANI rests, he will literally be saving power. By utilizing temporal calculations, we can also train him in real-time with near-zero impact on the system.

Furthermore, DANI’s brain becomes an event-driven architecture rather than a strict polling loop. Instead of processing rigidly on a timer, he will think and react in time with the world around him.

I will still be using the hormonal system I designed, but the key difference is how it integrates. The effective values of the hormones will now directly modulate the firing thresholds of the neurons. This allows his "moods" to physically alter the state of his brain and affect his behavior, even if his underlying memories remain unchanged.

I suppose I had better be nice to him.

Rethinking Asimov’s Laws

This pivot finally allows me to revisit a question I was pondering a while back: How do we encode Asimov’s Laws into an AI?

The answer is: we don’t. It’s as simple as that. We don’t hardcode the laws; we teach them.

By running an RSNN that learns in real-time through stimulus and reinforcement, the laws can become an emergent part of his personality. He will understand that hurting people is bad and that following human orders brings a reward, not because a line of code forces him to, but because his synapses have shaped themselves around those experiences.

So, I guess I really am going to have to be a parent to a robot.