Simulating neurons with an array of props.
kwinn
Posts: 8,697
My original thought was to have the neurons send messages to the individual neurons they are connected to, but perhaps it would be better to have the neurons listen for messages from neurons they are connected to. This may speed things up.
Comments
Guess we need to start looking for an appropriate camera for image capture if recognition is one of the goals of the Artificial Neural Network "ANN".
I agree with listening for messages. Imagine a COG listening to pre-determined HUB locations for a mailbox flag.
In the simplest case, a mailbox could be stuffed with information received from a "transport" COG. Neurons can send information to other Neurons via the transport COG. Neural network connections themselves are a bit "fuzzy" to me right now and deserve more reading.
The details of a Neuron would be determined later, but I expect that "K" Neural processing "jmpret threads" could be active in a COG.
People always fret about floating point with a Neuron, but I contend all that is really necessary is signed arithmetic and Propeller is pretty good with that using 32 bit longs.
If a COG based Neural processor can be developed, the actual physical transport mechanism can be whatever the user chooses. The only requirement for interfacing with a Neuron is the mailbox.
Since we were discussing an array of Propellers to do this it seemed like the natural choice.
Agreed, but we also need audio if we are to process the usual inputs to any intelligence.
On further thought it seemed like a less bandwidth intensive approach than sending messages to every connected neuron. It also seemed to be closer to the way our nervous system works.
True. A message could be broadcast and all the neurons could process it as required without regard as to how it was received. What the message is and how the neurons process it is a bit fuzzy to me as well. As Steve Ciarcia at Circuit Cellar said " my low level language is solder " and so is mine.
Some form of LMM and use of hub memory should allow a cog to simulate multiple neurons.
I don't really see floating point or even signed arithmetic as being required for this, and the beauty of the prop is that messages are not limited to any specific bit length. You may have to write a driver but you can choose whatever bit length suits your requirement.
That is why I proposed a cube or hypercube connection scheme. It allows for a low latency communications scheme while allowing a very flexible neuron interconnect method.
One project is building interconnect software to allow multiple propeller chips to form a simple network and cooperate on a task. If you solve this problem correctly it will be useful for more than just neural networks. This problem has been solved several times over the past 30 years, so it would be best to copy a known good approach. Obviously in a much reduced and less general form because of the limited resources.
The other project is simulating a neural network on a propeller or propellers using this interconnect software. But it might be worth just simulating a small network on a single propeller first. With 32K of RAM you should be able to have a sparse matrix of maybe 100 neurons to write all the math routines required.
If you allow multi-cast I would be curious how you'll handle bus collisions. In the robotics forum I was suggesting copying the concepts used with TCP/IP in a much reduced form. TCP allows nodes to broadcast on the bus and if they collide they rebroadcast after a back off algorithm.
Absolutely. The first challenge would be to settle on an interconnect scheme and a communications protocol.
Good point. I threw out the cube/hypercube idea for it's simplicity, ease of expansion, simple routing, and high speed. Other connection schemes should be discussed before a decision is made.
I was thinking that two of the TetraProp boards would be good for this. I was also considering laying out a board with a 3x3 array of props with one prop as a master that provides the clock and software download to the other 8 props. A stack of 3 such boards should be a pretty good start
My thought was also to use a reduced form of TCP/IP with the prop address being related to it's physical location in the array. Possibly a 12 bit address with 4 bits each for X, Y, and Z coordinates in the array. Individual neurons in each prop would be appended as IP addresses are done now. So it would be "PropCoordinate.NeuronNumber".
Again, alternate methods should be discussed before a decision is made. Perhaps some form of time slicing similar to hub access on the propeller could be combined with the CSMA/CD protocol used by TCP/IP. With all the props in the array in such close proximity they could share a common clock of some kind to coordinate things.
- Create "threaded" Neurons.
- Create a protocol independent communications layer.
- Connect the communications layer to some transport layer.
- Connect any hardware to support the transport layer.
Some parallelization of effort is possible of course.The type of hardware connections used is less important than the methodology for using it. Threaded Neurons would have training/operational modes and can read/write intra-chip mailbox pointers. Protocol independent communications layer will encode/decode buffers with inter-chip end-points. Transport layer would be encapsulating data on TCP/UDP or some other underlying protocol.
Additional topics for research:
Here's a source code repository: http://tralvex.com/pub/nap/zip/
I like nasanets.zip - except it has weird C BEGIN/END body macros
Same site, different code in C: http://tralvex.com/pub/nap/nn-src/
Wikipedia has lots of info. Some algorithms are buried there.
http://en.wikipedia.org/wiki/Artificial_neural_network
http://en.wikipedia.org/wiki/Artificial_neuron
http://en.wikipedia.org/wiki/Perceptron
http://en.wikipedia.org/wiki/Feedforward_neural_network#Multi-layer_perceptron
http://en.wikipedia.org/wiki/Computational_neuroscience
Here is my list of PDP links.
http://forums.parallax.com/showthread.php?124495-Fill-the-Big-Brain&p=1006938&viewfull=1#post1006938
Some spin character recognition code poorly documented.
http://forums.parallax.com/attachment.php?attachmentid=52090&d=1202847726
ftp://ftp.sas.com/pub/neural/FAQ.html
Does this mean there is a common bus that handles all the props communication?
I had the idea that biological neurons just have direct connections between neurons, and "use" determines the amount of traffic on a given channel.
Obviously, I have a lot of reading to catch up....
A "bus architecture" or some other connection interface will be required for multiple propeller connections. There are different ways to do it. Hopefully the code can be designed so that it can be used on any hardware connections.
If so, how frequently would the memory need to be written too? Would flash work as external memory or would one need to use SRAM?
A centralized SDCard or wear-leveled flash store may be useful for saving neuron "weights" ... a per-propeller flash like what SpinSocket-Flash offers could be used for saving weights, but it's probably not necessary. SRAM would be required for normal operations.
Since you are interested enough in the thread, it would be really nice for you to provide a couple of examples of pairing applications and techniques to further the cause.
http://www.cs.york.ac.uk/auramol/
I was thinking of doing a PhD with him when I worked for British Aerospace, and visited him once. He still had some transputer hardware I'd designed some years before.
The Propeller isn't suitable for any of those applications I've mentioned, and I can't think of one to which it is particularly well-suited. The architecture is all wrong, it hasn't got enough on-chip memory, external memory will be too slow, and the comms channels will slow everything down still more. You could try implementing something simple, and see how you get on, if you don't believe me.
That's the direction this is headed. I have no reason to question your judgement. Not trying is not knowing. One way or another something interesting will come out of it.
If you want to help, that's great. Otherwise, you should find something else to do with your time.
Maybe you would like to contribute to the infrastructure? It would be a great way to lift you Propeller standing and demonstrate that you're not just some other MCU suggesting distractor.
The bus architecture is still under discussion. What I proposed was multiple buses. One for each row and column of props on a board/board plane, and one for each stacked row of props. Each prop would then have a separate connection to one x, one y, and one z bus. This makes for simple fast routing, low bus contention, and redundant communications paths. Neuron connections would be virtual connections using these buses.
I also suggested that neurons "listen" for signals from neurons they have connections to rather than have them send multiple messages or messages with multiple recipients. Seems like it would keep message traffic down.
The software infrastructure should support any physical mechanism if possible to allow optimizing for *any* connection. If you want image recognition for example, the software should be able to configure the network for that purpose, but it should also be able to configure a network for aural processing too if there are enough resources available.
Physical mechanisms for optimizing any connection scheme would range from extremely difficult to impossible. For this particular project we should settle for a software connection mechanism that optimizes use of the chosen hardware connection scheme for the purpose intended.
In other words instead of trying to come up with an optimum connection scheme for current neuron software and attempting to fit the prop to that, lets come up with an optimum connection scheme for an array of propellers and write neuron software that takes advantage of the prop's strengths.
I wonder if the ROM math tables would be useful for generating a nice non-linear response...
Is this what we're talking about then, or is the quote talking about an option chosen for a particular research case?
Would a simple application for neural net be something like coordinating legs on a hexapod robot, so the feet all touch the ground at the same time on uneven surfaces?