Multi-Propeller System
HalcyonDaze
Posts: 17
I'm new to the propeller chip, but fascinated by the possibilities of the multi-core design. Ive worked with the Basic Stamp a number of times, but beyond that my design experience is all with larger scale industrial systems (Allen Bradley, Siemens, etc.) so I'm a novice when it comes to design at this level. I have a basic understanding of Assembly and of object based languages (C++, specifically), but have no actual experience with PASM or Spin.
I have a concept for a distributed control network platform that I would like to develop, and I think the Propeller would be uniquely equipped to accomplish this. From what I can tell there has been work in this direction (various supercomputer/chip array projects), though it appears somewhat fragmented (as all community projects are). The documentation in the Object Exchange is somewhat thin, and most of the forum threads I see referenced in various places have been lost/deleted.
My ideal system would basically mimic the internal chip Hub with a de-centralized network loop. I anticipate devoting one Cog on each Chip to this communication. Physical location is going to be an issue, so communication would need to be done over a serial connection (or some other protocol if if necessary). If needed each chip would get an external eeprom or sdram chip, etc to house the overall program, which will cascade through the network at bootup. All I/O pins would be a shared resource, addressed by chip and pin. System Memory would go in a common stack, to be accessed at need by any active cog. In the event of communication loss to any chip, the others will adapt to compensate by flagging External processes (requiring Pins) as on hold, and assigning internal tasks to the next available cogs.
So before I go out to re-invent the wheel, what have you, the experienced, found to be the extents and/or limits of propeller chip cooperation? What is out there, what hasn't been made yet, and what is impossible with current technology? Specific things Im looking for guidance on are:
-> Clock Synchronization
-> I/O pin addressing
-> multi-chip RAM Management
-> Communication Methodology
-> Trans-Chip registry handling
-> Program unity/integrity (cascade boot EEPROM or some such)
-> Communication Loss handling
I know what Im talking about is a large undertaking; Im just looking for someone to point me in the right direction.
Thoughts?
I have a concept for a distributed control network platform that I would like to develop, and I think the Propeller would be uniquely equipped to accomplish this. From what I can tell there has been work in this direction (various supercomputer/chip array projects), though it appears somewhat fragmented (as all community projects are). The documentation in the Object Exchange is somewhat thin, and most of the forum threads I see referenced in various places have been lost/deleted.
My ideal system would basically mimic the internal chip Hub with a de-centralized network loop. I anticipate devoting one Cog on each Chip to this communication. Physical location is going to be an issue, so communication would need to be done over a serial connection (or some other protocol if if necessary). If needed each chip would get an external eeprom or sdram chip, etc to house the overall program, which will cascade through the network at bootup. All I/O pins would be a shared resource, addressed by chip and pin. System Memory would go in a common stack, to be accessed at need by any active cog. In the event of communication loss to any chip, the others will adapt to compensate by flagging External processes (requiring Pins) as on hold, and assigning internal tasks to the next available cogs.
So before I go out to re-invent the wheel, what have you, the experienced, found to be the extents and/or limits of propeller chip cooperation? What is out there, what hasn't been made yet, and what is impossible with current technology? Specific things Im looking for guidance on are:
-> Clock Synchronization
-> I/O pin addressing
-> multi-chip RAM Management
-> Communication Methodology
-> Trans-Chip registry handling
-> Program unity/integrity (cascade boot EEPROM or some such)
-> Communication Loss handling
I know what Im talking about is a large undertaking; Im just looking for someone to point me in the right direction.
Thoughts?
Comments
One object I've found very helpful is Tim Moore's multi port serial object "pcFullDuplexSerial4FC." I've have modified Tim's code several times. I've posted much of my code in this thread.
One variation was to increase the rx buffer from 64 bytes to 128 bytes. This modification only increased the program size by a few longs.
I also have versions with several different buffer sizes.
One modification I find very useful is the addition of checking for end of line flags on each rx line. Instead of constantly monitoring for any new characters, I just check a counter that keeps track of the number of end of line flags detected. If this number increases, I know I have new message to process. (The version with end of line flags is attached to post #9.)
I've also adapted Tim's object for use with XBee transceivers. One version (attached to post #12) can automatically acknowledge a received message.
Hopefully one of these modified versions will be helpful to you.
Duane
http://code.google.com/p/propforth/wiki/HighSpeedSerialInterpropCommunications
The documentation is out of date due to lack of user interest, but if this is along the lines f what you are looking for,
I can update it.
Basically, the low level support is present for the items you mentioned, except for external RAM since this has not been needed.
BUT you would have to do the higher level implementation, or at least specifically state what you want it to do.
As to inter-Prop communication, the prop is such an open design that there are an extreme number of possibilities.
How many pins can you spare per Prop for the Inter-Prop comm?
This will determine if you would be best with a serial or parallel setup.
What type of speed is required?
Serial Vs Parallel again.
How much time can you spend selecting the Prop to communicate with?
Determine if you can use the same lines for selection and communication.
And many many more things to consider.
In sum the method chosen is up to your requirements.
In looking a various communication protocols there seems to be two flavors. One method is to have a message size field early in a transmission. Dynamixel AX-12 robot controllers use this type of protocol. Another method is have and end of message flag (I generally use this protocol).
A big draw back with using an end of message flag is you need to make sure your data never includes the flag.
For example, I commonly use a carriage return (ASCII 13) as an end of message flag. In order to make sure my data never includes the value "13" I convert the data to ASCII characters of the hexadecimal values. I also need to convert the ASCII characters back to the original numbers once they reach their final destination.
Continuing with this example, if I want to send the number 13 as data I would send ASCII 48 "0" and ASCII 68 "D". Of course this method uses two bytes to send one byte of information.
Using some sort of message length indicator rather than an end of message flag is starting to look like a good idea. I'm still resistant to make the change because of how many devices I currently have using the end of message protocol.
Another part of my system is to have each Propeller board assigned a letter. When a message is sent I use a couple of characters to indicate the beginning of a new message followed by the destination board character and then the sending boards character. Some of my boards can not communicate directly with each other so the message may make several hops to its final destination.
The easiest protocol to use is probably I2C. I use a combination of I2C + 8 data bits P0-7 on one project (I2C on P8/9). The advantage of that other than transfer speed once I2C is done packet termination is easy because the receiving side can stop saving bytes when bit P8 goes low.
If you're interested in Forth, that might provide a faster development path since it's supposedly already done for you. I have no idea what they do, but it is pushed a lot here, so there must be something useful in it.
To answer some of the questions:
This is intended to be a base platform for a number of projects Ive been mulling, so the quantitative requirements are somewhat fluid. In many cases its a matter of seeing how much can get, then building the rest to that. That is much of why I want this system to be as dynamically scalable and configurable as possible. I also want all the hardware to be a uniform design, with Available I/O pins going to a standard header for additional hardware to attach.
Speed requirements:
Optimized as best I can. It is intended as a control system for robotic applications, so the speed requirements will be dictated by the sensor signals more than anything else. That being said the philosophy of the program design will be to compartmentalize the physical operations as much as possible, so as to take full advantage of the multiprocessing capabilities. Eventually it would be nice to be able to communicate at comparable speeds to a single chip, as I have one hair brained idea for a dynamic pin routing system, meaning that any pin and/or processor can step in and take over the hardware control, should a pin or prop fail; basically to create But for now I could easily settle with a system that would differentiate between Hard processes that will require specific pins and wiring and such, and Floating Processes that handle internal tasks like data interpretation, calculations, etc, and can be reasonably done by any available cog on any chip. For those types the only thing going over the communication system would be the results.
Pin Requirements:
Optimize again. Its going to be a primarily I/O system, so the more the better. In a perfect world I could handle all the inter-chip comm during normal operation using just the 4 boot pins. That pretty much assumes Im not adding any external RAM though.
Communication:
I want to avoid a Master/Slave system as much as possible. The hardware must all be uniform, and Id prefer it if there is no actual centralized leadership to provide a full system failure point. I figure I use that object i found in the exchange that generates a unique ID number for the prop on first boot, then line themselves up based on that value. The eventual thought is to replace the communication wires with a drop-in blue tooth transmitter (this or similar: http://www.sparkfun.com/products/8497) so that physical location will no longer be a restraint (to the transmission range of course). So I need the system to be able to loose communication to any given prop without a complete system failure, ie. no Master or a dynamically re-assignable Master. On your recommendations I looked through the propforth stuff a little, but Im not sure I can implement it without a hardwired Master.
The basic idea I envision for this stage is some form a ring topology, where each prop devotes a cog to constant communication updates. All the Global information (as opposed to private or public) would be "published" to this Communication Cog for the others to use as needed. Then the communication just needs to update the next Prop in line with the most recent version, and changes will cascade around the ring. There will need to be some checking done to prevent error cascade. Then, so long as every prop is running the same program, actual coordination would only need to be handled through a flag registry to actually assign the cogs their various tasks. There will also have to be a pass-off operation where something is stored in the global dropbox long enough for target prop to pick it up and move it into another memory location; this will be the basis for the RAM management, basically using another prop and/or cog as spare storage (or sending it to that prop's external memory, if available), but the specific distribution scheme still needs to be ironed out. I also haven't worked out what would happen if the system lost the chip your information was stored on; some form of redundant storage I suppose... I wonder if you can build a miniature RAID array out of prop storage?
I have no idea at the momemt! I first looked at doing something with this mid 2009
I'm gonna sit down tonight and work out the logic, then I'll have a better idea of what's involved. I'll keep everyone informed of my progress.