Huge data read rate / speed needed. Suggested implementations? Methods? Examples? Objects?
gis667en11
Posts: 73
Hello,
I've been looking around for discussion on parallel data transmission because I have a project that requires a propeller reading a HUGE amount of data continuously. Serially, I would need to read at approximately 1Mbaud. After seeing that this isn't within the scope of most people's uart implementations, it occurred to me that an 8-bit parallel data bus clocked at, say 230400Hz, would give me an effective throughput of around 1.8MBytes/sec. Is this the implementation you'd use or have seen used for passing a massive amount of data? I've been having trouble finding much more than people bringing it up here and there, and can't find any examples or objects.
I've been looking around for discussion on parallel data transmission because I have a project that requires a propeller reading a HUGE amount of data continuously. Serially, I would need to read at approximately 1Mbaud. After seeing that this isn't within the scope of most people's uart implementations, it occurred to me that an 8-bit parallel data bus clocked at, say 230400Hz, would give me an effective throughput of around 1.8MBytes/sec. Is this the implementation you'd use or have seen used for passing a massive amount of data? I've been having trouble finding much more than people bringing it up here and there, and can't find any examples or objects.
Comments
I'll start by bragging about PropWare's serial (C++ implementation), which is capable of ~2.7 Mbps throughput when transmitting at 4.4 MBaud and 8N1. Receive is slower though, but I think should get you above 1 MBaud throughput still.
It's hard to track down, but there's Spin/PASM object floating around the forum (not in the OBEX last I checked) that gets you 14 MBaud!!! There are lots of others that go over 1 MBaud too, I'm sure someone else will jump in.
Also, I see you tagged UART, I2C, and SPI. The Propeller is no good at slave I2C or SPI, so if you need serial speed, stick with basic UART.
I thought about mentioning Tachyon, but I erased that sentence because I figured you would have seen this and replied before I ever hit "Post"! Glad to see some things are predictable
Parallel is what the FPGA loaders do, they use a FIFO Bridge, like FT240X, plus a CPLD, and they have to move a lot of data...
FTDI claims
Data transfer rates to 1 MByte/s
Above 1Mbd Serial you are going to hit a limited choice - the 480MBd USB FT232H can certainly manage >> 1MBd, but the simpler 12MBd bridge devices can struggle.
If you have the pins, parallel will always be faster.
@David: I see I didn't need to mention Tachyon, you did it for me!
BTW, Tachyon will easily handle constant megabaud streams and write them to SD.
Peter, I tried Tachyon (actually, my first "aha!" moment was holding my newborn daughter in the hospital. I made a few definitions to select how many times an LED flashed on my little breadboard, sitting in my "husband chair" while my girlfriend recovered on the bed). Trust me, from reading your posts through the forum, and the admittedly limited exposure I had, I can see that Tachyon is an extremely powerful environment and method for crafting a program. The interactivity and speed are very impressive. But, being a noobie, I didn't know what I was getting into. I looked into the source code for Tachyon and found that the dictionary is made up of basic functions of PASM instructions. Like, of course now I get that Tachyon is powerful because it's a smart and easy way to make programs that use the native language for the Propeller. But I don't know PASM. So, in order to learn Tachyon, which I would love and plan to do, first I realized I have to learn PASM well enough to figure out what every single Tachyon word does on the bit level. After realizing this, I couldn't convince myself that, with this understanding of PASM, adapting existing PASM objects to suit my needs wouldn't be easier and faster than also learning the second language of Tachyon. This project I'm working on is for my Electrical Engineering bachelor's degree thesis, which is holding me up on finally getting my degree. With the birth of my daughter, the Mrs. is understandably cracking the whip / turning up the heat on me completing this. Did I bite off an absurdly large amount for a stupid thesis project/report? Absolutely. But, this being the circumstance, I have concerns that the time needed to sift through the Tachyon source code to understand what all of the dictionary entries DO, and how to actually use them together effectively, isn't worth it just yet.
So, for now I'm just going with Spin and PASM. If only because I already have the textbook and, by now, a pretty good grip on how it works. Thanks for the advice, David, I'll keep digging and won't give up on serial data just yet.
P.S. I'm now looking into streaming data into the Prop from a Raspberry Pi B+. Features I decided I'd REALLY like to have include Ethernet, audio output, and USB host. Plus the raspi can read from a USB flash drive which is much faster than SD.
That sounds like a really good idea. Though a Propeller *can* do all of that, I think it's much better to off-load the heavy stuff (since bigger chips have things like Ethernet and USB in hardware, not software) and use the Propeller for what it is better at: real time.
Also, here's a thread which discusses some very cool serial objects: http://forums.parallax.com/discussion/156376/serial-i-o-throughput-a-comparison
I must admit that although this is a way of understanding the operation of each word it certainly isn't the way I would "learn" a language. I like to grok the "philosophy" of an approach or language, the rest is coding. With Forth it was certainly different for me but I loved the interactivity and extensibility of the language and just how easy it made it to gain access to the bare metal even if sometimes it meant creating your own extensions. I'm basically a hardware guy so in a way Forth is my means of connecting to the hardware, even if it means "reverse" notation, but then again that is simply a perspective and label applied by someone from the "reverse" of that.
Or PropBASIC
Ha, I missed that quote last night. In C/C++, I reach 4.4 MBaud transmit without using a driver at all - only a single cog for the whole application. There's a little bit of inline assembly to make it happen.
My transmit runs bit-bashed out of the main Tachyon cog as my rationale is that I'd rather bit-bash it directly to the transmit port than waste time checking buffer pointers and writing to a buffer only to dedicate another cog to unravelling it all again later. That's why it's better to run at a higher baud-rate as very little time is wasted in transmitting, it is actually faster.
My thoughts exactly
Receives a few commands over the terminal TX/RX to configure the output type.
Start the cogs with the correct output protocol object, passing the pins and buffer start address
Reads data into two buffers (a "back and forth", or whatever it's called) so one is being read/used by the output cogs while the other is being written to by the RX cog.
Each output send up to 512 bytes every 30 milliseconds.
So, in my example, cogs 4 and 5 would start up the shift register object program, being passed pins 10-13 and 14-17, and being pointed to the first and second buffer 512 byte chunk start addresses. Cog 6 would start the RS485 output object program being passed pins 18-21, and Cog 7 would start the i2c output object, passed pins 22-25.
This way, the first 512bytes buffer chunk, updated at 33.33Hz, could be sent over any protocol by simply moving that hardware input device to pins 10-13 and telling the prop what output 1 protocol should be.
What baud rate are you running the RS485 at?