Huge data read rate / speed needed. Suggested implementations? Methods? Examples? Objects?

gis667en11gis667en11 Posts: 73
edited 2016-01-02 - 02:28:32 in Learn with BlocklyProp
Hello,

I've been looking around for discussion on parallel data transmission because I have a project that requires a propeller reading a HUGE amount of data continuously. Serially, I would need to read at approximately 1Mbaud. After seeing that this isn't within the scope of most people's uart implementations, it occurred to me that an 8-bit parallel data bus clocked at, say 230400Hz, would give me an effective throughput of around 1.8MBytes/sec. Is this the implementation you'd use or have seen used for passing a massive amount of data? I've been having trouble finding much more than people bringing it up here and there, and can't find any examples or objects.

Comments

  • I routinely do 2M baud full throughout so you don't need parallel. It's a mistake to look at the fds object and think that is the limit, it serves a purpose but there are many ways. What language and what are you going to do with the data?
  • This surprises me a lot, that you're having trouble finding 1 MBaud serial implementations. There are a few in each language actually. Which language are you looking for? Do you need to transmit, receive, or both? And what's on the other end of the communication line?

    I'll start by bragging about PropWare's serial (C++ implementation), which is capable of ~2.7 Mbps throughput when transmitting at 4.4 MBaud and 8N1. Receive is slower though, but I think should get you above 1 MBaud throughput still.

    It's hard to track down, but there's Spin/PASM object floating around the forum (not in the OBEX last I checked) that gets you 14 MBaud!!! There are lots of others that go over 1 MBaud too, I'm sure someone else will jump in.

    Also, I see you tagged UART, I2C, and SPI. The Propeller is no good at slave I2C or SPI, so if you need serial speed, stick with basic UART.
  • I routinely do 2M baud full throughout so you don't need parallel. It's a mistake to look at the fds object and think that is the limit, it serves a purpose but there are many ways. What language and what are you going to do with the data?

    I thought about mentioning Tachyon, but I erased that sentence because I figured you would have seen this and replied before I ever hit "Post"! Glad to see some things are predictable :D
  • jmgjmg Posts: 14,424
    gis667en11 wrote: »
    ...Serially, I would need to read at approximately 1Mbaud. After seeing that this isn't within the scope of most people's uart implementations, it occurred to me that an 8-bit parallel data bus clocked at, say 230400Hz, would give me an effective throughput of around 1.8MBytes/sec. Is this the implementation you'd use or have seen used for passing a massive amount of data?

    Parallel is what the FPGA loaders do, they use a FIFO Bridge, like FT240X, plus a CPLD, and they have to move a lot of data...

    FTDI claims
    Data transfer rates to 1 MByte/s

    Above 1Mbd Serial you are going to hit a limited choice - the 480MBd USB FT232H can certainly manage >> 1MBd, but the simpler 12MBd bridge devices can struggle.

    If you have the pins, parallel will always be faster.


  • FT232R is good to about 3Mbaud but that isn't relevant to your requirement as you have data coming from somewhere and going somewhere, but where we do not know although you have enquired about SD speeds before.

    @David: I see I didn't need to mention Tachyon, you did it for me! :)
    BTW, Tachyon will easily handle constant megabaud streams and write them to SD.
  • gis667en11gis667en11 Posts: 73
    edited 2016-01-02 - 06:29:47
    Thanks everyone, I've been doing a lot of research into using the Propeller (I'm a new prop user, long time Arduino fan). From going through Programming the Propeller With Spin by Harprit Singh Sandhu, the Experiments for the Propeller Quickstart by Jon Titus, and using various Obex files as case studies to see how things are done by veterans. One of the things that convinced me that, at least initially, Spin and PASM were the way to go was the fact that PASM is the only language that can be run in a cog, meaning I would HAVE to know PASM to write a driver to read and write high data rate output protocols.

    Peter, I tried Tachyon (actually, my first "aha!" moment was holding my newborn daughter in the hospital. I made a few definitions to select how many times an LED flashed on my little breadboard, sitting in my "husband chair" while my girlfriend recovered on the bed). Trust me, from reading your posts through the forum, and the admittedly limited exposure I had, I can see that Tachyon is an extremely powerful environment and method for crafting a program. The interactivity and speed are very impressive. But, being a noobie, I didn't know what I was getting into. I looked into the source code for Tachyon and found that the dictionary is made up of basic functions of PASM instructions. Like, of course now I get that Tachyon is powerful because it's a smart and easy way to make programs that use the native language for the Propeller. But I don't know PASM. So, in order to learn Tachyon, which I would love and plan to do, first I realized I have to learn PASM well enough to figure out what every single Tachyon word does on the bit level. After realizing this, I couldn't convince myself that, with this understanding of PASM, adapting existing PASM objects to suit my needs wouldn't be easier and faster than also learning the second language of Tachyon. This project I'm working on is for my Electrical Engineering bachelor's degree thesis, which is holding me up on finally getting my degree. With the birth of my daughter, the Mrs. is understandably cracking the whip / turning up the heat on me completing this. Did I bite off an absurdly large amount for a stupid thesis project/report? Absolutely. But, this being the circumstance, I have concerns that the time needed to sift through the Tachyon source code to understand what all of the dictionary entries DO, and how to actually use them together effectively, isn't worth it just yet.

    So, for now I'm just going with Spin and PASM. If only because I already have the textbook and, by now, a pretty good grip on how it works. Thanks for the advice, David, I'll keep digging and won't give up on serial data just yet.

    P.S. I'm now looking into streaming data into the Prop from a Raspberry Pi B+. Features I decided I'd REALLY like to have include Ethernet, audio output, and USB host. Plus the raspi can read from a USB flash drive which is much faster than SD.
  • gis667en11 wrote: »
    I'm now looking into streaming data into the Prop from a Raspberry Pi B+. Features I decided I'd REALLY like to have include Ethernet, audio output, and USB host. Plus the raspi can read from a USB flash drive which is much faster than SD.

    That sounds like a really good idea. Though a Propeller *can* do all of that, I think it's much better to off-load the heavy stuff (since bigger chips have things like Ethernet and USB in hardware, not software) and use the Propeller for what it is better at: real time.

    Also, here's a thread which discusses some very cool serial objects: http://forums.parallax.com/discussion/156376/serial-i-o-throughput-a-comparison
  • Peter JakackiPeter Jakacki Posts: 9,380
    edited 2016-01-02 - 13:26:38
    gis667en11 wrote: »
    So, in order to learn Tachyon, which I would love and plan to do, first I realized I have to learn PASM well enough to figure out what every single Tachyon word does on the bit level..

    I must admit that although this is a way of understanding the operation of each word it certainly isn't the way I would "learn" a language. I like to grok the "philosophy" of an approach or language, the rest is coding. With Forth it was certainly different for me but I loved the interactivity and extensibility of the language and just how easy it made it to gain access to the bare metal even if sometimes it meant creating your own extensions. I'm basically a hardware guy so in a way Forth is my means of connecting to the hardware, even if it means "reverse" notation, but then again that is simply a perspective and label applied by someone from the "reverse" of that.

  • gis667en11 wrote: »
    Spin and PASM were the way to go was the fact that PASM is the only language that can be run in a cog, meaning I would HAVE to know PASM to write a driver to read and write high data rate output protocols.

    Or PropBASIC

  • Mickster wrote: »
    gis667en11 wrote: »
    Spin and PASM were the way to go was the fact that PASM is the only language that can be run in a cog, meaning I would HAVE to know PASM to write a driver to read and write high data rate output protocols.

    Or PropBASIC

    Ha, I missed that quote last night. In C/C++, I reach 4.4 MBaud transmit without using a driver at all - only a single cog for the whole application. There's a little bit of inline assembly to make it happen.
  • David, the transmit part is always easy but a sustained 100% throughput receive is that little bit harder due to hub access delays. If you are not worried about matching baud-rates you can bit-bash as fast as you want as long as the receiver is locked into the same clock of course as in the case of Beau's high-speed inter-Prop comms.

    My transmit runs bit-bashed out of the main Tachyon cog as my rationale is that I'd rather bit-bash it directly to the transmit port than waste time checking buffer pointers and writing to a buffer only to dedicate another cog to unravelling it all again later. That's why it's better to run at a higher baud-rate as very little time is wasted in transmitting, it is actually faster.
  • My transmit runs bit-bashed out of the main Tachyon cog as my rationale is that I'd rather bit-bash it directly to the transmit port than waste time checking buffer pointers and writing to a buffer only to dedicate another cog to unravelling it all again later. That's why it's better to run at a higher baud-rate as very little time is wasted in transmitting, it is actually faster.

    My thoughts exactly :). And if the user needs buffered output, it's not hard to setup a second cog reading from a queue.
  • gis667en11gis667en11 Posts: 73
    edited 2016-01-02 - 20:53:34
    Peter, the Raspberry Pi couldn't use the same clock as the propeller. So, beau's can't work for me then? Also, I realized I never answered your question of what I plan to do with the data. I need to send 2-3kbytes to hub memory once every 30 milliseconds. My intention is for 4, 5, or 6 cogs to operate as flexible output modules capable of delivering the data in selectable output protocols. So, say outputs 1 and 2 should go to shift registers and output 3 should go to an RS485 bus and output 4 should go to an i2c device. I plan to write the program such that it:

    Receives a few commands over the terminal TX/RX to configure the output type.

    Start the cogs with the correct output protocol object, passing the pins and buffer start address

    Reads data into two buffers (a "back and forth", or whatever it's called) so one is being read/used by the output cogs while the other is being written to by the RX cog.

    Each output send up to 512 bytes every 30 milliseconds.

    So, in my example, cogs 4 and 5 would start up the shift register object program, being passed pins 10-13 and 14-17, and being pointed to the first and second buffer 512 byte chunk start addresses. Cog 6 would start the RS485 output object program being passed pins 18-21, and Cog 7 would start the i2c output object, passed pins 22-25.

    This way, the first 512bytes buffer chunk, updated at 33.33Hz, could be sent over any protocol by simply moving that hardware input device to pins 10-13 and telling the prop what output 1 protocol should be.
  • That isn't all that fast at all then. You would find the SPIxx instructions in Tachyon very useful for the shift registers, there is no need for an "object program", you just use SPIRD and SPIWR directly in a Tachyon cog which take only a couple of microseconds to execute rather mucking about passing commands to a cog and waiting for a response.

    What baud rate are you running the RS485 at?
  • 250k, I'm outputting DMX512. I've been digging through the object jm_dmx_ts (Thanks for the code, @JonnyMac ) but I've been struggling to get it to work. In reality, I won't be using shift registers, I was just using that as an example. I'm actually getting pretty comfortable with PASM; I took microcomputers 1 and 2 at uni and it covered quite a bit of assembly programming, monitoring and predicting bit changes in memory, and such. PASM really isn't THAT different from the PIC implementation. It's all bits, after all. Anyways, Peter I will give Tachyon another look. When I finally get around to plunging into Tachyon, whether that happens tomorrow or in 6 months, I plan to document my learning experiments for case studies.
Sign In or Register to comment.