High speed data Transfer from RPI to Propeller2
chintan_joshi
Posts: 135
in Propeller 2
Hello i am working on data transfer from RPI to Propeller2. Data is something about 100KB to 10 MB
I have tried Serial transfer and SPI. but still its not that fast enough for Big Data.
Is there any other method or proto boards available for P2 to transfer data from RPI to P2?
Like if i can transfer data via USB or LAN or other methods?
Previously i was using LAN connection and TCP/IP sockect programming in c++ to transfer data from RPI to other controller and that was very fast.
I have checked Products on site and found below boards but not able to find any LAN related boards.
- WiFi BLE Click: Wifi and Bluetooth
- Parallax WX ESP8266 WiFi Module: Wifi
- P2 Eval Serial Host Add-on Board : May be transfer data serially
So for Wifi/ LAN connection(1 and 2), which board should i prefer?
For USB, Listed board(3) will provide serial data transfer? is there any fast method available?
Comments
i2s and UARTs can theoretically go up to 250 Mbps in RPi, but there is of course connection problem. I tried to use i2s for communication between 2 RPis, up to 10 Mbps.
The fastest thing RPi has is SMI - Secondary Memory Interface.
https://iosoft.blog/2020/07/16/raspberry-pi-smi/
I still didn't try this yet, but expected transfers are ~40 MB/s where the critical thing is still the connection/cable between Pi and P2
ManAWork had success with 100 Mbit Ethernet using a relatively basic SPI/I2S like interface chip of some sort. From memory, the Ethernet data was passed through unfiltered. Software handled all frame processing. I vaguely remember it was 2-bit parallel data each way, so not pure serial.
It used continuous clocking with preambled data. Maybe that's how Ethernet works at the low level, dunno. It suited the Prop2 at any rate. One of the other persons involved found the preamble allowed enough time for software to time things.
Here it is - https://forums.parallax.com/discussion/174351/rmii-ethernet-interface-driver-software/p1
It's raw RMII - it's very close to raw Ethernet, just with the MLT-3, NRZI, and 4B5B decoding, as well as link speed negotiation and such, already done. In a normal Ethernet interface, you'd have this PHY chip for the Layer 1 parts of Ethernet talking to a MAC chip via RMII to do Layer 2.
No, I still haven't gotten the IP stack I started writing to do more than UDP.
The P2 will have no problem implementing some other protocol on top of Layer 1 Ethernet, but the Pi might not like doing that.
Huh, I didn't have a clue that was anything more than some branded part number. The name, MII, certainly doesn't hint at being associated with Ethernet - https://en.wikipedia.org/wiki/Media-independent_interface
Double huh, MAC is short for media access control. Weirdest naming for network components. To me, media is a storage medium or the stored content, certainly not a network medium.
If you don't have it in place already, I would strongly recommend using CRC checking between the RPi and Propeller on both SEND and RECEIVE responses. Additionally error handling to re transmit data that did not get delivered correctly.
We have a ClearCore (CC) connected to a RPi and communicate with UDP with a dedicated Ethernet (USB<->Eth) and most of the data is sent that way.
We also have a USB (serial) connection from the RPi to the CC so we can remotely re-program the CC through a VPN<-->VNC connection.
The CC itself connects to a COM link that use a custom variation of CANBUS to communicate to over 300 addressable devices.
The main purpose of the RPi is to pass information to and from the CC and to the cloud for remote monitoring as well as some of the heavy lifting processing.
This was also discussed in the thread: https://forums.parallax.com/discussion/174283/using-a-raspberry-pi-as-an-iot-gateway-for-our-p2-projects#latest
What speeds did you achieve, with Serial and SPI, and what speed do you actually need ?
Serial will be simplest, with least wires, but Pi data suggests a HW limit of PeriCLK/16 or 15.625MBd for 250MHz, and there are also relatively small UART FIFOs in Pi, so SW limits can be lower.
P2 should have no problems managing the max Pi speed, and you could look at multiple UARTs in parallel, if those speeds are not enough.
The ultimate limit may be the Pi SW support ?
SPI should go faster, but will need more pins, and one side needs to be defined as a master.
For absolute top end speeds, this post about the pi SMI looks interesting - that's a DMA/FIFO/Clocked parallel bus, says : Transfer data widths are 8, 9, 16 or 18 bits,
https://iosoft.blog/2020/07/16/raspberry-pi-smi/
Disclaimer: I know absolutely nothing about this topic
But I only just discovered an example of ethernet magnetics (well, W5500 lite) coupled with the ESP32 (dirt cheap)
Could this be a ethernet-handling co-processor?
https://cicciocb.com/annex32help/V1.415/#ETHERNET%20Module%20W5500
Craig
Hello Everyone, I am looking for already built module which has library support available for P2. Like if i use arduino/Nucleo 144(STM32) Boards then already some modules and library available to support LAN.
I checked links provided in this thread but seems all point to new development.
To be Honest I am in hurry and can't put efforts to develop something new at this stage due to live project(I would love to but right now i can't). So i asked about supported modules and library for Fast data Transfer.
I prefer LAN/WiFi because i have already developed code to send my data from RPI to any TCP/IP server supported controller.
You still have not stated what speed you actually need ?
If the LAN work is still too beta for you, you could checkout other serial pathway, eg the CH347 or FT2232H perhaps ?
I find this, which says drivers exist on Pi for CH343 series
https://www.waveshare.com/wiki/CH343_USB_UART_Board
and this
https://www.aliexpress.com/item/1005004685449797.html
That CH347 is a Dual UART HS-USB interfaced, and it can also manage SPI ( 36MHz?) and JTAG, and it has 12k Buffers.
so that gives you 2 UARTS of 9MBd each, and I'd suggest hardware handshake initially, until you can determine you do not need it.
and I find also a RPI-HUB-MODULE, which runs FT2232H on Pi, and that can manage 2 x 12Mbd UART or MPSSE 30Mbps
Those shoudl at least be predictable pathways.
Have you considered an XPort module? It's not cheap, but takes care of the TCP/IP stuff and provides a simple serial interface to the controller it's connected to.
Along the same lines, suggested by @JonnyMac, I have one of these but never gotten around to powering it up:
https://tibbo.com/store/modules/em2000.html
I also have a bunch of their Tibbit modules:
https://tibbo.com/store/tps/tibbits.html
Craig
Sorry , speed of 2 Mbps will be fine for me and with SPI/UART i can't even reach to 500Kbps. i am checking interfacing details for suggested boards.
That does really look like a very useful part. The SPI interface should make it especially useful.
Since the Pi3 or PiZero2W with a CPLD can emulate a 68000 processor at more than 100MHz for the PiStorm, there is plenty of bandwidth on the Pi side. The downside is that takes 26 pins, but since both the Pi and the P2 are 3.3V, no level shifting is necessary, it might be capable of getting better performance, and a Pi would make quite a versatile & cheap interface device for the P2, or the other way round if that same interface used for the PiStorm is figured out for the P2. It might even do memory fast enough to run @Wuerfel_21 's MegaYume. Sure would love it if @rogloh picked this ball and ran with it since we have seen the amazing results he has given us with the HyperRAM and PSRAM.
I've got plenty of things to mess with at this time... not sure it's a good thing to add more to the list right now. Maybe on a rainy day I'll have a look though. The SMI bandwidth is reasonable, not sure about handshaking between the two devices however, and the Pi wants to be in control of things I think not the P2.
Might also be worth looking into Cypress fx2lp devices. They're frequently used as cheap logic analyzers, but they do support bidirectional parallel data transfer, at your choice of rate, up to 24MBytes/sec.
Yes, for parallel use, that's an old but good option.
I did recently find a second source part for the FX2LP, tagged CBM9002A which shows up in some AliExpress modules.
That seems quite low, what software are you using, and what size packets ?
I've tested FT2232H on a PC and it can sustain 12Mbd packed duplex , so would expect the newer Pi parts to hit the same numbers.
My CH347 are 'on the plane', and I expect them to sustain 9Mbd (and maybe 12Mbd) when I get them here for tests.
The Pi can do whatever you tell it to do over whatever interface. I see it being a device server, serving up Memory, Network, USB, RTC, microSD over it's GPIO pins however it is instructed by the P2. Case in point. Is the PiStorm a Pi controlling an Amiga, or the other way around? It certainly doesn't look it to the Amiga user. It runs native Amiga binaries and the Pi emulates the processor, memory, hard drive, network or graphics or any of those that you want...with tradeoffs.
Hi @hinv, I was talking about SMI in particular. From what I understand an SMI transfer needs to be started by the Pi which is in control of this bus. Of course one is free to instruct the Pi to do bus transfers from the P2 code via some other bus/signaling but that may add latency for memory uses. I've not looked in detail however and maybe there is a way to DMA trigger this etc, quickly. The Pi can certainly do a lot of fancy things.
RPi 3/4 under Linux does up to 4 Mbps UART "out of the box" with Linux API (no asm hardware config) - there are (was) problems with Linux API/documentation (documentation doesn't match reality for numbering these high speed modes).
My robot works with 4 Mbps UART connection between RPi and P2 using ethernet cable as a link.
Hello @pik33 which LAN module you have used for P2 LAN connectivity? can you please share some details? Have you used any library code for P2 LAN configuration or you have created your own code?
UART(Universal Asynronous Receiver Transmitter)
!=
LAN(Local Area Network)
Though technically a LAN can be implemented upon serial links using UARTS, usually LAN refers to Ethernet, and commonly Ethernet ports (10BaseT, 100BaseT, Gigabit Ethernet, etc.) are called LAN ports. Also typically LAN connections run TCP/IP protocols, though ManAtWork seems to have implemented a Ethernet connection at the MAC (Media Access Control) level without TCP/IP for a fast connection. What pik33 is talking about is a simple serial connection that is commonly used for serial consoles, downloading code into the Propeller, etc.
Where did you find out about how to get those high speeds on the Pi3/4?
I need a high speed UART connection betwen a P2 and a Pi 4 for the robot. RPi has a L515 depth camera connected to it, t preprocess the data from L515 and sends the result to the P2 based robot main board.
The Pi is running Linux (standard RPi OS), so I used standard functions to start a serial port and set its speed.
There are standard serial speed constants in termios.h file
The problem was: i use Pascal for programming, I can call the serial set baud function from it, but I have no such thing as termios.h there, so I had to use a constant. 10017 didn't work, so I had to find the proper one, which, if I can remember now, was 1017 instead of 10017.
There was also a cable problem. The solution was: make a motherboard with 40-pin socket for the RPi. It has also 24V-5V DC-DC converter for powering the Pi and an Ethernet socket to connect it to the main robot control board. Using the ethernet cable allowed to transmit this 4 Mbps without any problems.
I don't have P2 LAN, there was someone else
I only used 4 Mbps UART and tested i2s for communication between 2 RPis. Implementing i2s for a P2 should not be a problem, it is a synchronous serial bus intended for digital audio.
I've achieved >2Mbps between a Pi and P1 without a problem. That was the limit of the P1, but the P2 especially with smartpins should be significantly faster.
Those C constants are all octal numbers, due to the leading zeros. So octal 010017 = decimal 4111.
You might prefer using the mechanism described here to set a custom baud rate.
Now I understand. This was not $1017, this was $100F. Which is octal 10017.
Too bad because I think as an emulation Pi Hat it could be a killer P2 app, and maybe even displace the MiSTer, based on the performance that Ada was able to extract. Also with high bandwidth connection it could be the real-time IO performer that the Pi has needed.