Using a Raspberry Pi as an IoT gateway for our P2 Projects

Stephen Moraco · 2022-01-19 23:21

I introduced a new project during our Live Forum today.

Initial features like sending email, sending named values to the attached RPi, and reading named values from the RPi as viewing P2 values from the web backend of the RPi are all working.

I'm posting this here so we have a thread where we all can comment.

Any model RPi should work as long as it is running Raspberry Pi OS.

My git repo ironsheep/P2-RPi-IoT-gateway is live! You can find:

RPi Setup Instructions
Installation Instructions
Demos for each working feature (send email, display 1-wire on a web-page, etc.)
Upcoming features - I'm awakening these one-by-one - keep an eye on the repo!

On the future features list are:

SMS
Twitter
MQTT
MQTT w/Advertisement (for use with Home Assistant)

My existing Home Assistant Projects have shown me already how to add MQTT/HA support. ;-)

JonnyMac · 2022-01-19 23:50

Sorry, I missed it -- very busy with work. Looking forward to giving it a try.

pik33 · 2022-01-20 07:20

I have the same idea, but without RPi OS. I use Ultibo instead, a bare metal environment for Raspberry Pi. . I connected the Pi to P2 via UART and it works as a keyboard and mouse interface. The Ultibo powered RPi can be also a mass storage, Ethernet and Wifi interface for a P2.

While we have less flexibility using Ultibo than the OS - no Python with its libraries, etc - the big advantages of OS free RPi are (1) fast - 3-5 seconds - boot (2) no SD wear/system shutdown problems. You can simply switch it off as the Ultibo doesn't write anything on the SD on its own.

Stephen Moraco · 2022-01-24 20:57

@pik33 Useful to know, thank you!
It would be fun to see a project contributed with all the necessary effort to get that environment running described for us with step-by-step instructions. Then our community would have alternate choices when bringing up a gateway. There are a number of mentions of Ultibo but I've not seen any contributions yet.

I'm focused on this approach as, in my past, I spent years as a Debian developer so I have a lot of experience with this OS. (Those packages that we install with apt? I created and maintained a number of them.) I also have a small farm of RPi's running this way in my house. So I use, write about, and teach what I know best. As you can see here: Prior RPi work I have many open-source projects for this environment some of which have thousands of users.

Regarding the uSD write issues, I simply use a high write capability uSD card in RPi's where I need to care about such things. (Like using one as a security cam, etc.). This is mentioned in my RPi setup instructions.

Again, thanks for mentioning this alternative!

pik33 · 2022-01-25 12:14

I have a project open, but this is the very early stage.

https://github.com/pik33/P2-retromachine

As it is today, there is Ultibo directory in this repository. It contains the source code for Ultibo/RPi Zero and, in the Release directory, 4 files to copy onto a blank FAT32 formatted SD. These are 3 basic firmware files and a custom compiled kernel.img. These are all files needed to run the interface - the program works as a keyboard and a mouse interface for a P2. RPi zero has to be connected to a P2 via its UART which works at 1.92 Mbps

The P2 side of this interface is 'retrocog.spin2' file in Propeller directory. It runs at its own cog, receiving data from the interface and also from standard Flexprop serial terminal (at 2 Mbps) and a MIDI breakout board. To test this, received data are available at (unused in Flexprop) hub addresses $30, $34, $38, $3C. The main program has to check if there is non-zero value. If yes, then read it and write zero, then the retrocog will place the next data from its buffer, if there are any.

I will create a topic for this project when I manage to solder the device for it: I 3D-printed a box for a P2 Edge, Zero 2, MIDI breakout and a lot of DB connectors to play. To do is to solder all the planned wires and do some basic tests.

Ultibo as the envirinment can be installed on Windows machine (easiest way: use setup.exe) or Linux (using the install script) - including a Raspberry Pi OS. I use the RPi: a shell script can rename Linux kernel and replace with Ultibo one, then reboot. Ultibo program can then restore Linux kernel before rebooting - no SD card swapping needed until a bug prevents the program from restoring Linux kernel.

I managed to write an audio and camera drivers for Ultibo: https://github.com/pik33/SimpleAudio , https://github.com/pik33/SimpleCamera

hinv · 2022-07-24 14:37

@"Stephen Moraco" This looks like quite a useful tool. Will this gateway allow the pi to boot the P2 via the pi UART? If so, I could run FlexProp on the Pi, and that would open up a lot of possibilities.

@pik33 same question to you.

pik33 · 2022-07-24 16:53

To run flexprop with the Pi you need a linux running on the Pi and connect the Pi via P62,P63. Flexprop has way too many dependencies to port it to Ultibo but with Raspbian it works well. I already have a Pi configured for P2 programming.

However I use a configuration which allows reboot from Ultibo to Raspbian and from Raspbian to Ultibo, so yes, I can do this.

The device I mentioned before is still not completed. I was unable to get PSRAM working and I didn't have EC32. Now I have the EC32, and there are CLK/3 drivers available. Maybe I will try to use 4 chips and CLK/3 in the device.

The idea of using RPi+P2 combo in one device and programming the P2 with a Pi is interesting, but I think I need a more powerful Pi than a Zero. Either Zero 2 or a full 3B+. The main problem is you cannot buy any Raspberry Pi now, but I have several Pis to experiment with. I have a Zero 2 - moving to Zero 2 doesn't need to print a new box. Simply replace a Pi. There is however a bug in Ultibo which affects RPi Zero 2 and its USB subsystem. The bug is reported and confirmed, so I am waiting for the patch.

I have several new ideas now

Maybe a PCB with an edge connector for P2 and 40-pin connector for a Pi?

Wuerfel_21 · 2022-07-24 17:57

@pik33 said:
To run flexprop with the Pi you need a linux running on the Pi and connect the Pi via P62,P63. Flexprop has way too many dependencies to port it to Ultibo

Does it? I'm pretty sure the actual flexspin compiler has literally zero dependencies (aside from having bison installed on your build machine).

pik33 · 2022-07-24 20:01

Ultibo is a Pascal environment. It can link C (and not C++) static libraries ( .a file) compiled for arm-none-eabi. Only dependencies that are accepted are libc and libm.

If there is a chance that the compiler can be compiled with these restrictions, then yes, we can have a OS-less combo machine

I recompiled libmad mp3 decoder for Ultibo. The most time consuming thing was to remove all errno, logging and and printf to stderr stuff. After deleting them all from the code, Ultibo accepted the .a file I compiled. Then I lost a modified source somewhere I recompiled also libxmp for module playing: this was simpler to do, less to remove. The workflow was like this: compile to .a, try to link to Ultibo code, got a linker error, find the offending stuff, tell several bad words, remove the (bad word), repeat.

Wuerfel_21 · 2022-07-24 20:12

@pik33 said:
Ultibo is a Pascal environment. It can link C (and not C++) static libraries ( .a file) compiled for arm-none-eabi. Only dependencies that are accepted are libc and libm.

If there is a chance that the compiler can be compiled with these restrictions, then yes, we can have a OS-less combo machine

There isn't just a chance, that's literally just what it does.

gcc -g -Og -Wall -fwrapv -I. -I./backends -I./frontends -I./build -DFLEXSPIN_BUILD -o build/flexcc flexcc.c cmdline.c build/common.o build/case.o build/spinc.o build/lexer.o build/uni2sjis.o build/symbol.o build/ast.o build/expr.o build/dofmt.o build/flexbuf.o build/lltoa_prec.o build/strupr.o build/strrev.o build/strdupcat.o build/to_utf8.o build/from_utf8.o build/preprocess.o build/functions.o build/cse.o build/loops.o build/hloptimize.o build/hltransform.o build/types.o build/pasm.o build/outdat.o build/outlst.o build/outobj.o build/spinlang.o build/basiclang.o build/clang.o build/outasm.o build/assemble_ir.o build/optimize_ir.o build/inlineasm.o build/compress_ir.o build/outbc.o build/bcbuffers.o build/bcir.o build/bc_spin1.o build/outnu.o build/nuir.o build/nupeep.o build/outcpp.o build/cppfunc.o build/outgas.o build/cppexpr.o build/cppbuiltin.o build/directive.o build/expand.o build/mbchar.o build/mcpp_eval.o build/mcpp_main.o build/mcpp_system.o build/mcpp_support.o build/version.o build/becommon.o build/brkdebug.o build/spin.tab.o build/basic.tab.o build/cgram.tab.o -lm

Only -lm.

Bigger issue is that the code is totally not re-entrant and will leave all of its garbo on the heap after its done.

rogloh · 2022-07-25 05:23

I like the sound of that. I found Ultibo boots really fast on a RasPi. If it could become a debug host for a P2 and build code as well that could be useful down the track and could make a nice somewhat self contained setup without requiring a full PC for example. You might be able to have the P2 interact with a Pi and run external compilation jobs then reload into RAM and execute, or save to local filesystems on the P2. Things like that might become possible.

I picked up a Raspi400 recently because I noticed it was getting harder to get a hold of the Pi4's, but haven't used it in anger yet. Does Ultibo work on these systems as well pik33?

pik33 · 2022-07-25 06:21

USB 3 subsystem still doesn't work on Pi4 family. The best machine for Ultibo is still a 3B+

Stephen Moraco · 2022-07-25 07:40

@hinv said:
@"Stephen Moraco" This looks like quite a useful tool. Will this gateway allow the pi to boot the P2 via the pi UART? If so, I could run FlexProp on the Pi, and that would open up a lot of possibilities.

Yes, it will the way you suggest with a little work (the code in the repo is not set up that way but could be.)...

Regarding possibilities, yes! They are already there! I've been building for the P2 with FlexProp running on the RPi or running on my Macs or even running under windows for longer than this project existed thanks to Eric's (and others) excellent work on FlexProp. Let me explain:

My development environment is on large (many cores, lot's of ram & storage) mac's. I use VSCode as my coding environment (as you might expect from my other efforts for this community ;-) I use SSH extensions under VSCode and remotely log into the RPi which is running Raspios (Debian GNU Linux derivative) VScode simply downloads its extensions to the RPi when I first connect. The P2 is simply connected via USB on the RPi. I then have FlexProp running on the RPi compile and download to the P2 all from my mac keyboard controlled by VSCode.

Actually, this is only one of the arrangements I use to build code for the P2. I can (2) do this all on the mac again using VSCode and FlexProp both running on the max w/P2 connected via USB to the mac or I can also (3) run Windows in a virtual machine on my mac in which I run PropTool and PNut using them to download to the USB-connected P2. Of course, I do (4) have a rather smallish older actual Windows machine that mounts my dropbox files and in this case I still develop on my mac but this 4th form is now my less favorite of all of these environments I use.

I mention all of this as VSCode along with the wonderful FlexProp effort makes it really easy to develop for the P2 on any of our platforms including RPi's, and no you don't have to access them remotely as I am so used to doing ;-) The RPi's work as a standalone development environment with VSCode (or your favorite editor) along with FlexProp too.

Does this help?

-Stephen

pik33 · 2022-07-25 13:49

@Wuerfel_21 said:

@pik33 said:
Ultibo is a Pascal environment. It can link C (and not C++) static libraries ( .a file) compiled for arm-none-eabi. Only dependencies that are accepted are libc and libm.

If there is a chance that the compiler can be compiled with these restrictions, then yes, we can have a OS-less combo machine

There isn't just a chance, that's literally just what it does.
gcc -g -Og -Wall -fwrapv -I. -I./backends -I./frontends -I./build -DFLEXSPIN_BUILD -o build/flexcc flexcc.c cmdline.c build/common.o build/case.o build/spinc.o build/lexer.o build/uni2sjis.o build/symbol.o build/ast.o build/expr.o build/dofmt.o build/flexbuf.o build/lltoa_prec.o build/strupr.o build/strrev.o build/strdupcat.o build/to_utf8.o build/from_utf8.o build/preprocess.o build/functions.o build/cse.o build/loops.o build/hloptimize.o build/hltransform.o build/types.o build/pasm.o build/outdat.o build/outlst.o build/outobj.o build/spinlang.o build/basiclang.o build/clang.o build/outasm.o build/assemble_ir.o build/optimize_ir.o build/inlineasm.o build/compress_ir.o build/outbc.o build/bcbuffers.o build/bcir.o build/bc_spin1.o build/outnu.o build/nuir.o build/nupeep.o build/outcpp.o build/cppfunc.o build/outgas.o build/cppexpr.o build/cppbuiltin.o build/directive.o build/expand.o build/mbchar.o build/mcpp_eval.o build/mcpp_main.o build/mcpp_system.o build/mcpp_support.o build/version.o build/becommon.o build/brkdebug.o build/spin.tab.o build/basic.tab.o build/cgram.tab.o -lm
Only -lm.

Bigger issue is that the code is totally not re-entrant and will leave all of its garbo on the heap after its done.

First try: of course a lot of errno and stderr. As expected. A lot of broken references to standard functions was however not expected. It seems this will need a lot of work to make this compiler Ultibo compatible

Wuerfel_21 · 2022-07-25 13:54

If your link fails on stderr your libc is succ.

pik33 · 2022-07-25 14:13

Ultibo simply doesn't have such thing as stderr, so this is not a libc problem, this is the environment problem. This is bare metal, FreePascal based thing. While it can accept libraries coded in C, it has a lot of limitation. Ultibo, however, accepts opening and accessing files from C based code, but its libc is restricted.

As I wrote earlier, I managed to port libmad to Ultibo, where most time consuming process was removing all errno and stderr stuff. This I expected. However libmad was intended to be a library, while spin2cpp is intended to be a program. So the first thing I had to do was to rename the main

As libmad is a library, its makefile has options for build a static library, so this was much easier. make, try to link, remove all these unnecessary things that were broken.

My gcc/makefile programing experience is still too low. What I always used for programming everything, everywhere was Lazarus. That means the broken math function references the most probably are the result of my lack of knowledge ("how to compile this stuff as a library").

pik33 · 2022-07-25 14:55

Second try: now it is much better: no broken references for standard functions any more. There are however broken references to __ctype_something_loc and several other things.

hinv · 2022-07-27 02:59

@rogloh said:
I like the sound of that. I found Ultibo boots really fast on a RasPi. If it could become a debug host for a P2 and build code as well that could be useful down the track and could make a nice somewhat self contained setup without requiring a full PC for example. You might be able to have the P2 interact with a Pi and run external compilation jobs then reload into RAM and execute, or save to local filesystems on the P2. Things like that might become possible.

I picked up a Raspi400 recently because I noticed it was getting harder to get a hold of the Pi4's, but haven't used it in anger yet. Does Ultibo work on these systems as well pik33?

With a PiStorm type protocol, the Pi, which is connected via 26 GPIO's could also serve up RAM, probably at some good performance, and without an OS on the Pi, maybe even somewhat jitter free.

rogloh · 2022-07-27 03:58

@hinv said:
With a PiStorm type protocol, the Pi, which is connected via 26 GPIO's could also serve up RAM, probably at some good performance, and without an OS on the Pi, maybe even somewhat jitter free.

OMG do I now have another memory driver to write? I know about the SMI on the Pi but not much about PiStorm.

https://iosoft.blog/2020/07/16/raspberry-pi-smi/

The thing about SMI is that is seems to be more in line with an async register style transfer bus with its low address pin count and designed to be controlled from the Pi. Maybe an address phase from the P2 to the Pi could be done with SPI and then have the data transfer from Pi->P2 go via SMI (or the Pi could pull data from the P2 using SMI in case of writes to memory). The async nature of it probably makes it harder to transfer at high speed unless the Pi's DMA clock source or its SMI transfers could be driven from a P2 IO pin perhaps, so they can remain in sync during the streamed transfer...? But I quite like the idea of a large amount of RAM for cheap using a Pi Zero running Ultibo or something fast to boot. It does chew quite a lot of pins though vs PSRAM.

My driver is limited to 256MB right now per memory bus because the top nibble of the address defines the type of memory request instead of being upper address bits. I've already found a 2Gbit flash chip from Winbond that would max this size out, and a Pi acting as a memory device obviously could too.

Stephen Moraco · 2022-07-27 04:01

@hinv said:

With a PiStorm type protocol, the Pi, which is connected via 26 GPIO's could also serve up RAM, probably at some good performance, and without an OS on the Pi, maybe even somewhat jitter free.

Most of this discussion should probably be made to be its own thread as it really might be hard to find in this RPi gateway software/mechanism thead.

But, having said that...
@hinv this can also be done with the RaspiOS in place By using an LKM as I've done in My Tri-panel LED driver LKM experiment.

hinv · 2022-08-28 12:40

@rogloh said:
OMG do I now have another memory driver to write? I know about the SMI on the Pi but not much about PiStorm.

https://iosoft.blog/2020/07/16/raspberry-pi-smi/

>
Sound's good to me!

The thing about SMI is that is seems to be more in line with an async register style transfer bus with its low address pin count and designed to be controlled from the Pi. Maybe an address phase from the P2 to the Pi could be done with SPI and then have the data transfer from Pi->P2 go via SMI (or the Pi could pull data from the P2 using SMI in case of writes to memory). The async nature of it probably makes it harder to transfer at high speed unless the Pi's DMA clock source or its SMI transfers could be driven from a P2 IO pin perhaps, so they can remain in sync during the streamed transfer...? But I quite like the idea of a large amount of RAM for cheap using a Pi Zero running Ultibo or something fast to boot. It does chew quite a lot of pins though vs PSRAM.

Yeah, but you could also let the Pi do HDMI full 1080p 24ibit, just have the protocol write to the frame buffer.

My driver is limited to 256MB right now per memory bus because the top nibble of the address defines the type of memory request instead of being upper address bits. I've already found a 2Gbit flash chip from Winbond that would max this size out, and a Pi acting as a memory device obviously could too.

256MB is fine for a Pi Zero -> P2

VonSzarvas · 2022-08-31 12:33

Just trying demo_p2gw_web_control demo today, but getting this error when running the gateway manually/verbose:

[2022-08-31 14:30:53] - Thread: taskSerialListener() started
[2022-08-31 14:30:53] - Thread: FileSystemWatcher() started
[2022-08-31 14:30:54] - Thread: MainLoop() running
[2022-08-31 14:30:59] - (DBG): TASK-RX rxD(6)=(b'de=\x10\x8c\xf5')
[2022-08-31 14:30:59] - (DBG): Incoming line(6)=[de=��]
[2022-08-31 14:30:59] ERROR: line(6)=[de=��] ! P2 LINE NOT Recognized !

The baud rates match. Anyone seen this ?

VonSzarvas · 2022-08-31 12:55

Update: That issue can be solved by an extra config step at the RPi terminal:

sudo raspi-config
Interface Options
Serial Port
Disable login shell over serial
Enable serial port hardware
Save/OK/Reboot when prompted.

Stephen Moraco · 2022-09-02 08:32

@VonSzarvas hrmf... guess I better add that step in the Repo setup docs, huh? ;-)

VonSzarvas · 2022-09-02 09:01

@"Stephen Moraco" said:
@VonSzarvas hrmf... guess I better add that step in the Repo setup docs, huh? ;-)

Turned out it was already there in this doc : Unbox & setup new RPI - (if this is your first time or you just want a reminder)

Totally my fault that I missed it, and a self-inflicted opportunity for learning the hard way !

One thought... maybe the importance of disabling console serial could be mentioned as a NOTE3 under the Initial Test section, for anyone that can't seem to get the comms working. I'm thinking people that already have an RPi setup might not consider opening that first doc, so could easily miss the step and be frustrated.

Stephen Moraco · 2022-09-02 09:25

@VonSzarvas good suggestion, i’ll update that on my Fri afternoon. Thanks!

Stephen Moraco · 2022-09-03 01:02

@VonSzarvas ok, page updated. Note3 now exists.
Look sufficient?

VonSzarvas · 2022-09-03 10:42

@"Stephen Moraco" said:
@VonSzarvas ok, page updated. Note3 now exists.
Look sufficient?

looks great.

Couple small feedbacks...

if you seems (should be "seem" ?)
text that follows Note3, starting "Using the optional command line..." - seems related to Note3, but I don't think it is?

Stephen Moraco · 2022-09-03 21:32

@VonSzarvas sheesh thanks for catching that. I probably shouldn't have placed the note3 in the middle of stuff. ;-)

Repaired
text reordered and cleaned up

hinv · 2022-09-04 05:01

I think this conversation belongs here: https://forums.parallax.com/discussion/174806/high-speed-data-transfer-from-rpi-to-propeller2#latest

Using a Raspberry Pi as an IoT gateway for our P2 Projects

Comments