Catalina WiFi support
RossH
Posts: 5,553
Ok, I now have WiFi support working in Catalina, using the ESP8266 WiFi module. I will release it once I have done more testing.
But I have a couple of issues/questions:
The WiFi adapter only supports 4 concurrent listeners (e.g. for TCP or websocket connections). Is there a reason for this limit? Is it possible it could be expanded? Not a showstopper, but I would prefer 8 as a minimum. Is this possible?
On the P2 EDGE with PSRAM (i.e. a P2-EC32MB) I can download software to the P2 using WiFi if I put the WiFi adapter on pins 56-63, but I cannot use the WiFi adapter serial interface because the serial pins conflict with those used for the PSRAM. If I put the WiFi adapter on a different pin group (e.g. pins 32 - 39) I can use the WiFi adapter serial interface but cannot download software to the P2. Is there a way to do both on a P2-EC32MB?
The WiFi adapter only supports data sizes up to about 900 bytes (e.g. 900 works, 950 fails - I have not narrowed it down any further, I just limit it to 512 bytes) - but the documentation suggests it should support data sizes up to 1024 bytes. Again, not a showstopper, because it just means more data packets are required - but am I missing something here?
Ross.
Comments
Regarding no.2, the pins required for WiFi programming and comms are tx,rx,reset,power, ground.
Would disconnecting (or cutting the traces) to all other IO pins on the WiFi adapter resolve the issue ?
From memory, I think that means the signals on the WiFi adapter connecting to P2 IO pins 56 & 57.
Edit: Yes ... looking now, those were the two traces (56 & 57) I cut on the adapter I use for WiFi programming. Suitable for both P2 module types.
Thanks. So the only way to reset the WiFi module on the EC32MB is to power cycle the P2?
Aren't the PSRAM-related pins NC on the EC32MB, anyways?
I thought there was an API / serial command for reset… would that work in your application, or do you need hardware reset?
Yes, so there should be no need to cut traces. But I want to use the RES and PGM pins on the WiFi module (so that I can reset the module without powering off the Propeller every time) - and that means I can't use pin group 56 .. 63. So at the moment I am using pin group 32 .. 39.
Once I finish testing, I probably won't have so much need to reset the module, so I can then go back to using 56 .. 63.
Ah… in that case solder the 0.1” sip header into the wifi adapter and use an FF wire jumper to any other IO pin from the reset signal?
@RossH
Back at the office and found the details on that...
https://www.parallax.com/package/p2-wx-adapter-add-on-board-product-guide/
Pages 6 & 7
The SIP header provides access to the reset and PGM pins, as also connected to the P2 header #0 and #1.
Does that solve and provide the flexibility you needed for testing?
Oh yes, I can do everything I need. I just use a different pin group when I need to test the RES or PGM functionality.
It just seems odd that Parallax chose to use pins #0 and #1 (which are not connected when using a P2-EC32MB) when #2 to #5 were (AFAIK) available, connected and unused, and so would have worked on all P2 EDGE boards.
But I guess that's why I'm a software engineer and not a hardware engineer!
Yes, It’s interesting about the pinout. I suspect that adapter was made before P2EC32MB existed. That said, those other pins probably interfere with the onboard flash? Pros and cons I suppose
Either way, the SIP breakout was included for total flexibility, regardless of current or future boards. It’s a handy place to hook in a scope or LA too!
Glad you’re rolling.
Well, that's annoying. Had the WiFi all working fine on a standard pin group (e.g. pins 16 .. 23) but then I plugged it into pin group 56 to make sure it works on that one and it no longer workee at all. Not sure why. It worked ok initially, but some command I sent to it has either damaged it or put it into a mode I cannot get it out of. Putting it back on pin group 16 and it no longer works with the software that was working before. Tried variously grounding the RES, PGM and DI lines (as documented) but no response. I think it's kaput.
I have ordered another WiFi board. But it will take a couple of weeks to get here.
Ross.
How about if you plug the module back into pins 16...23, then pulse low the PGM pin 4 times with 100ms intervals ?
- electrical detail - PGM is pulled high (to VIN) internally, and protected against external high signals. You can either drive the IO low, then revert to an input. OR you can drive low/high @ 3.3V. Either way is fine.
Assuming the module is working, that would force the module back into AP mode. Then you could find it from your laptop/etc.. and connect to 192.168.4.1 for recovery ?
That functionality was designed for the DIP version of the module, so that on the Activity Board you could press the reset button 4 times to "reset" the module.
But I think it will work the same over the SIP pin. -- but if not... just grab a bit of ground wire and tap it onto the DIP socket pin directly, 4 times like you are pressing a button!
Another tip-- connect a UART-RX to the DBG-TX pin and see if anything useful is transmitted when you powercycle the module. Could be some diagnostic messages that give clues.
Finally - you could reflash the firmware back to defaults. I posted the instructions on the forums a long time ago, so I assume the firmware is old version! I'll check it in the week when I'm less mobile
https://forums.parallax.com/discussion/171487/wx-esp8266-wifi-module-new-problem
The latest OTA for SIP modules is here (v1.64): https://drive.google.com/drive/folders/1MDAuHjWESUkg14c3rj9p6B8i_Ob363D_
I'll add that to the Parallax product page this week, once I've a moment to check the instructions still make sense!
Well, some good news, some bad ... nothing I could do to the module itself worked, so I tried reflashing it. That works, which tells me it is not completely dead.
However, now all it does when booted is flash the DO led and when you decode what it is spitting out, it turns out to be the following over and over (at 74880 baud!!!) ...
It would appear that it is trying to boot the firmware, crashing and rebooting. This would seem to indicate that the firmware no longer matches the WiFi module. Or perhaps the FLASH is not working properly.
When I go here, I get a "You need access" message. I sent a request.
Ross.
Access granted. Also, all is not lost yet on the reflash - looks like another cfg file needed. Will arrange tomorrow.
@RossH
Here's an updated recovery loader. It's very similar to previous, just with a couple extra params in the esptool command to update/reset the filesystem along with the firmware. (Your error was because the original firmware in your module was compiled with an earlier version of the Espressif framework).
Would you kindly let us know if that works out, and I'll upload this zip to the webstore page.
VERY IMPORTANT to note for future readers---- this firmware version is ONLY for the SIP version of the module !!!
Good news!
My WiFi module is back on the air! And my software works again! Thanks for your help.
Is there documentation for this new firmware? I have a few questions:
Thanks again for your help.
Ross.
Great!
Q's..
Correct- it's there, but removed from the menu now that most/all IDE's have wireless programming directly. Having that option in the html's added quite an overhead.
CTS - leave disabled. It allows a P1 binary image saved in the WiFi module to load/run into a P1 when CTS is toggled 3 or 4 times. Not P2 ready yet. It was added for the Holiday pack- if you watch Ken's intro video for that, you'll see how it works.
Enforce ... not seen a case where it's required for P2, but it is required for users of P1 with BlocklyProp (and maybe SimpleIDE). It forces the reset pin to use the one selected by the user in the WiFi module web GUI. It can be used for either P1 or P2 applications if needed. Why was this added?... BlocklyProp is hardcoded to use the DTR pin for reset, so doesn't play well with a SIP WiFi module that uses CTS for reset! BlocklyProp ignores/overrides the user choice in the GUI- so we added this flag to allow the user to have the ultimate choice without breaking other functionality.
Docs will be updated no doubt at some stage, though I've not seen anyone assigned yet. I might get the CTS loader working for P2 applications too before the docs happen. It's quite a handy way to load a bunch of binaries into the WiFi module, and let the user choose which to run, or have one always run at power up, etc...
Hello @VonSzarvas
All good on the HTTP stuff - I can now send and receive HTTP requests, and even build a simple web server on the P2. But then I moved onto the TCP functionality, and had some issues.
Eventually, I realized that the TCP functionality is not a bidirectional connection on which both ends can simply send and receive data until it is closed - specifically, you can only do one one RECV on each connection - e.g. CONNECT RECV CLOSE. You can then CONNECT again if you want to send or receive more data, but you cannot use multiple RECV requests on the same connection.
I do not have the latest source code, but in the code I have seen (https://github.com/parallaxinc/Parallax-ESP) the reason seems to be the way the software manages the receive buffer, and in particular the connection->rxCount variable - it is only reset on initial connection, not when the incoming data is read by the application. So once the rx buffer fills up, that's it. No more data can be received on that connection.
I can work around this, but if this was a design decisions it seems like an odd one. To me, it looks more like a bug. If you can point me to the latest source code, I am happy to investigate further.
Ross.
Sure thing @RossH - latest source is here : https://github.com/VonSzarvas/Parallax-ESP
Always happy to merge in any improvements ; thank you for taking a look into it.
Here is a git diff (also attached) of the most minimal change I could make to TCP to make it work the way I think it should ...
diff --git a/parallax/sscp-tcp.c b/parallax/sscp-tcp.c index 268c712..6bea45c 100644 --- a/parallax/sscp-tcp.c +++ b/parallax/sscp-tcp.c @@ -155,7 +155,7 @@ static void ICACHE_FLASH_ATTR tcp_recv_cb(void *arg, char *data, unsigned short if (len > 0) { os_memcpy(c->rxBuffer + i, data, len); c->rxCount = i + len; - c->rxIndex = 0; + // c->rxIndex = 0; // reset only when all data read - see recv_handler if (c->rxCount >= SSCP_RX_BUFFER_MAX) c->flags |= CONNECTION_RXFULL; sscp_log("TCP: added %d bytes to buffer", len); @@ -269,8 +269,11 @@ static void ICACHE_FLASH_ATTR recv_handler(sscp_hdr *hdr, int size) connection->rxIndex += size; } - if (connection->rxIndex >= connection->rxCount) + if (connection->rxIndex >= connection->rxCount) { connection->flags &= ~CONNECTION_RXFULL; + connection->rxIndex = 0; // all data read => reset rxIndex + connection->rxCount = 0; // all data read => reset rxCount + } } static void ICACHE_FLASH_ATTR close_handler(sscp_hdr *hdr)With this change, once the TCP connection is established, both ends can arbitrarily SEND and RECV data as many times as they like until the connection is closed. While it is not a good thing to hold an internet connection open for extended periods, it seems odd to offer TCP without this ability, and on a private network it is perfectly fine.
The receiver can read the data in arbitrary sized chunks smaller than the buffer size, as long as it empties the buffer before it completely fills up (or data will be lost). A more robust solution would be to turn the buffer into a true circular buffer so that data can be both added and removed without it ever needing to be completely emptied. I may look at doing that, but first I want to look at the WebSockets support to see if that is a better option for what I need to do.
Ross.
EDIT: changes removed from this post, because more changes have now been made - see the zip file later in this thread.
Hello @VonSzarvas
I now have a version of Catalina with C and Lua support for the Parallax WiFi module ready for initial release, but I think I have found another issue with the WiFi module firmware. This one seems to prevent two propellers equipped with WiFi modules talking directly to each other (i.e. not via a WiFi router).
Everything works fine when I make both the client and server propeller's JOIN the AP offered by my WiFi router, but not when I ask the client propeller to JOIN the AP offered by the server propeller - when I do that the server propeller keeps "redirecting" all HTTP requests from the client propeller - but it redirects them to itself and never actually services them!
I can make it work by simply commenting out the following line in the file parallax/user_main.c:
{"*", cgiRedirectApClientToHostname, "192.168.4.1"},This line prevents a Propeller servicing HTTP requests from a client that joins the AP offered by the server and which then specifies that server using the IP address "192.168.4.1" - which it must because since there is no DNS in the AP. There are also comments in same file that indicate enabling DNS would break the BlockyProp Launcher.
Is there a better solution?
In the meantime, I can just include in the Catalina release the modified WiFi module firmware that makes it work.
Ross.
I suspect the Blockly problem was misunderstood at the time - looked like a gw issue to me, not dns. As such removing that redirect probably wouldn't impact other users. (Sure- there's the fancy autologin page, but my newer devices seem to ignore that anyway for security reasons I guess, or just obsolete protocol maybe- forever nudging us to buy new things).
Next time I'm working on that code I'll be sure to take a look. Or to let anyone else know who takes on a task with that code.
If you can distribute your own version in the meantime, that seems ideal. - Are you able to package up the .ota file ? Happy to generate that for you if that would help, based on your git diffs.
Attached is a complete package - source, binaries and git diff patch. I will also include this in the next Catalina release.
Ross.
Whew! Just got my first eLua/ALOHA programs working over WiFi, instead of using hardwired serial links.
I can now have a network of propellers that communicate using RPC over WiFi. Currently, the basic network configuration is similar to the ALOHA* configuration, with each MASTER propeller running a Lua Client that makes RPC calls, and multiple SLAVE propellers running Lua Servers that service those RPC calls. But because it uses WiFi and not hardwired links, there can be more than one MASTER propeller, and as many SLAVE propellers as you might need.
Sounds easy, right? ... but I can't believe how tricky it turned out to be! I had to use a combination of HTTP and TCP to implement the RPC calls. I decided not to use WebSockets just yet because each WiFi module can only have four WebSocket connections open at a time, and the cost and time taken to set them up and tear them down (as needed) would be high. But for small networks I may eventually offer WebSockets as an alternative - it is more robust, more secure, and would make sense if there are only four MASTERs or less, and four SLAVEs or less in the network.
Still a lot of tidying up to do, but I hope to release a first version soon - it will be posted in the main Catalina thread. Then, I can finally start building my home/farm automation system, implemented using Lua RPC calls across a network of WiFi linked Propellers, with a web interface that can be accessed from anywhere in the WiFi network.
Ross.
Well, while everything seems to be working, I don't think executing distributed Lua programs over WiFi is going to be breaking any speed records!
I've just modified the eLua version of the Life demo to work over WiFi instead of a serial link. The good news is that it executes perfectly. The even better news is that the only thing I need to do to make the program work over WiFi is add the following lines to the original Lua program:
-- define the WiFi network and the RPC calls supported: rpc_network = { ["SSID"] = "MyNetwork"; -- set to "" if this propeller offers the AP ["PASSPHRASE"] = "TellMeASecret"; -- set to "" if we are using a propeller AP [FETCH_SVC] = "xxx.xxx.xxx.xxx"; [QUIT_SVC] = "xxx.xxx.xxx.xxx"; }But bad news is that it currently takes about six times as long to execute. Some of that is because my code has not yet been optimized, and some of it is because I am using an intermediate WiFi network consisting of two WiFi repeaters and a WiFi router which is a few hundred metres away on the other side of the property (there is a bloody great hill between me and the satellite dish
) - but some of it is just that the WiFi adapter itself seems to be a quite slow.
Still, I don't need speed for my application, so I will plod on!
Ross.