Anyone know more about the data sent using WX command CONNECT? (and the subsequent SEND)
What is this format, and what are its variable options?
Are there options for "Connection: ?"
Are there other data type formats specific like this one that can be also sent?
ser.char(13) '\r 1
ser.char(10) '\n 1
ser.str(string("Host: caboose.networkA.local")) '28 Change host and domain here, change how many bytes it sends above, if your address is shorter/longer.
'ser.str(string("Host: car1.networkA.local")) '25
ser.char(13) '\r 1
ser.char(10) '\n 1
ser.str(string("Connection: keep-alive")) '22
ser.char(13) '\r 1
ser.char(10) '\n 1
ser.str(string("Accept: */*")) '11
ser.char(13) '\r 1
ser.char(10) '\n 1
ser.char(13) '\r 1
ser.char(10) '\n 1 92 or 89
Connect requires an IP address or Host name and a Port number to connect to. The send command needs to know how long the data is and then wait for you to send that amount of data.
An example would be a weather services call that returns the local weather.
Is there a faster way to connect to another prop and transfer data, but not have to disconnect every time you want to send more data?
The connect and disconnect timing for me is many seconds.
Can't the connection be used more than once?
That would be HTTP 1.1 - your "Connection: keep-alive" in the Spin Code. Not sure if the ESP supports it.
Another thing you can do is to resolve the IP address once and use IP instead of DNS name - DNS resolving takes time.
HTTP over TCP/IP is kind of a slow Protocol by design I think @localroger wrote something to just use UDP, there you can just send and receive data, but no build in error detection or correction, you need to do that by your own code.
With 2 P1's on each end you do not really need HTTP and TCP/IP I think the ESP supports TELNET so you could build some simple 'serial' bridge and just send and receive bytes like with a normal serial connection.
Another option would be to use UDP. This is a connectionless send where you listen on a port and then send a packet of data to that port. The problem with this method is there is no guarantee that the packet will received by the other unit.
HTTP tends to wait a bit to make sure all the data has been received before moving on. Some packets can be fragmented and if you just except the first packet you maybe missing some follow on packets that should have been included.
I notice when setting up a web page that by the time the propeller got around to responding to the request the other side had moved on and didn't receive the answer.
Another option would be to use UDP. This is a connectionless send where you listen on a port and then send a packet of data to that port. The problem with this method is there is no guarantee that the packet will received by the other unit.
HTTP tends to wait a bit to make sure all the data has been received before moving on. Some packets can be fragmented and if you just except the first packet you maybe missing some follow on packets that should have been included.
I notice when setting up a web page that by the time the propeller got around to responding to the request the other side had moved on and didn't receive the answer.
Mike
I tried running your demo code on the first post. Does the WX module send anything to the debug port when using that code? I can't seem to get it to do anything.
However I do not know how to make a host for the udp also I would probably need to see a working pair done in c?
Most of what I am doing is looking at the c code in the learn/examples/network/wifi directory in simpleide and trying it out.
Got any local udp Client/Host code i can try?
I don't mind that the data is a catch or nothing deal. I don't really need to know if the data arrived, the catcher better be ready, and know the size, format,etc.
I do the same with many of my other projects where I just send data to the other side, and if it gets there, good.
If not, another send will be made right away again anyway.
That is better than a connect: close: timing delay.
HTTP over TCP/IP is kind of a slow Protocol by design I think @localroger wrote something to just use UDP, there you can just send and receive data, but no build in error detection or correction, you need to do that by your own code.
Yes, TCP is bad for high-speed conversations because TCP will halt the whole exchange if a packet is lost in order to recover it, even if you don't care about that packet. All network games use UDP for this reason.
There is another option if the libraries support it. The ESP8266 and ESP32 support a custom protocol called ESPnow which allows direct peer-to-peer comms between the ESP's without joining a network. The peers just know each other by MAC address. It's limited to 250 bytes per packet but exchange is quick, taking about 5 milliseconds with confirmation to the sender if everything goes right, and it's optimized for long range at the expense of bit rate. I did a 700 meter range test with 4-inch 3 db antennas. I've been using the Arduino IDE though, and it's a real mess if you're not used to wading through C++ code.
I have more bad news.
When powering on any of my esp's (they all have the working firmware from the last working commit) and then connecting to them in ap mode.
When I navigate to the Networks page the module will do a hard crash and restart.
It will only do this when you first access the network page after it powers on, repeated accessing the page will not crash the module.
Heres the debug pin output. From power on, you can see where it crashes.
It has a E:M 2056 and a E:M 2096, I don't normally see those.
This is the parallax WX esp8266 module.
When it crashes, my webbrowser times out, my pc immediately reconnects(like it didn't even disconnect to the ap)
And I then click the esp address bookmark again and then the network page, and it doesn't crash.......
Hi Clock Loop !
About your post... are you still finding the "crash" with E:M 2096 to be an issue ?
I wanted to merge in the Parallax P2 loader with the current code-base in GitHub so manufacturing could flash a new batch of Parallax modules with the latest code and features. However... there's an issue (at least for my test setup).
Starting with a clean git clone of the current Parallax github version, it appears to run fine. And from my mobile or tablet I can connect to the module and use all the menus. BUT.... connect using Chrome browser and KABOOM !
At first I thought it might be a memory overflow, so I adjusted the makefile to 4096, and flashed with the appropriate settings for flash size and esp-init-data. That also runs fine and reports the right flash map at boot/etc... But crashes in the same way when Chrome connects. Works great from the tablet though !!
So then I looked at libesphttd gitHub (https://github.com/chmorgan/libesphttpd), and there's a recent big warning in README.md that the latest libesphttpd is broken for ESP8266 ! That might be something There's a couple issues on that GitHub too, that look like they won't help us.
I'm thinking the next thing I will try is an earlier version of libesphttpd.
Are you (or anyone) aware of anything like this on your builds ?
Does the latest (current) Parallax-ESP actually compile and work from desktop Chrome ? (Maybe I have a nasty Chrome setting that's flooding the connections somehow, or maybe you guys are using slightly different library versions?).
I believe your hitting the bug we noticed after upgrading to the latest ESP SDK.
It seems to only happen once after a new load of the firmware on the unit. I don't believe it's related to chrome.
The error does happen when trying to access a file on the HTTP server and the message you are getting is that it's decoding an encode file that needs to be expanded but tries to write to an address that is protected.
This code has not been changed in some time so it looks like a firmware issue with the SDK.
In any case I only see the error once after a new load of the firmware onto the ESP unit.
You can cause the error by accessing the network page with any browser.
I'm getting the error repeatably, not just once after flashing.
The new SDK also caught my eye- libesphttpd has some comments in the code to suggest the NONOS SDK doesn't handle comnections as well as desired.
Anyone know why the sdk was changed? I mean- if that was some specific benefit.?
Could we roll back to the previous sdk version, and still keep the new UI features? (Such as UDP, module visibility, etc...)
I'm planning to try the previous sdk in a couple hours. Swapping in the last libesphttpd didn't help, and I think you're right that the sdk was the only other "out of our control" thing that changed.
After further investigating it looks like the issue is with the http server. The reason I only see it once is that the web browser caches a lot of the elements that it needs for the page so it doesn't need to download them again. This causes the page to work the next time for me.
I did a force reload of a page and it seems to crash the system. May have something to do with the number of connections allowed to the https server. Going to be hard to debug.
It was thought the the latest SDK from Espressif was the thing to do. In this case it's not working. Going back maybe a little painful.
Yes, that's the bug. It gets worse when the dev console if open, and caching disabled
One thing stands out... that there appears to be a pause between the compression request and failure. Whereas when the serve is successful, there's no pause at all. The data just streams out.
There might be an inappropriate pause in the http server before it tries to re-send buffered data- and Chrome doesn't have such patience and opens another connection (and another, and another) to overwhelm the esp web server.
I'll take a quick look for that. Maybe we can fix a slow callback or remove the fail buffer completely (if it doesn't get sent, then give up and let Chrome re-request).
Otherwise, I'll just rollback to the previous SDK and libesphttpd which were working, and define-out any of the new features which are not compatible. Unfortunately there's not more time to play with this problem today, so if neither of those solutions work we'll have to go with the old version for the next build.
Shame. But I guess ESP moved to v3 for a reason!
I'll keep reporting back, and if there's any other ideas please shout. Happy to try anything.
Version 3 require a hole set of files that define where you want what that I have not been able to get to work. There maybe new docs out there now that describe what need to be built.
The problem is still the same. The HTTP server has a problem and is crashing the ESP unit. Would like to fix that if possible and move forward.
Also espressif may have written an HTTP server now as they do have one for ESP32. This however may cause other issues with some of the tools.
There's also a resources folder with the binaries and .ota file.
These are compiled with 4MB flash setting, so not sure if the .ota would work on a module that previously had a 2MB image. Might work
Flashing commands I used are in the README.md
It would be great to publish these formally, but with the module httpd crash bug that will be a no no!
To enable the P2 Loader, visit settings and set the Reset Pin to CTS, and the P2 Loader to enabled.
It should co-exist with Blockly/P1 loading. And at some point (hopefully) Propeller Tool will be able to post to the url of this firmware to program directly.
Sure these could be improved later into one command, and we could add other features such as "save to Flash" vs "run". Etc.. Etc..
This is meant to be a quick and simple start until the resources are available to get something done properly by our resident expert. Probably with direct streaming from IP port to serial.
I didn't feel it was fair to again ask David to help with this little thing as he already gave up so much of his free time in recent posts here. So this is a very simple starting point, and I would expect the entire code base to be changed before much more development work is done- to migrate to a stable and supported solution based on v.3x SDK hopefully, and to eradicate the unfortunate httpd server resets.
For now though, this allows users to Wirelessly program P2, and hopefully it will be handy for someone !
Yes, it will take the entire file and stream it via the WiFi module serial pins to the P2.
Reading back, I missed a step with those instructions above. I've corrected the post above.
The new "P2 Loader" page in the WiFi webgui works by first uploading the binary file to the WiFi module flash, and then the load-p2-file command starts the upload from that file to the attached P2.
There's also a resources folder with the binaries and .ota file.
These are compiled with 4MB flash setting, so not sure if the .ota would work on a module that previously had a 2MB image. Might work
Flashing commands I used are in the README.md
It would be great to publish these formally, but with the module httpd crash bug that will be a no no!
To enable the P2 Loader, visit settings and set the Reset Pin to CTS, and the P2 Loader to enabled.
It should co-exist with Blockly/P1 loading. And at some point (hopefully) Propeller Tool will be able to post to the url of this firmware to program directly.
Sure these could be improved later into one command, and we could add other features such as "save to Flash" vs "run". Etc.. Etc..
This is meant to be a quick and simple start until the resources are available to get something done properly by our resident expert. Probably with direct streaming from IP port to serial.
I didn't feel it was fair to again ask David to help with this little thing as he already gave up so much of his free time in recent posts here. So this is a very simple starting point, and I would expect the entire code base to be changed before much more development work is done- to migrate to a stable and supported solution based on v.3x SDK hopefully, and to eradicate the unfortunate httpd server resets.
For now though, this allows users to Wirelessly program P2, and hopefully it will be handy for someone !
Edit: Correct the API url/steps
Are you sure it's a good idea to write the file to flash memory on the wifi module? Unless someone has changed the code since I last worked on it, the flash filesystem wasn't very robust. It doesn't do any kind of wear leveling or error recovery. It was really designed to be simple and to host a static collection of files. I don't think it's a good idea to continually rewrite files. Shouldn't it be easy to just stream the file to the P2?
I have made some headway on the reboot issue with the HTTP server. I changed the number of connections it process from 4 down to 2 and all works with no reboots.
If I change it to 3 the problem returns. Need to look into this further. Maybe running out of connections.
I have made some headway on the reboot issue with the HTTP server. I changed the number of connections it process from 4 down to 2 and all works with no reboots.
If I change it to 3 the problem returns. Need to look into this further. Maybe running out of connections.
Mike
Sounds like we're running out of SRAM. I don't suppose Parallax would like to redesign this module using the ESP32?
Sounds good @iseries
I could adjust that max conns constant and test here too. Funny thing... I had tried increasing it to 6, but not reducing it !
What file is that rogue malloc in?
Are you sure it's a good idea to write the file to flash memory on the wifi module? Unless someone has changed the code since I last worked on it, the flash filesystem wasn't very robust. It doesn't do any kind of wear leveling or error recovery. It was really designed to be simple and to host a static collection of files. I don't think it's a good idea to continually rewrite files. Shouldn't it be easy to just stream the file to the P2?
Yeah, as stated this is a quick solution. I had only a couple hours to work on it, so went with the shortest path for me. I'm expecting a streaming solution to be required at some point, and if that's quick to add now I'd be all in for adding it now, but would need help to achieve that.
The Winbond flash on those modules are rated for 100000+ writes, so provided they don't fail, the lifespan should be many years for most users, and I'd expect an update long before then to improve life. Even adding a sequential digit to the end of each filename (per session) would be a massive improvement, to spread the files across the flash, rather than replacing the same file in the same flash location each time. If nothing else, I might add that when looking at the other things iSeries suggested.
Right now, I just wanted to share what I have. If anyone wants to use, improve or totally change things, then fantastic!
Think I found it. I added code to check for memory allocation errors and think I found the offending code.
Conn req from 101.1.1.7:60138, using pool slot 2
181854> E:M 2056
181854> Receive Call Back out of memory
181890> E:M 2056
181890> Send Buffer Out of Memory
Are you sure it's a good idea to write the file to flash memory on the wifi module? Unless someone has changed the code since I last worked on it, the flash filesystem wasn't very robust. It doesn't do any kind of wear leveling or error recovery. It was really designed to be simple and to host a static collection of files. I don't think it's a good idea to continually rewrite files. Shouldn't it be easy to just stream the file to the P2?
Yeah, as stated this is a quick solution. I had only a couple hours to work on it, so went with the shortest path for me. I'm expecting a streaming solution to be required at some point, and if that's quick to add now I'd be all in for adding it now, but would need help to achieve that.
The Winbond flash on those modules are rated for 100000+ writes, so provided they don't fail, the lifespan should be many years for most users, and I'd expect an update long before then to improve life. Even adding a sequential digit to the end of each filename (per session) would be a massive improvement, to spread the files across the flash, rather than replacing the same file in the same flash location each time. If nothing else, I might add that when looking at the other things iSeries suggested.
Right now, I just wanted to share what I have. If anyone wants to use, improve or totally change things, then fantastic!
Sorry, I didn't mean to be critical of your work. I just wanted to point out a possible downside to that approach. Another is that the current filesystem code doesn't reclaim the space used by files that are overwritten. You eventually have to empty the filesystem and start over if you keep rewriting the same file over and over. It would be better to find a more robust flash filesystem to use but the other options I found at the time used too much SRAM.
Comments
X and D are in the api pdf (32420-Parallax-Wi-Fi-Module-API-v1.0.pdf) but T is not.
So thank you! I wonder what that is telling me. '=T,5,0' Connect event on handle 5... ?
What is this format, and what are its variable options?
Are there options for "Connection: ?"
Are there other data type formats specific like this one that can be also sent?
ser.char(13) '\r 1 ser.char(10) '\n 1 ser.str(string("Host: caboose.networkA.local")) '28 Change host and domain here, change how many bytes it sends above, if your address is shorter/longer. 'ser.str(string("Host: car1.networkA.local")) '25 ser.char(13) '\r 1 ser.char(10) '\n 1 ser.str(string("Connection: keep-alive")) '22 ser.char(13) '\r 1 ser.char(10) '\n 1 ser.str(string("Accept: */*")) '11 ser.char(13) '\r 1 ser.char(10) '\n 1 ser.char(13) '\r 1 ser.char(10) '\n 1 92 or 89An example would be a weather services call that returns the local weather.
Connect: api.openweathermap.org, 80
Send: 62 "GET /data/2.5/weather?id=5274867&APPID=<YOURID>&units=imperial"
Mike
The connect and disconnect timing for me is many seconds.
Can't the connection be used more than once?
Another thing you can do is to resolve the IP address once and use IP instead of DNS name - DNS resolving takes time.
HTTP over TCP/IP is kind of a slow Protocol by design I think @localroger wrote something to just use UDP, there you can just send and receive data, but no build in error detection or correction, you need to do that by your own code.
With 2 P1's on each end you do not really need HTTP and TCP/IP I think the ESP supports TELNET so you could build some simple 'serial' bridge and just send and receive bytes like with a normal serial connection.
Mike
HTTP tends to wait a bit to make sure all the data has been received before moving on. Some packets can be fragmented and if you just except the first packet you maybe missing some follow on packets that should have been included.
I notice when setting up a web page that by the time the propeller got around to responding to the request the other side had moved on and didn't receive the answer.
Mike
I tried running your demo code on the first post. Does the WX module send anything to the debug port when using that code? I can't seem to get it to do anything.
However I do not know how to make a host for the udp also I would probably need to see a working pair done in c?
Most of what I am doing is looking at the c code in the learn/examples/network/wifi directory in simpleide and trying it out.
Got any local udp Client/Host code i can try?
I don't mind that the data is a catch or nothing deal. I don't really need to know if the data arrived, the catcher better be ready, and know the size, format,etc.
I do the same with many of my other projects where I just send data to the other side, and if it gets there, good.
If not, another send will be made right away again anyway.
That is better than a connect: close: timing delay.
One being getting the current time from a time server.
Also the current libraries don't support UDP so I will have to see what I can put together.
Mike
Yes, TCP is bad for high-speed conversations because TCP will halt the whole exchange if a packet is lost in order to recover it, even if you don't care about that packet. All network games use UDP for this reason.
There is another option if the libraries support it. The ESP8266 and ESP32 support a custom protocol called ESPnow which allows direct peer-to-peer comms between the ESP's without joining a network. The peers just know each other by MAC address. It's limited to 250 bytes per packet but exchange is quick, taking about 5 milliseconds with confirmation to the sender if everything goes right, and it's optimized for long range at the expense of bit rate. I did a 700 meter range test with 4-inch 3 db antennas. I've been using the Arduino IDE though, and it's a real mess if you're not used to wading through C++ code.
Hi Clock Loop !
About your post... are you still finding the "crash" with E:M 2096 to be an issue ?
I wanted to merge in the Parallax P2 loader with the current code-base in GitHub so manufacturing could flash a new batch of Parallax modules with the latest code and features. However... there's an issue (at least for my test setup).
Starting with a clean git clone of the current Parallax github version, it appears to run fine. And from my mobile or tablet I can connect to the module and use all the menus. BUT.... connect using Chrome browser and KABOOM !
At first I thought it might be a memory overflow, so I adjusted the makefile to 4096, and flashed with the appropriate settings for flash size and esp-init-data. That also runs fine and reports the right flash map at boot/etc... But crashes in the same way when Chrome connects. Works great from the tablet though !!
So then I looked at libesphttd gitHub (https://github.com/chmorgan/libesphttpd), and there's a recent big warning in README.md that the latest libesphttpd is broken for ESP8266 ! That might be something
I'm thinking the next thing I will try is an earlier version of libesphttpd.
Are you (or anyone) aware of anything like this on your builds ?
Does the latest (current) Parallax-ESP actually compile and work from desktop Chrome ? (Maybe I have a nasty Chrome setting that's flooding the connections somehow, or maybe you guys are using slightly different library versions?).
It seems to only happen once after a new load of the firmware on the unit. I don't believe it's related to chrome.
The error does happen when trying to access a file on the HTTP server and the message you are getting is that it's decoding an encode file that needs to be expanded but tries to write to an address that is protected.
This code has not been changed in some time so it looks like a firmware issue with the SDK.
In any case I only see the error once after a new load of the firmware onto the ESP unit.
You can cause the error by accessing the network page with any browser.
Mike
The new SDK also caught my eye- libesphttpd has some comments in the code to suggest the NONOS SDK doesn't handle comnections as well as desired.
Anyone know why the sdk was changed? I mean- if that was some specific benefit.?
Could we roll back to the previous sdk version, and still keep the new UI features? (Such as UDP, module visibility, etc...)
I'm planning to try the previous sdk in a couple hours. Swapping in the last libesphttpd didn't help, and I think you're right that the sdk was the only other "out of our control" thing that changed.
I did a force reload of a page and it seems to crash the system. May have something to do with the number of connections allowed to the https server. Going to be hard to debug.
It was thought the the latest SDK from Espressif was the thing to do. In this case it's not working. Going back maybe a little painful.
Mike
One thing stands out... that there appears to be a pause between the compression request and failure. Whereas when the serve is successful, there's no pause at all. The data just streams out.
There might be an inappropriate pause in the http server before it tries to re-send buffered data- and Chrome doesn't have such patience and opens another connection (and another, and another) to overwhelm the esp web server.
I'll take a quick look for that. Maybe we can fix a slow callback or remove the fail buffer completely (if it doesn't get sent, then give up and let Chrome re-request).
Otherwise, I'll just rollback to the previous SDK and libesphttpd which were working, and define-out any of the new features which are not compatible. Unfortunately there's not more time to play with this problem today, so if neither of those solutions work we'll have to go with the old version for the next build.
Shame. But I guess ESP moved to v3 for a reason!
I'll keep reporting back, and if there's any other ideas please shout. Happy to try anything.
If it's a seamless transition then nothing preventing it.
The problem is still the same. The HTTP server has a problem and is crashing the ESP unit. Would like to fix that if possible and move forward.
Also espressif may have written an HTTP server now as they do have one for ESP32. This however may cause other issues with some of the tools.
Mike
https://github.com/VonSzarvas/Parallax-ESP
There's also a resources folder with the binaries and .ota file.
These are compiled with 4MB flash setting, so not sure if the .ota would work on a module that previously had a 2MB image. Might work
Flashing commands I used are in the README.md
It would be great to publish these formally, but with the module httpd crash bug that will be a no no!
To enable the P2 Loader, visit settings and set the Reset Pin to CTS, and the P2 Loader to enabled.
It should co-exist with Blockly/P1 loading. And at some point (hopefully) Propeller Tool will be able to post to the url of this firmware to program directly.
The behind-the-scenes API / URL steps are:
1) POST the binary file to the WiFi module flash memory: http://moduleIP/userfs/write?file=<filename>
2) Then start the upload with: http://moduleIP/propeller/load-p2-file?file=<filename>
Sure these could be improved later into one command, and we could add other features such as "save to Flash" vs "run". Etc.. Etc..
This is meant to be a quick and simple start until the resources are available to get something done properly by our resident expert. Probably with direct streaming from IP port to serial.
I didn't feel it was fair to again ask David to help with this little thing as he already gave up so much of his free time in recent posts here. So this is a very simple starting point, and I would expect the entire code base to be changed before much more development work is done- to migrate to a stable and supported solution based on v.3x SDK hopefully, and to eradicate the unfortunate httpd server resets.
For now though, this allows users to Wirelessly program P2, and hopefully it will be handy for someone !
Edit: Correct the API url/steps
Mike
Reading back, I missed a step with those instructions above. I've corrected the post above.
The new "P2 Loader" page in the WiFi webgui works by first uploading the binary file to the WiFi module flash, and then the load-p2-file command starts the upload from that file to the attached P2.
Then the Telnet port could be used to load the P2 just like it currently is being loaded.
This is kind of how the P1 is loaded today.
Mike
I have made some headway on the reboot issue with the HTTP server. I changed the number of connections it process from 4 down to 2 and all works with no reboots.
If I change it to 3 the problem returns. Need to look into this further. Maybe running out of connections.
Mike
Mike
I could adjust that max conns constant and test here too. Funny thing... I had tried increasing it to 6, but not reducing it !
What file is that rogue malloc in?
Yeah, as stated this is a quick solution. I had only a couple hours to work on it, so went with the shortest path for me. I'm expecting a streaming solution to be required at some point, and if that's quick to add now I'd be all in for adding it now, but would need help to achieve that.
The Winbond flash on those modules are rated for 100000+ writes, so provided they don't fail, the lifespan should be many years for most users, and I'd expect an update long before then to improve life. Even adding a sequential digit to the end of each filename (per session) would be a massive improvement, to spread the files across the flash, rather than replacing the same file in the same flash location each time. If nothing else, I might add that when looking at the other things iSeries suggested.
Right now, I just wanted to share what I have. If anyone wants to use, improve or totally change things, then fantastic!
Think I found it. I added code to check for memory allocation errors and think I found the offending code.
Conn req from 101.1.1.7:60138, using pool slot 2
181854> E:M 2056
181854> Receive Call Back out of memory
181890> E:M 2056
181890> Send Buffer Out of Memory
Now how to find more memory?
Mike
Sorry, I didn't mean to be critical of your work. I just wanted to point out a possible downside to that approach. Another is that the current filesystem code doesn't reclaim the space used by files that are overwritten. You eventually have to empty the filesystem and start over if you keep rewriting the same file over and over. It would be better to find a more robust flash filesystem to use but the other options I found at the time used too much SRAM.