Shop OBEX P1 Docs P2 Docs Learn Events
Parallax-ESP Module Latest Hack - Page 7 — Parallax Forums

Parallax-ESP Module Latest Hack

15791011

Comments

  • iseries wrote: »
    Looking at the code there seems to be three different codes sent.

    T - connect event
    X - disconnect event
    D - send data event

    Mike

    X and D are in the api pdf (32420-Parallax-Wi-Fi-Module-API-v1.0.pdf) but T is not.

    So thank you! I wonder what that is telling me. '=T,5,0' Connect event on handle 5... ?
  • Anyone know more about the data sent using WX command CONNECT? (and the subsequent SEND)

    What is this format, and what are its variable options?
    Are there options for "Connection: ?"
    Are there other data type formats specific like this one that can be also sent?
            ser.char(13)                '\r                   1
            ser.char(10)                '\n                   1
            ser.str(string("Host: caboose.networkA.local")) '28   Change host and domain here, change how many bytes it sends above, if your address is shorter/longer.
            'ser.str(string("Host: car1.networkA.local"))    '25                                                        
            ser.char(13)                '\r                   1
            ser.char(10)                '\n                   1
            ser.str(string("Connection: keep-alive"))       '22
            ser.char(13)                '\r                   1
            ser.char(10)                '\n                   1
            ser.str(string("Accept: */*"))                  '11           
            ser.char(13)                '\r                   1
            ser.char(10)                '\n                   1
            ser.char(13)                '\r                   1
            ser.char(10)                '\n                   1    92  or 89
    
    

  • Connect requires an IP address or Host name and a Port number to connect to. The send command needs to know how long the data is and then wait for you to send that amount of data.

    An example would be a weather services call that returns the local weather.

    Connect: api.openweathermap.org, 80
    Send: 62 "GET /data/2.5/weather?id=5274867&APPID=<YOURID>&units=imperial"

    Mike
  • Clock LoopClock Loop Posts: 2,069
    edited 2020-10-02 22:49
    Is there a faster way to connect to another prop and transfer data, but not have to disconnect every time you want to send more data?
    The connect and disconnect timing for me is many seconds.
    Can't the connection be used more than once?
  • That would be HTTP 1.1 - your "Connection: keep-alive" in the Spin Code. Not sure if the ESP supports it.

    Another thing you can do is to resolve the IP address once and use IP instead of DNS name - DNS resolving takes time.

    HTTP over TCP/IP is kind of a slow Protocol by design I think @localroger wrote something to just use UDP, there you can just send and receive data, but no build in error detection or correction, you need to do that by your own code.

    With 2 P1's on each end you do not really need HTTP and TCP/IP I think the ESP supports TELNET so you could build some simple 'serial' bridge and just send and receive bytes like with a normal serial connection.

    Mike
  • Another option would be to use UDP. This is a connectionless send where you listen on a port and then send a packet of data to that port. The problem with this method is there is no guarantee that the packet will received by the other unit.

    HTTP tends to wait a bit to make sure all the data has been received before moving on. Some packets can be fragmented and if you just except the first packet you maybe missing some follow on packets that should have been included.

    I notice when setting up a web page that by the time the propeller got around to responding to the request the other side had moved on and didn't receive the answer.

    Mike
  • iseries wrote: »
    Another option would be to use UDP. This is a connectionless send where you listen on a port and then send a packet of data to that port. The problem with this method is there is no guarantee that the packet will received by the other unit.

    HTTP tends to wait a bit to make sure all the data has been received before moving on. Some packets can be fragmented and if you just except the first packet you maybe missing some follow on packets that should have been included.

    I notice when setting up a web page that by the time the propeller got around to responding to the request the other side had moved on and didn't receive the answer.

    Mike

    I tried running your demo code on the first post. Does the WX module send anything to the debug port when using that code? I can't seem to get it to do anything.
    However I do not know how to make a host for the udp also I would probably need to see a working pair done in c?
    Most of what I am doing is looking at the c code in the learn/examples/network/wifi directory in simpleide and trying it out.

    Got any local udp Client/Host code i can try?
    I don't mind that the data is a catch or nothing deal. I don't really need to know if the data arrived, the catcher better be ready, and know the size, format,etc.
    I do the same with many of my other projects where I just send data to the other side, and if it gets there, good.
    If not, another send will be made right away again anyway.
    That is better than a connect: close: timing delay.
  • UDP is new to the WX and I only have examples going from the WX to a host or sending a packet and getting a response.

    One being getting the current time from a time server.

    Also the current libraries don't support UDP so I will have to see what I can put together.

    Mike
  • msrobots wrote: »
    HTTP over TCP/IP is kind of a slow Protocol by design I think @localroger wrote something to just use UDP, there you can just send and receive data, but no build in error detection or correction, you need to do that by your own code.

    Yes, TCP is bad for high-speed conversations because TCP will halt the whole exchange if a packet is lost in order to recover it, even if you don't care about that packet. All network games use UDP for this reason.

    There is another option if the libraries support it. The ESP8266 and ESP32 support a custom protocol called ESPnow which allows direct peer-to-peer comms between the ESP's without joining a network. The peers just know each other by MAC address. It's limited to 250 bytes per packet but exchange is quick, taking about 5 milliseconds with confirmation to the sender if everything goes right, and it's optimized for long range at the expense of bit rate. I did a 700 meter range test with 4-inch 3 db antennas. I've been using the Arduino IDE though, and it's a real mess if you're not used to wading through C++ code.
  • VonSzarvasVonSzarvas Posts: 3,273
    edited 2020-12-10 15:29
    Clock Loop wrote: »
    David Betz wrote: »
    Now that the build problems are resolved, what is the status of the firmware itself. What problems have people noticed?

    I posted this earlier in this thread, but I will repost.
    Clock Loop wrote: »
    I have more bad news.
    When powering on any of my esp's (they all have the working firmware from the last working commit) and then connecting to them in ap mode.
    When I navigate to the Networks page the module will do a hard crash and restart.
    It will only do this when you first access the network page after it powers on, repeated accessing the page will not crash the module.

    Heres the debug pin output. From power on, you can see where it crashes.
    It has a E:M 2056 and a E:M 2096, I don't normally see those.
    This is the parallax WX esp8266 module.
    When it crashes, my webbrowser times out, my pc immediately reconnects(like it didn't even disconnect to the ap)
    And I then click the esp address bookmark again and then the network page, and it doesn't crash.......

    Hi Clock Loop !

    About your post... are you still finding the "crash" with E:M 2096 to be an issue ?

    I wanted to merge in the Parallax P2 loader with the current code-base in GitHub so manufacturing could flash a new batch of Parallax modules with the latest code and features. However... there's an issue (at least for my test setup).

    Starting with a clean git clone of the current Parallax github version, it appears to run fine. And from my mobile or tablet I can connect to the module and use all the menus. BUT.... connect using Chrome browser and KABOOM !
    22707> E:M 2056<\r><\n>
     22708> URL = /wifi/140medley.min.js<\r><\n>
     22711> Heatshrink compressed file; decode parms = b4<\r><\n>
     22715> E:M 2096<\r><\n>
     22717> Fatal exception 29(StoreProhibitedCause):<\r><\n>
     22721> epc1=0x4000df64, epc2=0x00000000, epc3=0x00000000, excvaddr=0x00000000, depc=0x00000000<\r><\r><\n> ets Jan  8 2013,rst cause:1, boot mode:(3,7)<\r><\n>
    <\r><\n>
    load 0x40100000, len 2592, room 16 <\r><\n>
    tail 0<\r><\n>
    chksum 0xf3<\r><\n>
    load 0x3ffe8000, len 764, room 8 <\r><\n>
    tail 4<\r><\n>
    chksum 0x92<\r><\n>
    load 0x3ffe82fc, len 676, room 4 <\r><\n>
    tail 0<\r><\n>
    chksum 0x22<\r><\n>
    csum 0x22<\r><\n>
    <\r><\n>
    2nd boot version : 1.7(5d6f877)<\r><\n>
    SPI Speed : 80MHz<\r><\n>
    SPI Mode : QIO<\r><\n>
    SPI Flash Size & Map: 16Mbit(512KB+512KB)<\r><\n>
    jump to run user1 @ 1000<\r><\n>
    <\r><\n>
    <\n><\n><\n><\n>Flash config restore ok<\r><\n>
       111> Reset Pin: 12<\r><\n>
       111> RX Pullup: 0<\r><\n>
       117> DISCOVER: initialized<\r><\n>
       117> Version v1.2 (2020-12-10 16:08:03 15-gae3b50e)<\r><\n>
       118> Using pin 12 for reset<\r><\n>
       119> 2MB flash: base 00100000, size 1048576<\r><\n>
       123> Flash filesystem mounted!<\r><\n>
       126> Httpd init<\r><\n>
       128> <\r><\n>
       129> Ready<\r><\n>
       130> mode : softAP(1a:fe:34:f9:f2:12)<\r><\n>
       134> add if1<\r><\n>
       135> dhcp server start:(ip:192.168.4.1,mask:255.255.255.0,gw:192.168.4.1)<\r><\n>
       143> bcn 100<\r><\n>
    

    At first I thought it might be a memory overflow, so I adjusted the makefile to 4096, and flashed with the appropriate settings for flash size and esp-init-data. That also runs fine and reports the right flash map at boot/etc... But crashes in the same way when Chrome connects. Works great from the tablet though !!

    So then I looked at libesphttd gitHub (https://github.com/chmorgan/libesphttpd), and there's a recent big warning in README.md that the latest libesphttpd is broken for ESP8266 ! That might be something :) There's a couple issues on that GitHub too, that look like they won't help us.

    I'm thinking the next thing I will try is an earlier version of libesphttpd.

    Are you (or anyone) aware of anything like this on your builds ?
    Does the latest (current) Parallax-ESP actually compile and work from desktop Chrome ? (Maybe I have a nasty Chrome setting that's flooding the connections somehow, or maybe you guys are using slightly different library versions?).




  • I believe your hitting the bug we noticed after upgrading to the latest ESP SDK.

    It seems to only happen once after a new load of the firmware on the unit. I don't believe it's related to chrome.

    The error does happen when trying to access a file on the HTTP server and the message you are getting is that it's decoding an encode file that needs to be expanded but tries to write to an address that is protected.

    This code has not been changed in some time so it looks like a firmware issue with the SDK.

    In any case I only see the error once after a new load of the firmware onto the ESP unit.

    You can cause the error by accessing the network page with any browser.

    Mike
  • I'm getting the error repeatably, not just once after flashing.

    The new SDK also caught my eye- libesphttpd has some comments in the code to suggest the NONOS SDK doesn't handle comnections as well as desired.

    Anyone know why the sdk was changed? I mean- if that was some specific benefit.?

    Could we roll back to the previous sdk version, and still keep the new UI features? (Such as UDP, module visibility, etc...)

    I'm planning to try the previous sdk in a couple hours. Swapping in the last libesphttpd didn't help, and I think you're right that the sdk was the only other "out of our control" thing that changed.
  • After further investigating it looks like the issue is with the http server. The reason I only see it once is that the web browser caches a lot of the elements that it needs for the page so it doesn't need to download them again. This causes the page to work the next time for me.

    I did a force reload of a page and it seems to crash the system. May have something to do with the number of connections allowed to the https server. Going to be hard to debug.

    It was thought the the latest SDK from Espressif was the thing to do. In this case it's not working. Going back maybe a little painful.

    Mike
  • Yes, that's the bug. It gets worse when the dev console if open, and caching disabled :)

    One thing stands out... that there appears to be a pause between the compression request and failure. Whereas when the serve is successful, there's no pause at all. The data just streams out.

    There might be an inappropriate pause in the http server before it tries to re-send buffered data- and Chrome doesn't have such patience and opens another connection (and another, and another) to overwhelm the esp web server.

    I'll take a quick look for that. Maybe we can fix a slow callback or remove the fail buffer completely (if it doesn't get sent, then give up and let Chrome re-request).

    Otherwise, I'll just rollback to the previous SDK and libesphttpd which were working, and define-out any of the new features which are not compatible. Unfortunately there's not more time to play with this problem today, so if neither of those solutions work we'll have to go with the old version for the next build.

    Shame. But I guess ESP moved to v3 for a reason!

    I'll keep reporting back, and if there's any other ideas please shout. Happy to try anything.
  • What is preventing us from moving to SDK v3?
  • The unknown I suppose.

    If it's a seamless transition then nothing preventing it.


  • Version 3 require a hole set of files that define where you want what that I have not been able to get to work. There maybe new docs out there now that describe what need to be built.

    The problem is still the same. The HTTP server has a problem and is crashing the ESP unit. Would like to fix that if possible and move forward.

    Also espressif may have written an HTTP server now as they do have one for ESP32. This however may cause other issues with some of the tools.

    Mike
  • VonSzarvasVonSzarvas Posts: 3,273
    edited 2020-12-11 11:35
    I've added an updated Parallax-ESP here, which includes the P2 Loader:

    https://github.com/VonSzarvas/Parallax-ESP


    There's also a resources folder with the binaries and .ota file.
    These are compiled with 4MB flash setting, so not sure if the .ota would work on a module that previously had a 2MB image. Might work :)

    Flashing commands I used are in the README.md

    It would be great to publish these formally, but with the module httpd crash bug that will be a no no!


    To enable the P2 Loader, visit settings and set the Reset Pin to CTS, and the P2 Loader to enabled.
    It should co-exist with Blockly/P1 loading. And at some point (hopefully) Propeller Tool will be able to post to the url of this firmware to program directly.

    The behind-the-scenes API / URL steps are:
    1) POST the binary file to the WiFi module flash memory: http://moduleIP/userfs/write?file=<filename&gt;
    2) Then start the upload with: http://moduleIP/propeller/load-p2-file?file=<filename&gt;


    Sure these could be improved later into one command, and we could add other features such as "save to Flash" vs "run". Etc.. Etc..
    This is meant to be a quick and simple start until the resources are available to get something done properly by our resident expert. Probably with direct streaming from IP port to serial.
    I didn't feel it was fair to again ask David to help with this little thing as he already gave up so much of his free time in recent posts here. So this is a very simple starting point, and I would expect the entire code base to be changed before much more development work is done- to migrate to a stable and supported solution based on v.3x SDK hopefully, and to eradicate the unfortunate httpd server resets.

    For now though, this allows users to Wirelessly program P2, and hopefully it will be handy for someone !



    Edit: Correct the API url/steps
  • So this change will take the entire file and load it on the P2 without the being fed over the telnet port or is this the bootloader program?

    Mike
  • VonSzarvasVonSzarvas Posts: 3,273
    edited 2020-12-11 11:36
    Yes, it will take the entire file and stream it via the WiFi module serial pins to the P2.

    Reading back, I missed a step with those instructions above. I've corrected the post above.
    The new "P2 Loader" page in the WiFi webgui works by first uploading the binary file to the WiFi module flash, and then the load-p2-file command starts the upload from that file to the attached P2.



  • VonSzarvas wrote: »
    I've added an updated Parallax-ESP here, which includes the P2 Loader:

    https://github.com/VonSzarvas/Parallax-ESP


    There's also a resources folder with the binaries and .ota file.
    These are compiled with 4MB flash setting, so not sure if the .ota would work on a module that previously had a 2MB image. Might work :)

    Flashing commands I used are in the README.md

    It would be great to publish these formally, but with the module httpd crash bug that will be a no no!


    To enable the P2 Loader, visit settings and set the Reset Pin to CTS, and the P2 Loader to enabled.
    It should co-exist with Blockly/P1 loading. And at some point (hopefully) Propeller Tool will be able to post to the url of this firmware to program directly.

    The behind-the-scenes API / URL steps are:
    1) POST the binary file to the WiFi module flash memory: http://moduleIP/userfs/write?file=<filename&gt;
    2) Then start the upload with: http://moduleIP/propeller/load-p2-file?file=<filename&gt;


    Sure these could be improved later into one command, and we could add other features such as "save to Flash" vs "run". Etc.. Etc..
    This is meant to be a quick and simple start until the resources are available to get something done properly by our resident expert. Probably with direct streaming from IP port to serial.
    I didn't feel it was fair to again ask David to help with this little thing as he already gave up so much of his free time in recent posts here. So this is a very simple starting point, and I would expect the entire code base to be changed before much more development work is done- to migrate to a stable and supported solution based on v.3x SDK hopefully, and to eradicate the unfortunate httpd server resets.

    For now though, this allows users to Wirelessly program P2, and hopefully it will be handy for someone !



    Edit: Correct the API url/steps
    Are you sure it's a good idea to write the file to flash memory on the wifi module? Unless someone has changed the code since I last worked on it, the flash filesystem wasn't very robust. It doesn't do any kind of wear leveling or error recovery. It was really designed to be simple and to host a static collection of files. I don't think it's a good idea to continually rewrite files. Shouldn't it be easy to just stream the file to the P2?

  • Yes, I don't like that upload process either. I would prefer that a reset URL be setup to just reboot the P2 and turn of command codes.

    Then the Telnet port could be used to load the P2 just like it currently is being loaded.

    This is kind of how the P1 is loaded today.

    Mike
  • @VonSzarvas, @"David Betz"

    I have made some headway on the reboot issue with the HTTP server. I changed the number of connections it process from 4 down to 2 and all works with no reboots.

    If I change it to 3 the problem returns. Need to look into this further. Maybe running out of connections.

    Mike
  • David BetzDavid Betz Posts: 14,511
    edited 2020-12-11 12:46
    iseries wrote: »
    @VonSzarvas, @"David Betz"

    I have made some headway on the reboot issue with the HTTP server. I changed the number of connections it process from 4 down to 2 and all works with no reboots.

    If I change it to 3 the problem returns. Need to look into this further. Maybe running out of connections.

    Mike
    Sounds like we're running out of SRAM. I don't suppose Parallax would like to redesign this module using the ESP32?

  • I see in the http section that he does a malloc for the connection but doesn't check if he really got a spot in memory which could be are issue.

    Mike
  • Sounds good @iseries
    I could adjust that max conns constant and test here too. Funny thing... I had tried increasing it to 6, but not reducing it !
    What file is that rogue malloc in?

  • iseries wrote: »
    I see in the http section that he does a malloc for the connection but doesn't check if he really got a spot in memory which could be are issue.

    Mike
    That could certainly cause a crash!

  • David Betz wrote: »
    Are you sure it's a good idea to write the file to flash memory on the wifi module? Unless someone has changed the code since I last worked on it, the flash filesystem wasn't very robust. It doesn't do any kind of wear leveling or error recovery. It was really designed to be simple and to host a static collection of files. I don't think it's a good idea to continually rewrite files. Shouldn't it be easy to just stream the file to the P2?

    Yeah, as stated this is a quick solution. I had only a couple hours to work on it, so went with the shortest path for me. I'm expecting a streaming solution to be required at some point, and if that's quick to add now I'd be all in for adding it now, but would need help to achieve that.

    The Winbond flash on those modules are rated for 100000+ writes, so provided they don't fail, the lifespan should be many years for most users, and I'd expect an update long before then to improve life. Even adding a sequential digit to the end of each filename (per session) would be a massive improvement, to spread the files across the flash, rather than replacing the same file in the same flash location each time. If nothing else, I might add that when looking at the other things iSeries suggested.

    Right now, I just wanted to share what I have. If anyone wants to use, improve or totally change things, then fantastic!


  • @VonSzarvas, @"David Betz" ,

    Think I found it. I added code to check for memory allocation errors and think I found the offending code.

    Conn req from 101.1.1.7:60138, using pool slot 2
    181854> E:M 2056
    181854> Receive Call Back out of memory
    181890> E:M 2056
    181890> Send Buffer Out of Memory

    Now how to find more memory?

    Mike
  • VonSzarvas wrote: »
    David Betz wrote: »
    Are you sure it's a good idea to write the file to flash memory on the wifi module? Unless someone has changed the code since I last worked on it, the flash filesystem wasn't very robust. It doesn't do any kind of wear leveling or error recovery. It was really designed to be simple and to host a static collection of files. I don't think it's a good idea to continually rewrite files. Shouldn't it be easy to just stream the file to the P2?

    Yeah, as stated this is a quick solution. I had only a couple hours to work on it, so went with the shortest path for me. I'm expecting a streaming solution to be required at some point, and if that's quick to add now I'd be all in for adding it now, but would need help to achieve that.

    The Winbond flash on those modules are rated for 100000+ writes, so provided they don't fail, the lifespan should be many years for most users, and I'd expect an update long before then to improve life. Even adding a sequential digit to the end of each filename (per session) would be a massive improvement, to spread the files across the flash, rather than replacing the same file in the same flash location each time. If nothing else, I might add that when looking at the other things iSeries suggested.

    Right now, I just wanted to share what I have. If anyone wants to use, improve or totally change things, then fantastic!


    Sorry, I didn't mean to be critical of your work. I just wanted to point out a possible downside to that approach. Another is that the current filesystem code doesn't reclaim the space used by files that are overwritten. You eventually have to empty the filesystem and start over if you keep rewriting the same file over and over. It would be better to find a more robust flash filesystem to use but the other options I found at the time used too much SRAM.
Sign In or Register to comment.