Shop OBEX P1 Docs P2 Docs Learn Events
A Propeller-WebTool Framework for Compiled Languages? - Page 11 — Parallax Forums

A Propeller-WebTool Framework for Compiled Languages?

18911131421

Comments

  • jazzedjazzed Posts: 11,803
    edited 2014-02-06 12:20
    msrobots wrote: »
    By now I am able to run editor and compiler from the spinneret load the needed files from the sd card of the spinneret or any other webserver supporting CORS and save source and/or binary to the sd card of the spinneret or any other webserver supporting CORS and PUT requests.

    If I could load another propeller from the spinneret the IDE on a prop-plug is doable. But by now my spinneret code is quiet big (just 200 longs left for stack) and I need to remove stuff to gain space for a loader.

    Yes, getting/saving source and binary files directly on the Propeller SDcard solves many issues. It's great that you are willing to do that.

    Do you have a Digi S6B Wifi XBee? That offloads all of the network support from running in Propeller. ActivityBoard (ActivityBot) with S6B is a key target platform.

    At the moment PUT/GET/POST are blocked by the S6B server, but other commands get through. Digi says that an S6B update will be available that allows GET, etc... in a few months.

    I'll ask Ken about supporting this effort with hardware for you and Heater. What hardware do you need?
  • msrobotsmsrobots Posts: 3,709
    edited 2014-02-06 12:35
    I do have a activity board, but have not tested It for the bug the first versions had.

    I actually just use it to power 2 spinnerets...
    At the moment PUT/GET/POST are blocked by the S6B server, but other commands get through.

    Now you are confusing me. Without GET/POST/PUT you do not have a webserver at all?

    So the S6B is just a serial to IP-Port bridge?

    Enjoy!

    Mike
  • jazzedjazzed Posts: 11,803
    edited 2014-02-06 13:51
    msrobots wrote: »
    Now you are confusing me. Without GET/POST/PUT you do not have a webserver at all?

    "At the moment." S6B has a full web-server built-in and it eats GET/PUT/POST, but passes any other requests to the serial port. Digi will release a version that allows GET/PUT/POST to pass through to the serial port later. We use other requests for communicating now which is a fine work-around.
  • msrobotsmsrobots Posts: 3,709
    edited 2014-02-06 14:55
    Does It allow PUT requests to the build in webserver? If so could we use It as a storage for files?

    I found the Example of @Mike G. to use @Chips PropellerLoader from the spinneret to load/program another propeller connected to the spinneret via the serial bootloader.

    About 88 Longs. This is doable I think. I may need to remove REST and/or PROPFIND support. If this works the spinneret is some sort of PropPlug with build in web based Editor/Compiler for SPIN/PASM. No Internet required. all included.

    Now I need to track down your? JavaScript Terminal I stumbled upon lately. FourPortSerial is already part of the spinneret so after programming I could use pin 30/31 of the attached propeller for a serial terminal in the browser .

    Funny times ahead.

    I do not know how you are planning to program the activity boards over WIFI. The propeller serial bootloader does not use standard baudrates and is very timing sensitive. I doubt it will work over WIFI even if RTS or CTS is available for reset.

    I have played around with Xbees 2 years ago so I do have Xbees and adapter for usb / breadboard. So if you send me a S6B I guess I could handle the additional clutter on my desk. It is not that big I guess.

    Anyways I first work on the wired version without WIFI. Just plug spinneret into network and Propeller board to be programmed into spinneret. Then you can program it from any web browser able to reach the spinneret. That is what I will try as soon as it is weekend.
    @Heater. wants to look into getting propellerload running in JavaScript using local usb ports. This may work with ChromeBooks and even Chrome Apps running in the Chrome Browser. @Heater. has some cool ideas.

    If you have some sort of protocol to communicate with the S6B to allow me to send a binary to be saved on sd or whatever I am sure JavaScript in the browser is able to handle this.

    Enjoy!

    Mike
  • jazzedjazzed Posts: 11,803
    edited 2014-02-06 15:42
    msrobots wrote: »
    Does It allow PUT requests to the build in webserver? If so could we use It as a storage for files?
    I'm not sure. The built-in server is kind of mysterious. It's billed as a cloud server. Problem with the internal storage is the flash write limit is on the order of 20K writes. We really have to use the propeller attached SDCard for "served files."

    msrobots wrote: »
    Now I need to track down your? JavaScript Terminal I stumbled upon lately. FourPortSerial is already part of the spinneret so after programming I could use pin 30/31 of the attached propeller for a serial terminal in the browser .
    It is here: spinaweb.googlecode.com. It uses long polling, so the responsivness is not great. I wanted to change it to use CGI push, but I've been very busy.

    msrobots wrote: »
    I do not know how you are planning to program the activity boards over WIFI. The propeller serial bootloader does not use standard baudrates and is very timing sensitive. I doubt it will work over WIFI even if RTS or CTS is available for reset.
    Ya, most of us are familiar with that problem :) David Betz has a loader for small files though ...

    msrobots wrote: »
    If you have some sort of protocol to communicate with the S6B to allow me to send a binary to be saved on sd or whatever I am sure JavaScript in the browser is able to handle this.
    At the moment there are problems in David's loader with loading files bigger than about 1KB. I've suggested changing the load mechanism from a single binary file to using several chunks ... until we get more support from Digi on how we can get RTS/CTS to work correctly.
  • Heater.Heater. Posts: 21,230
    edited 2014-02-06 21:01
    msrobots,
    I think openSpin needs to run/init direct after webworker start.
    Remember that the webworker page is only an experiment so far.

    Certainly I want to get rid of all those buttons. Ultimately we only want "Compile" which will start a web worker, get the compilation done, and kill the web worker. Or whatever it takes.
    msgType 'file' needs to put the file direct into the FS - no need for a variable file in the worker.
    Quite so. That "file" thing was just a quick hack to get it to work at all. Certainly I want to be able to send and receive files from my web worker as at will. Directly in and out of it's FS.
    OpenSpin has a command line option to list the files needed by the top object. Using that I will be able to GET the needed files from sd card or external libraries on a webserver supporting CORS. So Libraries - here we come!
    I think we have a chicken and egg problem here. In order for openspin to tell you all the files a build needs it needs to be able to access all the files!

    This means the webworker has to have all the files before you start.

    I guess we could hook into the file open calls and request files that are to be opened. Somehow I'n not attracted to that idea,

    I was imagining building a multi-file project would need some kind of project definition that would list all the required files. They could be fetched from an on-line library. Or from local store. And so on. This way the web worker can be sent all the projects files prior to running.

    Yesterday I started changing the web worker messaging system. Instead of having lots of message types like "get_this", "put_that", "compile_the_other" etc etc I wanted to make it simpler and more generally useful.

    I thought I'd take a lesson from the web and its RESTfull API and CRUD ideas.

    "CRUD" = Create, Read, Update, Delete.

    So there are now only the four message types for CRUD operations and a fifth one for responses. Which I have called the "YES" type even though a response could be negative or failure. This way I can call it a CRUDY interface :)

    Each message gets a "resource" field which looks like a web URL, just what you need for the file name parameters.

    And a "body" field which can holds file content or other data for the message.

    You might like to take a look at the new code. It's not totally sorted yet.
  • jazzedjazzed Posts: 11,803
    edited 2014-02-07 07:24
    Heater. wrote: »
    I was imagining building a multi-file project would need some kind of project definition that would list all the required files.
    That won't go over very well with the spinnyistas. I think it's a perfectly sane approach, but what do i know?
  • Heater.Heater. Posts: 21,230
    edited 2014-02-07 07:40
    Jazzed,

    I'm in two or three minds about this.

    On the one hand Spin was obviously designed with the idea that all the object dependencies are specified in the source with OBJ sections. So the idea of needing another external project configuration file seems to run against the grain. I like the simplicity of this.

    On the the other hand as soon as you can set object search paths in your dev tool you are necessarily in the world of project files or make files or whatever.

    I forget what was on the third hand.

    Anyway. There are a few solutions:

    1) Use a hammer. Run the compiler and check for missing file errors. Fetch those missing files from wherever. Keep doing that until it builds.

    2) Create a simple parser that finds the OBJ sections and fetch whatever is specified. Recursively. Then build the thing with openspin.

    3) Have a project file that lists all the dependencies. This could of course be generated from 1) or 2) above initially.
  • jazzedjazzed Posts: 11,803
    edited 2014-02-07 08:20
    Option 2 would be easy. By the time the Javascript has parsed the objects, the source would be loaded. :)
  • Heater.Heater. Posts: 21,230
    edited 2014-02-07 08:54
    Quite so. Parsing and loading can be going on at the same time. Cool.
  • msrobotsmsrobots Posts: 3,709
    edited 2014-02-07 13:49
    I could not wait for the weekend and integrated a webworker into the editor.

    Openspin.js is now the same as @Heaters example from yesterday Editorww.js the new webworker.

    Enjoy!

    Mike
  • Heater.Heater. Posts: 21,230
    edited 2014-02-07 15:49
    Cool.

    What did you think about my new worker message interface?

    I have just been reading around CodeMirror. Seems you can edit many files in one instance of the editor. It has a swapDocumet method. Or something like that.
  • msrobotsmsrobots Posts: 3,709
    edited 2014-02-07 16:37
    I just used the first one and did not look at your new creation yet.

    My compile-button creates a new editor as console output. and then plays PingPong with the webWorker thru messages.

    I did remove most of your messages and added new ones. See Editorww.js

    Before running the compiler I add all open spin files to the file system - If you have all needed files open in a TAB then you can compile top object and everything works.

    There is some debug hook to show files opened by emscripten, but - I tried - it does not help. It just shows files opened successful. Well. Then we do not need it.

    I made the console window green. Liked your retro feeling of WebDemo3...

    Enjoy!

    Mike
  • Heater.Heater. Posts: 21,230
    edited 2014-02-09 02:35
    A small steps forward a big set back today.

    openspin.js is now maintaining multiple documents in a single editor window. I use a different technique than msrobots. Instead of creating an editor instance for each file I create a CodeMirror "document" object for each file. Document objects can then be swapped in and out of the editor with the "swapDoc" method. This is the technique suggested by CodeMirror here :
    http://codemirror.net/demo/buffers.html

    Now you can't actually see that in the on-line demo as there is no buttons to swap documents yet. But the mechanism is in place.

    For the first time openspin.js has built a Spin program with multiple source files. How ever I have hit a snag.

    Firstly I could not get CodeMirror to understand the source of the fullDuplexSerial4port demo package. Those files are little edian UTF-16 with two BOM bytes at the beginning. For some reason those upset openspin.js so had to be removed from the source files.

    Then we have a little difficulty reading some of the content. This for example:
    :bit                    add     rxcnt,bit_ticks       '1 bit period
    ...
    ...
                            djnz    rxbits,#:bit          'get remaining bits
    
    Fails with an error about a missing ":bit" variable. Which is odd because there are similar cases like that in the source, ":bi1", ":bit2", ":bit3"
    When I edit ":bit" to "bit0" there is no more complaining about that.

    The bomb shell is that the compilation fails with this error :
    Compiling...
    demo1_FDS4port.spin
    |-FullDuplexSerial4port.spin
    |-dataIO4port.spin
      |-FullDuplexSerial4port.spin
    demo1_FDS4port.spin(1:1) : error : Invalid object file ullDuplexSerial4port.OBJ
    
    This kind of implies that whatever openspin has written out as a compiled spin object file is not correct or what it reads back from the emscripten files system is corrupted!

    I'm at a loss for now.

    You can play with it here still:
    http://a.linuxsecured.net:3000

    msrobots can Edior5 compile demo1_FDS4port.spin? Perhaps I'm handling my text encoding and binary stuff incorrectly.
  • jazzedjazzed Posts: 11,803
    edited 2014-02-09 08:24
    UTF-16 causes me no end of grief in multi-platform. Someone should be taken out to the woodshed for using that.

    The only thing that really works is to convert all UTF-16 files to UTF-8 (or open as one and save as the other).
  • dgatelydgately Posts: 1,630
    edited 2014-02-09 10:22
    Heater. wrote: »
    You can play with it here still:
    http://http://a.linuxsecured.net:3000/

    Getting "Can't find server" error from your URL, today.


    dgately
  • Heater.Heater. Posts: 21,230
    edited 2014-02-09 12:11
    Oops, some how that link was a bit mangled. I it fixed now.
  • Heater.Heater. Posts: 21,230
    edited 2014-02-09 14:10
    Jazzed,
    UTF-16 causes me no end of grief in multi-platform. Someone should be taken out to the woodshed for using that.
    Totally agree.

    Also all those who ignore case sensitivity in file names.

    Also all those who put spaces and odd characters in file names.
    The only thing that really works is to convert all UTF-16 files to UTF-8 (or open as one and save as the other).
    Yep. I started playing with that. Ended up loading stuff into the editor that looked like Chineese!

    Everytime I start looking into character encodings I get headache. For example JavaScript is internally working in UTF-16. Except whem it's UCS-2 of course.

    http://mathiasbynens.be/notes/javascript-encoding
  • Heater.Heater. Posts: 21,230
    edited 2014-02-10 01:57
    So:

    I converted all the UTF-16LE Spin sources from fullDuplexSerial4Port to UTF-8.
    I changed my XMLHTTPRequest to not ask for binary data.
    I removed my half baked UTF-16 as binary array and string handling, all is now just "string"

    My web worker still chokes on that ":bit" label,

    When I change that to ":bit0" compilation goes ahead but still fails with this out put:
    Propeller Spin/PASM Compiler 'OpenSpin' (c)2012-2013 Parallax Inc. DBA Parallax Semiconductor.Compiled on Jan  9 2014
    Compiling...
    demo1_FDS4port.spin
    Warning: Enlarging memory arrays, this is not fast, and ALLOW_MEMORY_GROWTH is not fully tested with all optimizations on! 39096320,33554432
    |-fullDuplexSerial4port.spin
    |-dataIO4port.spin
      |-fullDuplexSerial4port.spin
    demo1_FDS4port.spin(1:1) : error : Invalid object file ullDuplexSerial4port.OBJ
    Line:
    
    
    Offending Item: 
    
    

    That "Warning" comes from Emscripten complaining that I did not give it enough memory to start with. It always been there when things are working. I will rebuild openspin to remove that. It may help but I doubt it.
  • dgatelydgately Posts: 1,630
    edited 2014-02-10 10:59
    Heater. wrote: »
    demo1_FDS4port.spin(1:1) : error : Invalid object file ullDuplexSerial4port.OBJ
    .

    It's also a little weird that it states: error : Invalid object file ullDuplexSerial4port.OBJ With, "ullDuplexSerial4port.OBJ". Why is it losing the "f" in the file name? I had an issue with compiling this same project in iOS until I realized/remembered that iOS's file system is case-sensitive and 3 of the sources used "FullDuplexSerial4port" to define the object whose file has an lowercase "f".

    dgately
  • Heater.Heater. Posts: 21,230
    edited 2014-02-10 14:14
    dgately,

    Oh yes. The OBEX is riddled with code that will not build on Linux because of case sensitivity in file names.

    For sure I have fixed the OBJ declarations already, they build fine on my Linux PC now, but you are right that missing "f" is a suspicious coincidence.

    There is something else weird going on, or perhaps the same thing.

    If I have a label ":bit" in the code an instruction like "djnx x, #:bit" it will fail to compile. It complains of an undefined variable ":bit".

    But if I change the label to ":bit0" all is well.

    I have tried this with other examples ":f", ":fr", ":fre" all fail but ":fred" compiles.

    Weird!
  • Heater.Heater. Posts: 21,230
    edited 2014-02-10 14:28
    Hmm..the demo1 does compile if I remove all reference to "dataIO4port" from it.

    Still does not like the ":bit" label though.
  • Heater.Heater. Posts: 21,230
    edited 2014-02-10 16:01
    Yay! fullduplexSerial4Port now compiles in the browser !

    Turns out we have a bug in openspin.

    In PropellerCompiler.cpp there is code that makes pointers to bytes like so:
    unsigned char* pData = &(g_pCompilerData->obj_data[g_pCompilerData->obj_offsets[nFile]]);
    
    Never mind how complex that looks, the important thing is that pData is a pointer to bytes.

    This pointer then gets used to access 16 bit words like so:
    short vsize = *((short*)(&pData[0]));
    
    Again, more complex than it need be but the important thing is that it is accessing a byte array as words.

    What if pData is pointing to an odd address?

    Then the word access is unaligned. This is not allowed on some architectures like some ARMs and Emscripten by default. It will also fail on machines that have a different byte order for shorts, integers etc.

    Never mind, we set the magic Emscripten compile option "UNALIGNED_MEMORY=1" and poof it starts to work!

    See link given above.

    A better solution will be to fix openspin. Unaligned access slows down emscripten.

    There is still a problem with the ":bit" label though....
  • msrobotsmsrobots Posts: 3,709
    edited 2014-02-11 12:37
    @Heater,

    I see that I am not alone with my problem!

    I decided to go for the 'Hammer' method and rerun openspin after providing the missing file.

    If a file is missing I first try to load it from the same path as the source to be compiled. Say compiling http://192.168.1.117/source/Socket.spin misses W5100.spin. I look for http://192.168.1.117/source/W5100.spin first.
    Then I try to load from a folder '/Library/ in the root of the server serving the source. So I look for http://192.168.1.117/Library/W5100.spin next.
    If still not found I do look at http://parallax.msrobot.net/Library/W5100.spin. where a copy of PropTools Library and Demos files is located, ready to access with CORS enabled.

    Does work like a charm - except that missing first letter of files. Or a message stating 'To Many Files'

    The thing with local labels is also here. To me it looks like local label need a minimum of 4 characters?

    Can you post the new openspin.js?

    Enjoy!

    Mike
  • Heater.Heater. Posts: 21,230
    edited 2014-02-11 13:44
    msrobots,

    I think the unaligned access thing will fix some of your problems.

    However the "Too Many Files" thing I am sure will happen if you run openspin's main() too many times without totally reinitializing it's environment This is because the file count is a global variable in openspin which does not get reinitialized, to zero, when you run main(). You have to reload the whole process, as happens when you run it on a PC normally.

    I will probably have a go at removing all of openspin's global variables at some point, there are many more, so as to avoid these kind of problems.

    No need for me to post the new openspin.js you only have to ask your browser. It's here:
    http://a.linuxsecured.net:3000/openspin.js
  • msrobotsmsrobots Posts: 3,709
    edited 2014-02-11 14:24
    @Heater. Yes. Thank you.

    Could have tried that link before. Sort of obvious. There are days like that.

    Sad to read that with the file count. I currently just add the one missing file to the FS and then run Main() again. (All other files still there)

    I MAY need to change that, need more tests. No clue about Unicode/utf-8/utf-16. In windows PropTool warns you when you try to save a file containing special characters (like Prop Font resistors or so).

    The editor reads and save both versions. I've updated the spin-theme to support background color for sections. Works but empty lines get no style. as soon as a character is entered (even space) background is OK.

    I guess Phil's RegEx stuff is skipping complete empty lines somewhere instead of returning style 'normal'.

    Now replacing OpenSpin.js and testing. Will post results and Editor.zip soon.

    Enjoy!

    Mike
  • Heater.Heater. Posts: 21,230
    edited 2014-02-11 14:50
    msrobots,

    Imagine you have C code like this:
    int x = 0;
    
    int main(int argc, char* argv[])
    {
        x =  x + 1;
    }
    
    That x is initialized to zero when the process is loaded. Which is what we do when we call openspin() in the browser.

    Then we call main() and x becomes 1. Then we call main() again and x becomes 2. Etc etc.

    The openspin source has a lot of this kind of stuff.

    Of course we have wrapped the emscripten generated code in the openspin() function so as to:

    1) Avoid that global initialization problem.
    2) Avoid the memory leak in Chrome.
    3) Get access to FS.

    But then it did not help with 2) and I put in the web worker to:

    1) Really fix any possible memory leak.
    2) Keep the page responsive while the compiler is running.

    I feel I might be going round in circles here. Sometimes your avatar is just too appropriate :)

    I'm guessing the ":bit" vs ":bit0" thing is an encoding issue somewhere. All my files go into a CodeMirror document object before finding their way to the web worker so there is room for mishap there. Even if I read that CodeMirror does not care about encodings.

    All my Spin sources are now pure ASCII with some illegal symbols detected and removed by the iconv program. Somewhere along the line they must become UTF-16/UCS-2 which is JavaScripts internal representation. That may be a problem somehow.
  • msrobotsmsrobots Posts: 3,709
    edited 2014-02-11 15:58
    OK.

    This needs to be addressed then. Will think about restarting webworker every turn around and keep a second file list for missing files.

    But the new OpenSpin.js did the trick for me. Attached Editor7 has now access to library files and finds all included objects. Compiles to binary or eeprom.

    If you have a file open (even unsaved) it will be the primary source else Editor looks in the current source path for file, else in root folder /Library/, else at http://parallax.msrobots.net/Library/ (PropTool Library)

    PropTool Demo files are also available at http://parallax.msrobots.net/Library/_Demos/

    @Heater. Would you mind if I update your Editor5 to Version 7? Or do you want to do it by yourself?

    And can you maybe look at Phil's regEx stuff and my white empty lines?

    Enjoy!

    Mike
  • Heater.Heater. Posts: 21,230
    edited 2014-02-12 00:35
    msrobots,

    Sounds cool.

    What do you mean by "I update your Editor5 to Version 7"

    I was thinking that if you have a new improved Spin mode you could provide the spin.js and related css as separate files that anyone can use. Ultimately the Spin mode should be given to the CodeMirror guys to include in there downloads.

    I have to say that regular expressions give me severe migraine :)

    I don't have much time to work on this, I thought my next effort would be to try and track down that ":bit0" label issue.

    And then there is that thing about being able to Program a Prop directly from a Chrome Web App using the serial api.

  • Cluso99Cluso99 Posts: 18,069
    edited 2014-02-12 22:10
    I came across this today and wondered if any of the info was of any use to you. Its downloading via WiFi and the internet to an Arduino. Here is the link
    http://www.lucadentella.it/en/2014/02/10/yun-sketch-uploader/
Sign In or Register to comment.