Given that you started from a x86 assembler implementation of the Spin compiler, there is no formal definition of the Spin language and the Spin interpreter byte codes are not documented I think you were probably better off using C/C++. A brilliant job by the way.
Otherwise I'm leaning toward the idea that doing it in JS would have been a lot easier, quicker to develop and the result would have run at close enough the same speed.
JS is constantly surprising me. Only a couple of years ago I would have said it was madness to even think about doing this kind of thing.
These passed couple of weeks I created a process in JS that parses huge piles of data in csv format (with other bells an whistles thrown in) from MQTT and emits XML or JSON over a socker/websocket as a replacement for a C++ program that does the same. Well the lines of code went down from about 5000 to about 1000 and the performance is about the same, which is critical as we are using it on slow ARM based embedded systems. Development time was a lot less.
It's getting to the point that I don't have any need for C++ any more!
(Well, except for building JS engines that is )
stunning work. I was not able to do anything on the Editor but will move the latest version of OpenSpin.js from @Heater over to the editor.
Just need to add the return-code handling. A small change to OpenSpin.js.
Next goal on my list is to get directory browsing working. The directories /Library/ and /propeller-w5200-driver/ on parallax.msrobots.net allow directory browsing and CORS support.
So even if you run the editor from your own webserver (or local FS with IE) you still can compile files on my server and use the Libraries / Demos there.
I do not even played with node yet. This is all standard JavaScript, CSS and HTML. No external files. All included.
I am actually pretty proud of my 'Upload Local File' Menu Link. And nobody mentioned it here. File Upload as simple as it gets.
So no new Editor available by now, maybe I should skip Version13 entirely and put out 14 next ...
Weekend is coming.
For some reason (Windows ate?) I can not program my spinnerets from the propTool since a couple of days. Serial Terminal works fine, but no joy programming.
Which is sad since I included code to program another propeller from the spinneret and can not test it. Or the PropTool is jealous. Will figure out or just use another computer.
I am actually pretty proud of my 'Upload Local File' Menu Link. And nobody mentioned it here. File Upload as simple as it gets.
Actually, I was rather taken with that. Always wondered how uploading local files was done. Didn't have time to look at how you did it till now.
The "FilerReader" API, easy!
But I like the sneaky way you made the input element hidden and then fired up it's file input dialogue by calling it's click() method from a menu item. That is cool.
Don't forget there should be no reason to keep starting and stopping the web worker any more. I have been hammering on that quite a bit and I can't get it to fail or eat memory any more.
Sneaky. Yes. That might fit. The download part was actual even more complicated to figure out.
Don't forget there should be no reason to keep starting and stopping the web worker any more. I have been hammering on that quite a bit and I can't get it to fail or eat memory any more.
I might skip on that for the next release - do not want to change things too fast. But I would like to get the Object List into the top left frame and will need to touch the webworker state machine anyways...
@Heater. This is quite funny. I am German. While working from California for a German company in Vechta (North Germany) I followed your progress with Roy daily.
I need to work nightshift in CA (and Germany) to minimize disturbances on the system while updating.
You are a British and live somewhere so far up north in Europe that I start to shiver here on my porch just thinking about it.
@Roy is somewhere near me (relatively) but working on a complete different time zone as I do.
So one of us is probably sleeping, one on his paid 'Day' Job and one has time to tinker with the Propeller.
And there is a thread about missing collaboration?
Wow.
I am very impressed with what you all have accomplished so far, using JS and Webworkers to compile code in-browser, and especially with CodeMirror3 and the block highlighting.
With the way that this has been set up, it would be fairly trivial to serve the Editor from a Spinneret or RaspberryPi, or even write a simple web plugin for Chrome for development on an attached propeller.
yes. The spinneret-msrobots branch of the spinneret repo on google code does exactly that.
Support for GET, PUT and directory browsing. I may have even the ability to program another propeller from the spinneret, but that part is not working yet.
@Heater. suggested Chrome Apps also. There may be the possibility that as Chrome App we can access the local serial/usb ports. And simply run some loader written in C and compiled to JavaScript to program a propeller.
I was thinking about your technique for finding all the files that a Spin program needs: Compile top object, see what sub objects it fails on, download the missing file. recompile, and so on and so on. All very long winded and slow.
I thought what we need is a pre-parser that runs in the main thread, not the worker, that scans a Spin file, reads all the sub-object dependencies and the fetches those. Recursively. When it's done all the files can be sent to the worker for a super quick compilation.
Attached is my first effort at such a parser. The "parse" method takes a file name and the content of the file as a string. It returns an object listing all the Spin sub-object file names. The test, at the end of the file, which runs under node.js, shows how to fetch all the sub-objects recursively.
Also in the returned object are all the public and private methods in the given object, together with their parameters. With that we can build a method browser like the one in PropellerIDE ! (It's much faster than the one in ProplellerIDE, there is a challenge Roy).
It's probably not perfected yet but I though you might like to take a look. One thing it needs to be able to do is return the object tree for a Spin program instead of just a flat list of files.
To install emscripten I had to install something called 7-zip and - haha - I can open tgz. So I was able to look at your parser.
Nice. smaller then I thought it would be.
This will speed up things very much. My brute force way to rerun everything to get the next file is very time consuming.
I have directory browsing ready to go and your parser is exactly what I need for loading. And the method list can fit perfectly in the bottom left.
If I remember correctly openspin can show the object tree via command line option. But since the parser is going thru the files anyway ...
Hopefully I have some time today to pull in the latest openspin.js.
And yes, WebServer_W5100_RTC is a good test. Most of the code is from @Mike G. and he did a very structured approach in 'layering' of the sub-objects to be able to move code between spinneret (WizNet 5100) the Parallax Eth. Adapter (WizNet 5200) and the new Spinneret 2.0 (WizNet 5500?).
So we have multiple objects included multiple times and later pruned by the compiler. If this displays correctly it will probably display every object correctly.
Currently the spin parser rumages though all the source files and spits out an object containing info on all the sources, file name, PUBs, PRIs in a "flat" manner. That is at least good to get all the required files loaded at start up.
Having done that of course it seems it would be better to be able to get the source "tree" structure out of it. Actually that is what it did originally but then I decided to allow for fetching files in an asynchronous manner which kind of messed up the recursive tree idea.
There is a issues with comments in the source. Currently I don't deal with brace comments at all "{...}" so any commented out methods will appear in the list. If anyone has a nice regexp that will remove brace comments that would be great.
I'll have to fix that. Might not be this week though.
I can see now how tempting it is to resolve both tasks with one parser. But I disagree with it. We have two entirely different use cases.
case 1
- we need to resolve all files needed to compile a given 'current' file, the one we need to compile NOW.
- for this your recursive approach is fine. I just need to enhance the loadFile thing to check if the file is already loaded.
- but here we only need the OBJ section, nothing else.
- the output may/can be used to show the Object Tree after compiling also. Like the PropTool does.
case 2
- we need a method-list for a current OPEN (and maybe changed) file with CURRENT line numbers to use as jump target for the Editor Tab we are working in.
- this needs to run on inserts/deletes of lines and also on other edits of the source. So quite often but just for this ONE file.
- maybe even as a timer event like the highlighting does.
optional we may need some sort of cache for this info to support something like subobj dot (show methodlist) if parsing is to time consuming.
the current code completion is VERY simple and just parses the current source for matching words. But thanks to @PhiPI we have some options there.
I'm tired and not sure if I understand you correctly. But:
Don't confuse the parser object with the test harness included in the same file.
The parser JS object only works on one Spin source file at a time.
Give it a Spin source as a string and it returns a list of all the sub-objects specified in OBJ sections. It also makes available all the PUBs and PRIs in the given file together with their parameters.
That's it. No recursion. No fetching. Done.
What happens next is up to the user of the parser, for example your cases:
Case 1) If you want to know the whole programs object tree then you have to fetch all sub-objects from the list returned by the parser and call the parser on each one. Then fetch and parse all of the sub-objects returned from those. And so on recursively. At the end of that process the parser has accumulated all PUB's and PRI's in the program as it goes.
Of course whilst you are doing that you can load the files into the editor and send them to the web worker at the same time.
Case 2) If you want a list of all the PUBs accessible from a given Spin file then you have to fetch the listed sub-objects and call the parser again for each one. It will then return other lists of sub-objects, which you can ignore, and all the PUB's you are interested in. No recursion. Only one level deep.
Of course if you happen to have loaded the files already then just feed them in from memory (cache), no need to load them again. Beside they may have been edited since the last download.
Hope all that made sense.
Parsing is not time consuming. 200ms for the full recursive descent on your web server objects. On my slow old and64. Ten time quicker than the C++ parser that does the same job in Propeller IDE!
Code completion will be very fast if you only search the PUBs in the object returned by the parser.
I was thinking about building support for Cases 1) and 2) into the parser object itself but ran out of time. Not sure if we need it really given what you have said.
Some folks on the forum of have been saying JS is interpreted and slow. We have to show them what it can do
I notice you've been working on a Spin parser. You might be interested to know that Roy has added a feature to OpenSpin that outputs the names of all of the PUB methods and their parameters as well as any CON symbols. He did this to support my spinwrap program but it sounds like it could be of use to your IDE as well. You might be able to get him to add PRI methods as well if you need those too.
Lets say you want to compile Spin file A. Here is what happens:
1) Download A
2) Send A to the web worker for compilation.
3) Worker says A needs B,C, D...
4) Down load B, C, D...
5) Send them to the worker for compilation.
6) Worker says B, C, C need X, Y, Z...
All of which is slow and tedious.
Enter the Javascript pre-parser that can scan and fetch all the required files. Whist waiting on downloads the compiler could be busy compiling in the web worker process in the back ground.
This was important when the JavaScript version of openspin was very slow. Now that we have optimized it and it is ten times faster perhaps it's not such a big deal.
Hmmm...thinks, looks like the preparser is still a win when editing code and you want the list of available methods updated in real time.
There is a issues with comments in the source. Currently I don't deal with brace comments at all "{...}" so any commented out methods will appear in the list. If anyone has a nice regexp that will remove brace comments that would be great.
> data = "All {{work}} and no {play} makes Jack a dull boy."
'All {{work}} and no {{play}} makes Jack a dull boy.'
> data = data.replace(/(\{\{[^\}]*\}\})|(\{[^\}]*\})/g, '');
'All and no makes Jack a dull boy.'
>
Looks like the regexps I was trying that did not work. But then all regexps look the same to me
Great thanks you. That seems to work:
<snip code>
Looks like the regexps I was trying that did not work. But then all regexps look the same to me
Aah well regexps are black magic indeed. Never was really that good with then and sure enough I just found out it will not work for the following case:
"All {{work {and} no}} {play} makes Jack a dull boy."
Been trying some solutions, but they are not working yet
We might just have to do it the good old fashioned way:
function bracket (s, i) {
var start, end, depth = 0;
i = i || 0;
for (i = 0; i < s.length; i += 1) {
if (s[i] === "{") {
depth += 1;
if (start === undefined) {
start = i;
}
} else if (s[i] === "}") {
depth -= 1;
if (depth === 0) {
end = i;
front = s.slice(0, start);
back = s.slice(end + 1);
s = front + back;
s = bracket(s, start + 1);
return (s);
}
}
}
if (depth !== 0) {
s = s.slice(0, start - 1);
}
return (s);
}
var s = bracket ('"All {{work {and} no}} {play} makes Jack a dull boy."');
console.log(s);
I do have your parser now integrated into the editor. Works almost as expected.
But I need to move the whole loading process to a webworker. Loading all files takes longer then compiling them.
So no console output after start until all files loaded. Was tired yesterday so no new Editor yet. Maybe I can get away with some timer events to update console...
But basically it works like advertised. Loads recursively needed files and then runs openspin once.
Thanks for commenting on my language thread. I thought I'd pop over here to see what you are up to. The link on post 401 is pretty nifty.
Actually, what you have working is very clever. The 'standard' way of running spin, C, Basic etc is you download an IDE, install it, then test it. Maybe that takes 5 mins. Sometimes that takes an hour or more. Your system works in a few seconds. That is a huge improvement.
So maybe that can be the start of a different way of programming the propeller?
Instead of starting with a blank PUB Main and adding in OBJ in the text, maybe you could also have a GUI alternative where you start with a blank screen, and drop things on that screen. Some things, like buttons, you would then click on and add some code when that button is clicked. Implied in that would be existing propeller objects to do the clicking and to drive the screen, eg a mouse driver, vga driver, or i2c touchscreen object. Not really any new code there, just a different way of using existing code.
Then there is another concept in some IDEs, where you drop objects on the main screen, but they are invisible when compiled. Timers are an example - you drag a little clock icon to the form, or a serial port.
That was written in Spin using the Proptool, but it could have been written faster and more intuitively by dragging and dropping elements in a js program.
you can resize the frames with dragging. you can drag some filename into an open editor. appears as text.
you can drag a FILE from your desktop into a editor window and it will get uploaded. Alas it messes up with utf-8/Unicode sometimes.
@Heaters first demos were about a new Interface also. I just try to be as PropTool-like as I can for now.
What is missing is a way to program a propeller from the editor. @Heater thinks it may be possible to use Chrome(apps?) and the regular local serial/usb port.
I am trying to get my spinneret program another board. Then we would have a PropPlug with build in IDE.
And some other thread tries to get loading a propeller via Parallax wifi module running.
But all of this is still in flux.
I do like the Idea of VB style editing. But all my Propeller stuff uses mostly serial output since video is eating to much precious RAM in the propeller.
If you have Ethernet on the prop (like the spinneret) you can build a very rich GUI as HTML and run it from the browser. no RAM on the prop needed.
so much stuff to play with and so less time to play.
BTW. I did not do much before in JavaScript (besides the usual link stuff in html). And then @Heater got me interested in doing this Editor thing. JavaScript is pretty easy to learn and way more powerful as I thought it is. Funny Language.
But I need to move the whole loading process to a webworker. Loading all files takes longer then compiling them.
So no console output after start until all files loaded. Was tired yesterday so no new Editor yet. Maybe I can get away with some timer events to update console...
No. You are doing your downloading wrong:)
You are using synchronous XMLHTTPRequests that block the browser until the data has arrived. You should make them asynchronous and us callbacks to handle the data when it arrives. That way the browser remains responsive at all times.
You will notice that is what my parser test function does as it reads from files. Same idea. That's why I made the test asynchronous.
In my opinion this is the way to go because it simplifies things. You want the files in the editor first. The editor is in the main thread so just fetch them there. Then the interface to the worker is only, send files, compile, get binary and errors.
I don't think timer events to update the console during your download will work. The events won't happen until your down load loop is finished.
I do like the Idea of VB style editing. But all my Propeller stuff uses mostly serial output since video is eating to much precious RAM in the propeller.
True - I got a bit spoiled with 1 meg of graphics ram for the touchscreen.
Ok, here is another idea. Let's ignore all the buttons and pictures for the moment, and think about a programming language that you write in a graphical style. Consider a serial and keyboard object, each represented with an icon. Drop those objects onto the main form. But these are invisible objects when the code is running, and you use the main form in the way vb.net uses the form to hold serial objects and timers. The form doesn't appear when the program is running, because there is no graphics object. But it is a way of writing code that is familiar to users of GUI IDEs.
So you grab a keyboard object and a serial object and drop them on the main form.
Then you click the serial object and up comes a text window, and you write one line of code:
serial.out(keyboard.key)
And even better, as you write that, autocomplete gives the object methods.
One more thing - the serial and the keyboard have some startup properties. Click once on these and the properties come up in a list on the side of the screen, and there you can put in the pins to use and the baudrate.
Most of the code now gets written automatically. Behind the scenes, it can be Spin, or C or even javascript. The syntax in that one line of code is kind of common to all those languages. In this example, when you hit the compile button, the IDE will put in the xtal frequency and PUB Main etc.
I dunno, just an idea. I was impressed with the way XOJO could write a web browser with one line of code. It got me quickly interested in the language. Maybe such similar simplicity could be useful for the propeller too?
This need not be a core feature of the program you are writing. Maybe it is in the Advanced drop down menu instead as a GUI option, and the default is text. But it could be the start of an object oriented interface. Define an object, say a serial port. It is just the standard serial port code from the objex. It has methods, which instead of having to look at the object source code, now appear in dropdown menus after the name of the object is typed. It also has startup parameters, and instead of these being a list of values that one has to look at the source code to know what order to put them in, they appear in a menu on the side of the screen.
Maybe I'm getting ahead of things, if you still are working on things like loading.
There is a simple function to download all those documents into editor doc objects:
function loadProject () {
// Load all Spin sources from server.
spinSources.forEach (function (source) {
var oReq = new XMLHttpRequest();
oReq.open("GET", "/project/" + source.fileName, true);
oReq.onload = function (oEvent) {
var arrayBuffer = oReq.responseText; // Note: not oReq.response
if (arrayBuffer) {
// Make a new Spin document in the editor like so
source.doc = CodeMirror.Doc(arrayBuffer, 'text/x-spin');
editor.swapDoc(source.doc);
}
};
oReq.send(null);
});
}
This downloads asynchronously. The browser does not stall.
Of course we need to actually get the file list from the top file to kick off with using the parser. Then parse those as they come in, in the callback, and kick off more downloads. That's what my parser test does by calling itself when it gets files.
Then as a file arrives at the call back we can also do any adding of buttons and sending to the worker etc.
Of course we need to actually get the file list from the top file to kick off with using the parser. Then parse those as they come in, in the callback, and kick off more downloads. That's what my parser test does by calling itself when it gets files.
This looks fun!
Ok, may I be so bold as to add in a design feature early in the project that might be harder to add later? If you are calling in sub objects in the compiler parser, that will result in the quickest compile. But let's say you want an IDE which is fully aware of all its objects and sub objects. It would need to be in order to do autocomplete on methods. So if it is doing that, presumably this means it has already downloaded all the objects it needs once at startup. Is there a way of holding that text in memory, so it is available for subsequent compiles?
How much memory does javascript have? Can you define an array with 10,000 lines of text?
Ok, here is another idea. Let's ignore all the buttons and pictures for the moment, and think about a programming language that you write in a graphical style.
I think you are onto something there. It's an idea that has been hinted at on the fourums over the years.
I have imagined it like this. There are three main areas on you screen:
1) An editor with your currently open Spin file. Starts out empty but for an empty OBJ section and a PUB method named after the file name:
OBJ
PUB myObject
2) An area full of icons for commonly used driver and other sub-objects. A picture of a serial port connector for FDS.A picture of a TV for a TV driver. A keyboard, a mouse and so on.
From this are you can drag and drop an object onto your editor window. This adds the object to your project as a sub-object of the open file. It can auto-edit that file adding an OBJ entry and perhaps some start up code.
At this point the editor also knows what PUB methods FDS has (and CON definitions) so it can offer those up as auto completions. It can provide a list of available methods via a pop up etc.
3) An area connected with the editor that contains icons of the sub-objects in use by the currently open file. Clicking on these might bring up a list of their methods which can be dragged into the editor. Or perhaps dialogue boxes to fill in any configuration and start up parameters the sub object needs.
I guess it might be nice to have an area with an iconic tree view of the entire programs object structure so that you can navigate around files at will.
The problems with this for me is that:
1) Any such GUI coding system is inevitably restricted. Good for beginners perhaps but there will only be so far you can go with it. For example that area of usable object icons. Where does that come from? The OBEX? Who is going to make all those icon? Who is going to ensure all those objects are easy to use in such a system? How do I create new ones of my own?
2) Perhaps because of 1) I would never use such a GUI programming environment. As a result have no motivation to start creating one.
Currently msrobots and I have had enough "fun" just getting a regular text based Prop tool going on the web page. With a lot of help from Roy and co.
You can blame Jazzed for all the fun we are having. It was his challenge to the forum
Comments
Given that you started from a x86 assembler implementation of the Spin compiler, there is no formal definition of the Spin language and the Spin interpreter byte codes are not documented I think you were probably better off using C/C++. A brilliant job by the way.
Otherwise I'm leaning toward the idea that doing it in JS would have been a lot easier, quicker to develop and the result would have run at close enough the same speed.
JS is constantly surprising me. Only a couple of years ago I would have said it was madness to even think about doing this kind of thing.
These passed couple of weeks I created a process in JS that parses huge piles of data in csv format (with other bells an whistles thrown in) from MQTT and emits XML or JSON over a socker/websocket as a replacement for a C++ program that does the same. Well the lines of code went down from about 5000 to about 1000 and the performance is about the same, which is critical as we are using it on slow ARM based embedded systems. Development time was a lot less.
It's getting to the point that I don't have any need for C++ any more!
(Well, except for building JS engines that is
stunning work. I was not able to do anything on the Editor but will move the latest version of OpenSpin.js from @Heater over to the editor.
Just need to add the return-code handling. A small change to OpenSpin.js.
Next goal on my list is to get directory browsing working. The directories /Library/ and /propeller-w5200-driver/ on parallax.msrobots.net allow directory browsing and CORS support.
So even if you run the editor from your own webserver (or local FS with IE) you still can compile files on my server and use the Libraries / Demos there.
I do not even played with node yet. This is all standard JavaScript, CSS and HTML. No external files. All included.
I am actually pretty proud of my 'Upload Local File' Menu Link. And nobody mentioned it here. File Upload as simple as it gets.
So no new Editor available by now, maybe I should skip Version13 entirely and put out 14 next ...
Weekend is coming.
For some reason (Windows ate?) I can not program my spinnerets from the propTool since a couple of days. Serial Terminal works fine, but no joy programming.
Which is sad since I included code to program another propeller from the spinneret and can not test it. Or the PropTool is jealous. Will figure out or just use another computer.
Enjoy!
Mike
The "FilerReader" API, easy!
But I like the sneaky way you made the input element hidden and then fired up it's file input dialogue by calling it's click() method from a menu item. That is cool.
Don't forget there should be no reason to keep starting and stopping the web worker any more. I have been hammering on that quite a bit and I can't get it to fail or eat memory any more.
Sneaky. Yes. That might fit. The download part was actual even more complicated to figure out.
I might skip on that for the next release - do not want to change things too fast. But I would like to get the Object List into the top left frame and will need to touch the webworker state machine anyways...
@Heater. This is quite funny. I am German. While working from California for a German company in Vechta (North Germany) I followed your progress with Roy daily.
I need to work nightshift in CA (and Germany) to minimize disturbances on the system while updating.
You are a British and live somewhere so far up north in Europe that I start to shiver here on my porch just thinking about it.
@Roy is somewhere near me (relatively) but working on a complete different time zone as I do.
So one of us is probably sleeping, one on his paid 'Day' Job and one has time to tinker with the Propeller.
And there is a thread about missing collaboration?
Play Time now. Will report later...
Enjoy!
Mike
I am very impressed with what you all have accomplished so far, using JS and Webworkers to compile code in-browser, and especially with CodeMirror3 and the block highlighting.
With the way that this has been set up, it would be fairly trivial to serve the Editor from a Spinneret or RaspberryPi, or even write a simple web plugin for Chrome for development on an attached propeller.
You guys are awesome!
yes. The spinneret-msrobots branch of the spinneret repo on google code does exactly that.
Support for GET, PUT and directory browsing. I may have even the ability to program another propeller from the spinneret, but that part is not working yet.
@Heater. suggested Chrome Apps also. There may be the possibility that as Chrome App we can access the local serial/usb ports. And simply run some loader written in C and compiled to JavaScript to program a propeller.
way to go still, but very promising ...
Enjoy!
Mike
I was thinking about your technique for finding all the files that a Spin program needs: Compile top object, see what sub objects it fails on, download the missing file. recompile, and so on and so on. All very long winded and slow.
I thought what we need is a pre-parser that runs in the main thread, not the worker, that scans a Spin file, reads all the sub-object dependencies and the fetches those. Recursively. When it's done all the files can be sent to the worker for a super quick compilation.
Attached is my first effort at such a parser. The "parse" method takes a file name and the content of the file as a string. It returns an object listing all the Spin sub-object file names. The test, at the end of the file, which runs under node.js, shows how to fetch all the sub-objects recursively.
Also in the returned object are all the public and private methods in the given object, together with their parameters. With that we can build a method browser like the one in PropellerIDE ! (It's much faster than the one in ProplellerIDE, there is a challenge Roy).
It's probably not perfected yet but I though you might like to take a look. One thing it needs to be able to do is return the object tree for a Spin program instead of just a flat list of files.
To install emscripten I had to install something called 7-zip and - haha - I can open tgz. So I was able to look at your parser.
Nice. smaller then I thought it would be.
This will speed up things very much. My brute force way to rerun everything to get the next file is very time consuming.
I have directory browsing ready to go and your parser is exactly what I need for loading. And the method list can fit perfectly in the bottom left.
If I remember correctly openspin can show the object tree via command line option. But since the parser is going thru the files anyway ...
Hopefully I have some time today to pull in the latest openspin.js.
And yes, WebServer_W5100_RTC is a good test. Most of the code is from @Mike G. and he did a very structured approach in 'layering' of the sub-objects to be able to move code between spinneret (WizNet 5100) the Parallax Eth. Adapter (WizNet 5200) and the new Spinneret 2.0 (WizNet 5500?).
So we have multiple objects included multiple times and later pruned by the compiler. If this displays correctly it will probably display every object correctly.
Enjoy!
Mike
Currently the spin parser rumages though all the source files and spits out an object containing info on all the sources, file name, PUBs, PRIs in a "flat" manner. That is at least good to get all the required files loaded at start up.
Having done that of course it seems it would be better to be able to get the source "tree" structure out of it. Actually that is what it did originally but then I decided to allow for fetching files in an asynchronous manner which kind of messed up the recursive tree idea.
There is a issues with comments in the source. Currently I don't deal with brace comments at all "{...}" so any commented out methods will appear in the list. If anyone has a nice regexp that will remove brace comments that would be great.
I'll have to fix that. Might not be this week though.
Editor13.htm & co attached as zip.
It uses the last Openspin.js from @Heater without modification. I removed my return code handling for lack of usefulness.
I do not use the new parser yet, but do NOT restart the webworker anymore. This finally keeps the memory usage down. Success.
The editor does now directory handling.
here a link to test: http://parallax.msrobots.net/Editor13.htm
Enjoy!
Mike
I was hesitant to hit the button on version 13.
But hey, that is sweet. Pick a file, hit compile, it's done. Super quick.
Well done!
Watching that other thread I think too that methods should just be needed at a per file basis. The Treeview of sub objects needs to be shown as tree.
Maybe I should take a closer look at your parser and cannibalize it a little bit.
I was hesitant too with number 13 ...
Enjoy!
Mike
after studying your parser for a while ...
I can see now how tempting it is to resolve both tasks with one parser. But I disagree with it. We have two entirely different use cases.
case 1
- we need to resolve all files needed to compile a given 'current' file, the one we need to compile NOW.
- for this your recursive approach is fine. I just need to enhance the loadFile thing to check if the file is already loaded.
- but here we only need the OBJ section, nothing else.
- the output may/can be used to show the Object Tree after compiling also. Like the PropTool does.
case 2
- we need a method-list for a current OPEN (and maybe changed) file with CURRENT line numbers to use as jump target for the Editor Tab we are working in.
- this needs to run on inserts/deletes of lines and also on other edits of the source. So quite often but just for this ONE file.
- maybe even as a timer event like the highlighting does.
optional we may need some sort of cache for this info to support something like subobj dot (show methodlist) if parsing is to time consuming.
the current code completion is VERY simple and just parses the current source for matching words. But thanks to @PhiPI we have some options there.
what do you think about this?
Enjoy!
Mike
I'm tired and not sure if I understand you correctly. But:
Don't confuse the parser object with the test harness included in the same file.
The parser JS object only works on one Spin source file at a time.
Give it a Spin source as a string and it returns a list of all the sub-objects specified in OBJ sections. It also makes available all the PUBs and PRIs in the given file together with their parameters.
That's it. No recursion. No fetching. Done.
What happens next is up to the user of the parser, for example your cases:
Case 1) If you want to know the whole programs object tree then you have to fetch all sub-objects from the list returned by the parser and call the parser on each one. Then fetch and parse all of the sub-objects returned from those. And so on recursively. At the end of that process the parser has accumulated all PUB's and PRI's in the program as it goes.
Of course whilst you are doing that you can load the files into the editor and send them to the web worker at the same time.
Case 2) If you want a list of all the PUBs accessible from a given Spin file then you have to fetch the listed sub-objects and call the parser again for each one. It will then return other lists of sub-objects, which you can ignore, and all the PUB's you are interested in. No recursion. Only one level deep.
Of course if you happen to have loaded the files already then just feed them in from memory (cache), no need to load them again. Beside they may have been edited since the last download.
Hope all that made sense.
Parsing is not time consuming. 200ms for the full recursive descent on your web server objects. On my slow old and64. Ten time quicker than the C++ parser that does the same job in Propeller IDE!
Code completion will be very fast if you only search the PUBs in the object returned by the parser.
I was thinking about building support for Cases 1) and 2) into the parser object itself but ran out of time. Not sure if we need it really given what you have said.
Some folks on the forum of have been saying JS is interpreted and slow. We have to show them what it can do
Will pry SpinParser out and insert into Editor. I see now how to use it. Nice. Will accomplish both cases.
Thanks!
Mike
Lets say you want to compile Spin file A. Here is what happens:
1) Download A
2) Send A to the web worker for compilation.
3) Worker says A needs B,C, D...
4) Down load B, C, D...
5) Send them to the worker for compilation.
6) Worker says B, C, C need X, Y, Z...
All of which is slow and tedious.
Enter the Javascript pre-parser that can scan and fetch all the required files. Whist waiting on downloads the compiler could be busy compiling in the web worker process in the back ground.
This was important when the JavaScript version of openspin was very slow. Now that we have optimized it and it is ten times faster perhaps it's not such a big deal.
Hmmm...thinks, looks like the preparser is still a win when editing code and you want the list of available methods updated in real time.
Try this one:
/(\{\{[^\}]*\}\})|(\{[^\}]*\})/gGreat thanks you. That seems to work:
> data = "All {{work}} and no {play} makes Jack a dull boy." 'All {{work}} and no {{play}} makes Jack a dull boy.' > data = data.replace(/(\{\{[^\}]*\}\})|(\{[^\}]*\})/g, ''); 'All and no makes Jack a dull boy.' >Looks like the regexps I was trying that did not work. But then all regexps look the same to meAah well regexps are black magic indeed. Never was really that good with then and sure enough I just found out it will not work for the following case:
"All {{work {and} no}} {play} makes Jack a dull boy."
Been trying some solutions, but they are not working yet
We might just have to do it the good old fashioned way:
function bracket (s, i) { var start, end, depth = 0; i = i || 0; for (i = 0; i < s.length; i += 1) { if (s[i] === "{") { depth += 1; if (start === undefined) { start = i; } } else if (s[i] === "}") { depth -= 1; if (depth === 0) { end = i; front = s.slice(0, start); back = s.slice(end + 1); s = front + back; s = bracket(s, start + 1); return (s); } } } if (depth !== 0) { s = s.slice(0, start - 1); } return (s); } var s = bracket ('"All {{work {and} no}} {play} makes Jack a dull boy."'); console.log(s);I do have your parser now integrated into the editor. Works almost as expected.
But I need to move the whole loading process to a webworker. Loading all files takes longer then compiling them.
So no console output after start until all files loaded. Was tired yesterday so no new Editor yet. Maybe I can get away with some timer events to update console...
But basically it works like advertised. Loads recursively needed files and then runs openspin once.
will update progress...
Enjoy!
Mike
That always works
Thanks for commenting on my language thread. I thought I'd pop over here to see what you are up to. The link on post 401 is pretty nifty.
Actually, what you have working is very clever. The 'standard' way of running spin, C, Basic etc is you download an IDE, install it, then test it. Maybe that takes 5 mins. Sometimes that takes an hour or more. Your system works in a few seconds. That is a huge improvement.
Ok, maybe you don't have a need yet for drag and drop. But it can be done - see http://www.w3schools.com/html/html5_draganddrop.asp
So maybe that can be the start of a different way of programming the propeller?
Instead of starting with a blank PUB Main and adding in OBJ in the text, maybe you could also have a GUI alternative where you start with a blank screen, and drop things on that screen. Some things, like buttons, you would then click on and add some code when that button is clicked. Implied in that would be existing propeller objects to do the clicking and to drive the screen, eg a mouse driver, vga driver, or i2c touchscreen object. Not really any new code there, just a different way of using existing code.
Then there is another concept in some IDEs, where you drop objects on the main screen, but they are invisible when compiled. Timers are an example - you drag a little clock icon to the form, or a serial port.
The propeller can do GUI - see http://www.youtube.com/watch?v=o6l2sNKswUY
That was written in Spin using the Proptool, but it could have been written faster and more intuitively by dragging and dropping elements in a js program.
actual I already support some Drag and Drop.
you can resize the frames with dragging. you can drag some filename into an open editor. appears as text.
you can drag a FILE from your desktop into a editor window and it will get uploaded. Alas it messes up with utf-8/Unicode sometimes.
@Heaters first demos were about a new Interface also. I just try to be as PropTool-like as I can for now.
What is missing is a way to program a propeller from the editor. @Heater thinks it may be possible to use Chrome(apps?) and the regular local serial/usb port.
I am trying to get my spinneret program another board. Then we would have a PropPlug with build in IDE.
And some other thread tries to get loading a propeller via Parallax wifi module running.
But all of this is still in flux.
I do like the Idea of VB style editing. But all my Propeller stuff uses mostly serial output since video is eating to much precious RAM in the propeller.
If you have Ethernet on the prop (like the spinneret) you can build a very rich GUI as HTML and run it from the browser. no RAM on the prop needed.
so much stuff to play with and so less time to play.
BTW. I did not do much before in JavaScript (besides the usual link stuff in html). And then @Heater got me interested in doing this Editor thing. JavaScript is pretty easy to learn and way more powerful as I thought it is. Funny Language.
Enjoy!
Mike
You are using synchronous XMLHTTPRequests that block the browser until the data has arrived. You should make them asynchronous and us callbacks to handle the data when it arrives. That way the browser remains responsive at all times.
You will notice that is what my parser test function does as it reads from files. Same idea. That's why I made the test asynchronous.
In my opinion this is the way to go because it simplifies things. You want the files in the editor first. The editor is in the main thread so just fetch them there. Then the interface to the worker is only, send files, compile, get binary and errors.
I don't think timer events to update the console during your download will work. The events won't happen until your down load loop is finished.
True - I got a bit spoiled with 1 meg of graphics ram for the touchscreen.
Ok, here is another idea. Let's ignore all the buttons and pictures for the moment, and think about a programming language that you write in a graphical style. Consider a serial and keyboard object, each represented with an icon. Drop those objects onto the main form. But these are invisible objects when the code is running, and you use the main form in the way vb.net uses the form to hold serial objects and timers. The form doesn't appear when the program is running, because there is no graphics object. But it is a way of writing code that is familiar to users of GUI IDEs.
So you grab a keyboard object and a serial object and drop them on the main form.
Then you click the serial object and up comes a text window, and you write one line of code:
And even better, as you write that, autocomplete gives the object methods.
One more thing - the serial and the keyboard have some startup properties. Click once on these and the properties come up in a list on the side of the screen, and there you can put in the pins to use and the baudrate.
Most of the code now gets written automatically. Behind the scenes, it can be Spin, or C or even javascript. The syntax in that one line of code is kind of common to all those languages. In this example, when you hit the compile button, the IDE will put in the xtal frequency and PUB Main etc.
I dunno, just an idea. I was impressed with the way XOJO could write a web browser with one line of code. It got me quickly interested in the language. Maybe such similar simplicity could be useful for the propeller too?
This need not be a core feature of the program you are writing. Maybe it is in the Advanced drop down menu instead as a GUI option, and the default is text. But it could be the start of an object oriented interface. Define an object, say a serial port. It is just the standard serial port code from the objex. It has methods, which instead of having to look at the object source code, now appear in dropdown menus after the name of the object is typed. It also has startup parameters, and instead of these being a list of values that one has to look at the source code to know what order to put them in, they appear in a menu on the side of the screen.
Maybe I'm getting ahead of things, if you still are working on things like loading.
Here is what happens in my experimental page:
There is a hard wired list of filenames:
// All sorces required to build a project. var spinSources = [ { fileName: 'Dhcp.spin', doc: undefined}, { fileName: 'NetBios.spin', doc: undefined}, { fileName: 'Sntp.spin', doc: undefined}, { fileName: 'W5100.spin', doc: undefined}, { fileName: 'Dns.spin', doc: undefined}, { fileName: 'S35390A_RTCEngine.spin', doc: undefined}, { fileName: 'Socket.spin', doc: undefined}, { fileName: 'fullDuplexSerial4port.spin', doc: undefined}, { fileName: 'S35390A_SD-MMC_FATEngine.spin', doc: undefined}, { fileName: 'Spi5100CounterPasm.spin', doc: undefined}, { fileName: 'HttpHeader.spin', doc: undefined}, { fileName: 'S35390A_SD-MMC_FATEngineWrapper.spin', doc: undefined}, { fileName: 'WebServer_W5100_RTC.spin', doc: undefined} ];There is a simple function to download all those documents into editor doc objects:function loadProject () { // Load all Spin sources from server. spinSources.forEach (function (source) { var oReq = new XMLHttpRequest(); oReq.open("GET", "/project/" + source.fileName, true); oReq.onload = function (oEvent) { var arrayBuffer = oReq.responseText; // Note: not oReq.response if (arrayBuffer) { // Make a new Spin document in the editor like so source.doc = CodeMirror.Doc(arrayBuffer, 'text/x-spin'); editor.swapDoc(source.doc); } }; oReq.send(null); }); }This downloads asynchronously. The browser does not stall.Of course we need to actually get the file list from the top file to kick off with using the parser. Then parse those as they come in, in the callback, and kick off more downloads. That's what my parser test does by calling itself when it gets files.
Then as a file arrives at the call back we can also do any adding of buttons and sending to the worker etc.
This looks fun!
Ok, may I be so bold as to add in a design feature early in the project that might be harder to add later? If you are calling in sub objects in the compiler parser, that will result in the quickest compile. But let's say you want an IDE which is fully aware of all its objects and sub objects. It would need to be in order to do autocomplete on methods. So if it is doing that, presumably this means it has already downloaded all the objects it needs once at startup. Is there a way of holding that text in memory, so it is available for subsequent compiles?
How much memory does javascript have? Can you define an array with 10,000 lines of text?
I think you are onto something there. It's an idea that has been hinted at on the fourums over the years.
I have imagined it like this. There are three main areas on you screen:
1) An editor with your currently open Spin file. Starts out empty but for an empty OBJ section and a PUB method named after the file name: 2) An area full of icons for commonly used driver and other sub-objects. A picture of a serial port connector for FDS.A picture of a TV for a TV driver. A keyboard, a mouse and so on.
From this are you can drag and drop an object onto your editor window. This adds the object to your project as a sub-object of the open file. It can auto-edit that file adding an OBJ entry and perhaps some start up code.
OBJ serial1 : "FullDuplexSerial" PUB myObject serial1.start (.....)At this point the editor also knows what PUB methods FDS has (and CON definitions) so it can offer those up as auto completions. It can provide a list of available methods via a pop up etc.3) An area connected with the editor that contains icons of the sub-objects in use by the currently open file. Clicking on these might bring up a list of their methods which can be dragged into the editor. Or perhaps dialogue boxes to fill in any configuration and start up parameters the sub object needs.
I guess it might be nice to have an area with an iconic tree view of the entire programs object structure so that you can navigate around files at will.
The problems with this for me is that:
1) Any such GUI coding system is inevitably restricted. Good for beginners perhaps but there will only be so far you can go with it. For example that area of usable object icons. Where does that come from? The OBEX? Who is going to make all those icon? Who is going to ensure all those objects are easy to use in such a system? How do I create new ones of my own?
2) Perhaps because of 1) I would never use such a GUI programming environment. As a result have no motivation to start creating one.
Currently msrobots and I have had enough "fun" just getting a regular text based Prop tool going on the web page. With a lot of help from Roy and co.
You can blame Jazzed for all the fun we are having. It was his challenge to the forum