...let's say you want an IDE which is fully aware of all its objects and sub objects...autocompletion...
We are on that case as you speak
Is there a way of holding that text in memory, so it is available for subsequent compiles?
Yes, that is what we do. The content of a file is just a string variable as far as JS is concerned. For example you can load that source code into an editor with something as simple as:
s = "....a string which is an entire Spin source file..."
editor.doc.setValue(s);
All files in use are in memory. In fact they are in memory at least twice because the actual compiler is a "web worker", a totally separate process that needs to be sent it's own copies of the files.
How much memory does javascript have? Can you define an array with 10,000 lines of text?
How much memory does your machine have. 4GB? 16GB, 32GB...Take it all if you like !
This looks fun!
Yep. OK let me hit you hard with the awesome power and unholy majesty of JavaScript. Hold tight because there are things going on here that you may never have seen in any other low level programming language like VB, Java, C++...
01) Here we have a function "parseProject" that is going to fetch the file "fileName" read it's contents and produce a data structure containing all OBJs PUBs and PRIs within. It's also going to fetch all those sub-objects and do the same to them. And the sub-sub-sub-objects and so on.
02) "fileStack" keeps a stack of files to be processed initially undefined.
04) Call "readFile" to get the file "fileName" with some "encoding" (ASCII, UTF-8 etc). Notice the third parameter here is a function definition which in turn has parameters "err" and "data". What happens is that "readFile" returns immediately, continuing at line 16. Which is actually where "parseProject" also ends.
Nothing happens until the operating system has fetched the file from disk (or over the network). Could be milliseconds or minutes later. During that time your JS code can continue doing something else, like responding to user input. Then that strange function parameter actually gets called, it's like an event handler. When it is called the system gives it the "err" and "data" parameters. "data" is the content of the fetched file as a string.
08) Let's ignore errors, now we have the file content as a string in "data" and we call parser.parse with the file name and file content string. The parser does whatever it does and returns a list of sub-object file names in "fileStack". It also builds up an internal data structure containing all PUBs, PRIs etc.
09) Now we want to process any sub-objects so we pop a "fileName" off of the "fileStack".
11) If there was a "fileName" on the stack what do we do? It's a sub-project so obviously we call the "parseProject" function again to deal with it! Magic hey?
13) If there were no sub-objects returned on the "fileStack" we call a parser method that prints out it internal data structure of PUBs, PRIs etc. And we are done.
16) Now here is a mind bender. Up to this point in the description no code has actually been executed. It's only a definition of the "parseProject" function. To get things started we could call it in a normal fashion "parseProject (topFileName)".
But notice we do not do that. Instead we just add "()" after the function definition. Now, the function definition is an expression that happens to return a function so adding "()" calls that function. This thing calls itself. Both internally (line 08) and externally (line 16) !!
Phew, that's a lot for a little harmless looking function.
Where does that come from? The OBEX? Who is going to make all those icon? Who is going to ensure all those objects are easy to use in such a system? How do I create new ones of my own?
Hmm - good points.
Well, first thing I guess is what you are doing is open source. So if there is a framework for a compiler, it should not be too hard to add another tab/menu item. That can add the drag and drop part of the program.
Icons. Well, they are pretty easy to design. Make them 32x32 pixels and bitmaps. Paintshop pro or any graphics program can do this. I have a really old version of paint shop pro, I set it for screen capture, go searching in Images on google, then capture what I want, resize it, save it as a bitmap.
Now the next bit might need to be a bit cunning. Let's define the icon in text. So, rather than having associated icons, then having to keep them in the same folder etc, how about we take that 32x32 pixel icon, and we convert it to base64. Now the icon is text, and you can include that in a Spin (or C) program as a comment block. And any program parsing this can look for such a block and extract it.
Next little thing, say we want to make it a bit easier to work with the .start method. Usually that is a line of numbers, or sometimes a line of constants that link to numbers in the CON section. So maybe that needs an explanatory series of comment lines that explains what each parameter is, so it can appear in a format the IDE can display and the user can edit. So instead of
keyboard.start(28,29)
You might have
keyboard.start(28,29)
' begin method variables
' Data pin = 28
' Clock pin = 29
' end method variables
So it is a bit more verbose, but those comments would be useful anyway. Just need to use the same format.
Maybe this is a bit pointless for Serial and Keyboard objects, but they would be the easy ones to code, and then you can use the same framework for real GUI objects, like a button. There are VGA and TV objects that can display buttons and text boxes, even if they are a bit chunky on only in monochrome. And touchscreens can look just as good as a PC screen.
Where this would really start to get useful is where you drop a textbox and a button onto a screen, and you click the button and the textbox says 'Hello World'. In pseudo code that many languages use
if button1.click
textbox1.text := "Hello World"
and then to be able to run this on a PC, debug it on a PC, and then download and run the same program to a Propeller.
This doesn't have to need a display on the propeller. Say I want to turn on a pump when my solar panels are producing more than 4kw. Ok, I can write that in Spin. But I can also write it in a graphical sort of language, and instead of worrying about the complexities of talking to an ADC device via the I2C bus, I can abstract that to an icon that reads an analog voltage. Then I drag a relay icon onto the screen as well. I can then write one line of code to link them together. The propeller need not have a display at all, maybe in this case it just has a LED that lights up.
Anyway, getting something working in text is the first priority. But it is nice to know that javascript can be used to build graphical things too.
Addit - cross posts. Reading your recursive post now. Feeling a bit weird...
Opensource for sure. Nothing would make me happier than to see the results of this effort get taken up and extended. Perhaps not even the code itself. We have only demonstrated what can be done. Ken at parallax already has his eye on this..
Icons are easy. What I was hinting at is the huge job it might be to maintain the whole infrastructure. In the extreme I want to be able to drag and drop any object from OBEX into program and have it do all that start up code editing and stuff. No messing with searching obex, downloading unpacking etc etc,
Now the icon is text, and you can include that in a Spin (or C) program as a comment
No. Ideally such icons and "meta-data" would just be files included in the object package. Ideally of course all the objects would be kept in a source code repository where such things can be easily managed.
maybe that needs an explanatory series of comment lines that explains what each parameter is, so it can appear in a format the IDE can display.
Nice. But to do that the descriptions of the parameters have to be included in the objects Spin source as comments in a format that an IDE or other tool can use. There are such systems in use for JavaScript itself for example:
/**
* Queries a Baz for items.
* @param {number} groupNum Subgroup id to query.
* @param {string|number|null} term An itemName,
* or itemId, or null to search everything.
*/
goog.Baz.prototype.query = function(groupNum, term) {
...
};
The formalized comment there is designed to be readable by a tool to generate documentation or otherwise process the source code. It tells the parameter types and describes them. It can also specify the return types.
Who is going to document every method in OBEX like that so that the GUI builder can use it?
JavaScript does that to people. It looks like a simple harmless stupid scripting language. And has been used as such in web pages for years. But when you start to look at it hard it starts to look very weird. That's because it is not the simple language it appears. It has features that other high level languages are only now starting to adopt. "First class functions", "nested functions", "anonymous functions (lambdas)", "proto-typical object inheritance", "closures", "self invocation", and so on.
That is an expression which returns a function, which has no name, which is immediately called.
That nameless function sets up a timer that calls the inner, also nameless, function every 1000ms.
The inner function just prints "Hello".
So this program sits there printing "Hello" every second.
But here is the thing. This code has not defined any function we can call or any variables we can use. Nothing. But still that unnamed, unaccessible thing continues to print "Hello" in the background!
Lets say you want to compile Spin file A. Here is what happens:
1) Download A
2) Send A to the web worker for compilation.
3) Worker says A needs B,C, D...
4) Down load B, C, D...
5) Send them to the worker for compilation.
6) Worker says B, C, C need X, Y, Z...
All of which is slow and tedious.
Enter the Javascript pre-parser that can scan and fetch all the required files. Whist waiting on downloads the compiler could be busy compiling in the web worker process in the back ground.
This was important when the JavaScript version of openspin was very slow. Now that we have optimized it and it is ten times faster perhaps it's not such a big deal.
Hmmm...thinks, looks like the preparser is still a win when editing code and you want the list of available methods updated in real time.
You're right of course. Sorry for the distraction. I didn't read the thread carefully enough I guess. Also, I thought you were running OpenSpin in the web browser. Did you abandon that idea?
No. Ideally such icons and "meta-data" would just be files included in the object package. Ideally of course all the objects would be kept in a source code repository where such things can be easily managed.
...
Nice. But to do that the descriptions of the parameters have to be included in the objects Spin source as comments in a format that an IDE or other tool can use. There are such systems in use for JavaScript itself for example:
Ok, well there is always the 'standard' way. For icons, include a file with the same filename as the spin code, and the extension .ico
For autocompletion, the parser simply collects up all the PUBs in the object. Let's grab a random object, say the Keyboard, and look at its Start PUB. It is
PUB start(dpin, cpin) : okay
As soon as you type "keyboard." it can give the suggestion of this line. Maybe leave out the : okay bit. So you get dpin and cpin as suggestions. It should be possible to take a reasonable guess what those two variables mean. But if not, maybe you have to take a look at the code. Or make a slight change to the object to make the variables a bit more descriptive.
What I can anticipate with autocompletion suggestions is a new way of coding - instead of having multiple objects open in tabs, most of the time the only object open would be the main object. Using objects would become easier.
For autocompletion, the parser simply collects up all the PUBs in the object. Let's grab a random object, say the Keyboard, and look at its Start PUB. It is
PUB start(dpin, cpin) : okay
As soon as you type "keyboard." it can give the suggestion of this line. Maybe leave out the : okay bit. So you get dpin and cpin as suggestions. It should be possible to take a reasonable guess what those two variables mean. But if not, maybe you have to take a look at the code. Or make a slight change to the object to make the variables a bit more descriptive.
What I can anticipated with autocompletion suggestions is a new way of coding - instead of having multiple objects open in tabs, most of the time the only object open would be the main object.
But it should be possible to not have to change anything in many spin objects. And with
Thanks for the link jazzed. Great ideas there! Looks like this has all been considered before, but maybe the IDEs didn't exist back then?
I'm trying to hold several concepts in my head at the same time.
1) Reading about html 5 http://www.w3schools.com/html/default.asp - things like web workers, local storage (5Mb) and drag and drop.
2) Learning javascript http://www.w3schools.com/js/default.asp and integrating this with html.
3) Testing and trying to understand the code on this thread
4) Thinking about whether it is possible to write spin code to understand html (or at least a much simpler subset of html)
5) Thinking about whether it is possible to interpret javascript with spin
6) Thinking about more graphical ways of programming using icons and links
7) Wondering whether Spin is too limited to do any of this in terms of code size, and whether overlays are the answer or whether running C with external memory is the best solution
8) Thinking if C is a better solution, whether simple html and javascript interpreters have been written in C already and whether these could be ported to the propeller
9) Wondering if the web based framework could then run on the propeller too.
I'm still trying to climb out of heater's recursive code from yesterday - it might need more caffeine!
In the webserver program there are 13 spin files. Containing 329 PUB's and 186 PRI's. A total of 515 methods. Including commented out ones because I was counting with a simple grep.
If you look in the output of the parser test it includes 13 spin files. Containing 515 PUB and PRI (If I don't use the comment removal).
How is it that it works without that loop you added?
fileStack is what it says, a stack. The parser pushes an entry on the stack every time it finds a sub-object.
Now, every time we come out of parser.parse we pop one more file from the stack to process. We process it by calling "parseProject". Which calls parser.parse and pops another file from the stack. And around we go again.
The "loop" is already there in that recursion. Cunning hey?
I put that stack there exactly so that it would process all files even if it is doing async file fetching.
Now. If there is one thing missing from the parser it is that it does not build a "tree" structured output refelcting the nesting of sub-objects within the program. I was about to tackle that next.
I firmly believe that async file fetching should be used. Otherwise the browser is going to freeze while loading all the files. Else you put the parsing/fetching into the loader which I think is messy and backwards.
Attached is a new parser. I added detection of curly comments "{..}". There is also a new bit of test code (Commented out) that simply lists the sub-objects required by a single given spin file.
I thought you were running OpenSpin in the web browser. Did you abandon that idea?
Absolutely not.
When I said "web worker process in the background" I meant a "Web Worker" process. This is actually another totally separate JavaScript program running as a sub process spawned by the browser. It's a real OS process with it's own memory space and all. It listens for messages from the web page containing Spin source and sends messages back containing the compiled binary.
Web workers are a recent HTML5 standard thing. Ask Google about them.
They are intended for doing that long winded compute intensive work without hanging up the main user interface.
I'm trying to hold several concepts in my head at the same time. 1)....9)
That's about 8 more than I can manage now a days I guess you don't sleep much.
Items 1) and 2) (and 3)) are of course huge by themselves. My plan in life, for now at least, is to avoid 1) as much as possible. All that HTML/CSS page layout stuff is a mess.
I believe it's a good idea to tackle JS as a language by itself initially. Like you might learn the working of VB or C++ syntax and semantics before getting bogged down in the GUI building business.
The best way to do that is get node.js installed on your machine and experiment with JS with nothing but an editor and the command line. Then you can write programs, run them, save them, experiment with them, all without all that browser hassle. JS is a sophisticated language it deserves study by itself like any other.
Item 4) is no doubt possible. But a lot more work than I would dream of taking on. Spin really is not suited to all that string handling.
Item 5) is clearly impossible. It would be terrible slow anyway. JS works with strings a lot and all numbers are 64 bit floating point!
Item 8) amazingly has already been done. Well at least there is TinyJS that is a very small JS engine that has been compiled with propgcc and run on a Propeller with external memory. It's very crude and limited and has been superseded by Espruino. A very complete JS engine by the same author used on micro-controllers. Hopefully we will get that running on the P2 one day!
I'm still trying to climb out of heater's recursive code from yesterday
Here is a new twist for you. That code is not actually recursive. Not in the traditional sense of a function calling itself and running up a huge stack of return addresses.
Notice how the outer function "parseProject" returns immediately after kicking off "readFile". The inner function, passed to "readFile" is called some time later by the JS event loop system when data from the file has arrived from disk.
It's a bit of a brain twister. It depends on JavaScripts "event loop". That's rather like an "event loop" in Visual Basic or whatever. That inner function passed to readFile is rather like a mouse click or keyboard press handler you might set up on the code for a GUI widget. In this case the function is called when data arrives from disk or from the network. It's kind of subtle because that inner event handler is nameless, unlike having something like "blabla.onClick" defined in "normal languages".
It's time you installed node.js, Instead of spending hours building that test harness you could have just typed "node spin-parser.js" and seen the result immediately
spin_parser still has issues. For example I have totally ignored strings. If somebody puts a "{" or "'" in a string then a lot of following code will be ignored when it should not.
I'm starting to wonder if it's safer not to use the curly comment remover and accept some commented out PUBs and PRIs some times. It's only one line of code to comment out.
Looking forward to Editor15. I might not have much time to play the weekend.
About climbing out of that non-recursive recursion:
Consider the following program:
var greet = function () {
console.log("Hello");
}
setInterval (greet, 1000);
console.log("Started...");
That creates a function which prints "hello", and assigns that function to the variable "greet".
Then it sets up a timer that will call "greet" after one second.
The program does not exit immediately at the bottom, after printing "Started...". Rather it waits for the pending timer event. When that fires, one second later, the "greet" function is called "Hello" is printed and the program terminates.
We could wrap all that up in a function, in case we want to reuse it, like so:
function makeGreeting () {
var greet = function () {
console.log("Hello");
}
setInterval (greet, 1000);
}
makeGreeping();
console.log("Started...");
Now we don't have a named variable or named function "greet", we just define it on the fly as a parameter to "setInterval". It works the same way. "Started" is printed first. After one second "Hello" is printed and the program terminates.
Now, that timer event fires after one second, "Hello" is printed, and then we call makeGreeting again which sets up a new timer and creates a new function to call on the timer event. So "Hello" gets printed every second. Forever.
There is no recursion here in the traditional sense. It's just a timer event firing off and creating a new timer event for the next second ahead.
Now, instead of using a timer to generate events we can use something like "fs.readFile".
"readFile" returns immediately, just like the "setInterval". But it sets up an event that will fire off sometime later, when the data arrives from that slow old disk. When the event fires the data is used and we go round and call "readFile" again which sets up a whole new event for more data in the future. And so on until all the data is read.
There we have it, non-recursive recursion
This is some of the magic of JS. Your program could have many things going on at the same time. Something is waiting for data from disk, something is waiting for data from the network. In the browser you are waiting on the users mouse and keyboard actions.
It's like having a lot of threads but without the complexity of using threads.
Thing is "==" is comparing values. Well, JS being a dynamically typed language has a habit of changing the types of things. For example:
console.log("Hello prisoner number " + 6);
Does not explode complaining that it cannot add a number to a string. No, it does what you want, convert the number to a string "6" and then concatenate the strings and print it.
So what about:
if ("6" == 6)...
Well that number 6 is going to be converted to a string and then the two string compared and the result will be true.
Fine, but that is often not what you want. It's also a common source of programmer error. A number is not a string. Enter "===".
if ("6" === 6)...
That will check that the types of the things being compared are the same. A string is not a number so this returns false.
Bottom line: Always, Always use ===. You will save yourself a lot of head scratching some days.
Whilst we are at it. Always put "use strict"; at the top of your JS files. And always check your code with jslint or jshint. Those tools will point out all kind of gotchas like using "==". They will drive you mad because they are very fussy about formatting. Take their advice and format your code as they say. There are gotchas in the way you format JS. See this video about that: http://www.youtube.com/watch?v=_EANG8ZZbRs
functions defined inside of functions can access all variables of parent? or copy of?
Now we are talking about "closures". Consider:
function makeGreeter (message) {
var date = new Date();
return function greet () {
console.log(message + " from " + date);
}
}
var g1 = makeGreeter ("Hello");
var g2 = makeGreeter ("Good evening");
g1();
g2();
That inner function can see "message" and "data" in the surrounding scope. When the function is returned by makeGreeter it is actually returning a new function object that captures the values of "message" and "date" at the time.
So g1 and g2 are different function objects with different values of "message" and "date", they just happen to share the same code.
This is a very powerful feature and one that is only recently coming to C++ as Java (in a very clumsy and crude way).
Be warned. This works for numbers and strings and such. What if "message" were a complex object? Then it would actually be a reference to the same object rather than a new copy. In that case you need "new" in there somewhere to create a new message instance for each new greeter to grab.
Cool. Shamefully I did not get a chance to make any progress here.
A little issue. When I load WebServer_W5100_RTC.spin from my local machine, and all the files in depends on, and try to compile it it fails complaining that fullDuplexSerial4port.spin is missing. Even though it has it and has sent it to the web worker.
2014-03-17 08:36:28 . execute Openspin -b WebServer_W5100_RTC.spin...
2014-03-17 08:36:28 w Worker started - consoleID: 17
2014-03-17 08:36:29 w OpenSpin run time env. created.
2014-03-17 08:36:29 . init_openspin_done - now adding files...
2014-03-17 08:36:29 . init_openspin_done - parsing WebServer_W5100_RTC.spin ...
2014-03-17 08:36:29 w Writing WebServer_W5100_RTC.spin to MemFS.
2014-03-17 08:36:29 . init_openspin_done - parsing HttpHeader.spin ...
2014-03-17 08:36:29 w Writing HttpHeader.spin to MemFS.
2014-03-17 08:36:29 . init_openspin_done - parsing Sntp.spin ...
2014-03-17 08:36:29 w Writing Sntp.spin to MemFS.
2014-03-17 08:36:29 . init_openspin_done - parsing Dns.spin ...
2014-03-17 08:36:29 w Writing Dns.spin to MemFS.
2014-03-17 08:36:29 . init_openspin_done - parsing NetBios.spin ...
2014-03-17 08:36:29 w Writing NetBios.spin to MemFS.
2014-03-17 08:36:29 . init_openspin_done - parsing Dhcp.spin ...
2014-03-17 08:36:30 w Writing Dhcp.spin to MemFS.
2014-03-17 08:36:30 . init_openspin_done - parsing Socket.spin ...
2014-03-17 08:36:30 w Writing Socket.spin to MemFS.
2014-03-17 08:36:30 . init_openspin_done - parsing W5100.spin ...
2014-03-17 08:36:30 w Writing W5100.spin to MemFS.
2014-03-17 08:36:30 . init_openspin_done - parsing S35390A_SD-MMC_FATEngineWrapper.spin ...
2014-03-17 08:36:30 w Writing S35390A_SD-MMC_FATEngineWrapper.spin to MemFS.
2014-03-17 08:36:30 . init_openspin_done - parsing S35390A_RTCEngine.spin ...
2014-03-17 08:36:30 w Writing S35390A_RTCEngine.spin to MemFS.
2014-03-17 08:36:30 . init_openspin_done - parsing fullDuplexSerial4port.spin ...
2014-03-17 08:36:30 . call_openspin_done - ... finished all parsing. calling compiler.
2014-03-17 08:36:30 w Writing fullDuplexSerial4port.spin to MemFS.
2014-03-17 08:36:30 w Running openspin main
2014-03-17 08:36:30 + Propeller Spin/PASM Compiler 'OpenSpin' (c)2012-2013 Parallax Inc. DBA Parallax Semiconductor.
2014-03-17 08:36:30 + Compiled on Mar 6 2014 19:58:09
2014-03-17 08:36:30 + Compiling...
2014-03-17 08:36:31 + WebServer_W5100_RTC.spin
2014-03-17 08:36:31 + |-fullDuplexSerial4port.spin
2014-03-17 08:36:31 + fullDuplexSerial4port.spin : error : Can not find/open file.
2014-03-17 08:36:31 w Finished openspin main.
2014-03-17 08:36:31 . call_openspin_done - now retrieving result...
2014-03-17 08:36:31 w Reading WebServer_W5100_RTC.binary from MemFS.
2014-03-17 08:36:32 w No such file.
2014-03-17 08:36:32 . Result file size 0 bytes.
2014-03-17 08:36:32 . Worker terminated itself correctly.
Then your 'recursive' parser stops parsing the files and returns, starting the compiler before all files are there.
2014-03-17 12:43:12 . execute Openspin -b WebServer_W5200_RTC.spin...
2014-03-17 12:43:13 w Worker started - consoleID: 1
2014-03-17 12:43:13 w OpenSpin run time env. created.
2014-03-17 12:43:13 . init_openspin_done - now adding files...
2014-03-17 12:43:14 . init_openspin_done - parsing WebServer_W5200_RTC.spin ...
2014-03-17 12:43:14 . Try to GET [URL]http://parallax.msrobots.net/propeller-w5200-driver/S35390A_RTCEngine.spin[/URL] - Result: OK
2014-03-17 12:43:14 . init_openspin_done - parsing S35390A_RTCEngine.spin ...
2014-03-17 12:43:15 . Try to GET [URL]http://parallax.msrobots.net/propeller-w5200-driver/SNTPSimpleNetworkTimeProtocolv2.01.spin[/URL] - Result: Not Found
2014-03-17 12:43:15 . Try to GET [URL]http://parallax.msrobots.net/Library/SNTPSimpleNetworkTimeProtocolv2.01.spin[/URL] - Result: Not Found
2014-03-17 12:43:16 . Try to GET [URL]http://www.parallax.msrobots.net/Library/SNTPSimpleNetworkTimeProtocolv2.01.spin[/URL] - Result: Not Found
2014-03-17 12:43:16 . call_openspin_done - ... finished all parsing. calling compiler.
2014-03-17 12:43:16 w Writing WebServer_W5200_RTC.spin to MemFS.
2014-03-17 12:43:17 w Writing S35390A_RTCEngine.spin to MemFS.
2014-03-17 12:43:17 w Running openspin main
2014-03-17 12:43:18 + Propeller Spin/PASM Compiler 'OpenSpin' (c)2012-2013 Parallax Inc. DBA Parallax Semiconductor.
2014-03-17 12:43:18 + Compiled on Mar 6 2014 19:58:09
2014-03-17 12:43:18 + Compiling...
2014-03-17 12:43:18 + WebServer_W5200_RTC.spin
2014-03-17 12:43:19 + |-Parallax Serial Terminal.spin
2014-03-17 12:43:19 + Parallax Serial Terminal.spin : error : Can not find/open file.
2014-03-17 12:43:19 w Finished openspin main.
2014-03-17 12:43:19 . call_openspin_done - now retrieving result...
2014-03-17 12:43:20 w Reading WebServer_W5200_RTC.binary from MemFS.
2014-03-17 12:43:20 w No such file.
2014-03-17 12:43:20 . Result file size 0 bytes.
2014-03-17 12:43:21 . Worker terminated itself correctly.
2014-03-17 12:45:17 . execute Openspin -b WebServer_W5100_RTC.spin...
2014-03-17 12:45:18 w Worker started - consoleID: 1
2014-03-17 12:45:18 w OpenSpin run time env. created.
2014-03-17 12:45:18 . init_openspin_done - now adding files...
2014-03-17 12:45:18 . init_openspin_done - parsing WebServer_W5100_RTC.spin ...
2014-03-17 12:45:19 . Try to GET [URL]http://parallax.msrobots.net/propeller-w5200-driver/Spinneret-msrobots/HttpHeader.spin[/URL] - Result: OK
2014-03-17 12:45:19 . init_openspin_done - parsing HttpHeader.spin ...
2014-03-17 12:45:20 . Try to GET [URL]http://parallax.msrobots.net/propeller-w5200-driver/Spinneret-msrobots/Sntp.spin[/URL] - Result: OK
2014-03-17 12:45:20 . init_openspin_done - parsing Sntp.spin ...
2014-03-17 12:45:20 . Try to GET [URL]http://parallax.msrobots.net/propeller-w5200-driver/Spinneret-msrobots/Dns.spin[/URL] - Result: OK
2014-03-17 12:45:21 . init_openspin_done - parsing Dns.spin ...
2014-03-17 12:45:21 w Writing WebServer_W5100_RTC.spin to MemFS.
2014-03-17 12:45:21 . Try to GET [URL]http://parallax.msrobots.net/propeller-w5200-driver/Spinneret-msrobots/W5100.spin[/URL] - Result: OK
2014-03-17 12:45:22 . init_openspin_done - parsing W5100.spin ...
2014-03-17 12:45:22 . Try to GET [URL]http://parallax.msrobots.net/propeller-w5200-driver/Spinneret-msrobots/Spi5100CounterPasm.spin[/URL] - Result: OK
2014-03-17 12:45:23 . init_openspin_done - parsing Spi5100CounterPasm.spin ...
2014-03-17 12:45:23 w Writing HttpHeader.spin to MemFS.
2014-03-17 12:45:23 . Try to GET [URL]http://parallax.msrobots.net/propeller-w5200-driver/Spinneret-msrobots/Socket.spin[/URL] - Result: OK
2014-03-17 12:45:24 . init_openspin_done - parsing Socket.spin ...
2014-03-17 12:45:24 . Try to GET [URL]http://parallax.msrobots.net/propeller-w5200-driver/Spinneret-msrobots/W5100.spin[/URL] - Result: OK
2014-03-17 12:45:25 . init_openspin_done - parsing W5100.spin ...
2014-03-17 12:45:25 w Writing Sntp.spin to MemFS.
2014-03-17 12:45:25 . Try to GET [URL]http://parallax.msrobots.net/propeller-w5200-driver/Spinneret-msrobots/NetBios.spin[/URL] - Result: OK
2014-03-17 12:45:26 . init_openspin_done - parsing NetBios.spin ...
2014-03-17 12:45:26 . Try to GET [URL]http://parallax.msrobots.net/propeller-w5200-driver/Spinneret-msrobots/W5100.spin[/URL] - Result: OK
2014-03-17 12:45:27 . init_openspin_done - parsing W5100.spin ...
2014-03-17 12:45:27 w Writing Dns.spin to MemFS.
2014-03-17 12:45:27 . Try to GET [URL]http://parallax.msrobots.net/propeller-w5200-driver/Spinneret-msrobots/Socket.spin[/URL] - Result: OK
2014-03-17 12:45:28 . init_openspin_done - parsing Socket.spin ...
2014-03-17 12:45:28 . Try to GET [URL]http://parallax.msrobots.net/propeller-w5200-driver/Spinneret-msrobots/Dhcp.spin[/URL] - Result: OK
2014-03-17 12:45:29 . init_openspin_done - parsing Dhcp.spin ...
2014-03-17 12:45:29 w Writing W5100.spin to MemFS.
2014-03-17 12:45:30 . Try to GET [URL]http://parallax.msrobots.net/propeller-w5200-driver/Spinneret-msrobots/W5100.spin[/URL] - Result: OK
2014-03-17 12:45:30 . init_openspin_done - parsing W5100.spin ...
2014-03-17 12:45:30 . Try to GET [URL]http://parallax.msrobots.net/propeller-w5200-driver/Spinneret-msrobots/Socket.spin[/URL] - Result: OK
2014-03-17 12:45:31 . init_openspin_done - parsing Socket.spin ...
2014-03-17 12:45:31 w Writing Spi5100CounterPasm.spin to MemFS.
2014-03-17 12:45:32 . Try to GET [URL]http://parallax.msrobots.net/propeller-w5200-driver/Spinneret-msrobots/Socket.spin[/URL] - Result: OK
2014-03-17 12:45:32 . init_openspin_done - parsing Socket.spin ...
2014-03-17 12:45:33 . Try to GET [URL]http://parallax.msrobots.net/propeller-w5200-driver/Spinneret-msrobots/W5100.spin[/URL] - Result: OK
2014-03-17 12:45:33 . init_openspin_done - parsing W5100.spin ...
2014-03-17 12:45:34 w Writing Socket.spin to MemFS.
2014-03-17 12:45:34 . Try to GET [URL]http://parallax.msrobots.net/propeller-w5200-driver/Spinneret-msrobots/S35390A_SD-MMC_FATEngineWrapper.spin[/URL] - Result: OK
2014-03-17 12:45:35 . init_openspin_done - parsing S35390A_SD-MMC_FATEngineWrapper.spin ...
2014-03-17 12:45:35 . Try to GET [URL]http://parallax.msrobots.net/propeller-w5200-driver/Spinneret-msrobots/S35390A_SD-MMC_FATEngine.spin[/URL] - Result: OK
2014-03-17 12:45:36 . init_openspin_done - parsing S35390A_SD-MMC_FATEngine.spin ...
2014-03-17 12:45:36 w Writing W5100.spin to MemFS.
2014-03-17 12:45:37 . Try to GET [URL]http://parallax.msrobots.net/propeller-w5200-driver/Spinneret-msrobots/S35390A_RTCEngine.spin[/URL] - Result: OK
2014-03-17 12:45:37 . init_openspin_done - parsing S35390A_RTCEngine.spin ...
2014-03-17 12:45:38 w put_file: already there : W5100.spin
2014-03-17 12:45:38 . Try to GET [URL]http://parallax.msrobots.net/propeller-w5200-driver/Spinneret-msrobots/S35390A_RTCEngine.spin[/URL] - Result: OK
2014-03-17 12:45:39 . init_openspin_done - parsing S35390A_RTCEngine.spin ...
2014-03-17 12:45:40 . Try to GET [URL]http://parallax.msrobots.net/propeller-w5200-driver/Spinneret-msrobots/fullDuplexSerial4port.spin[/URL] - Result: OK
2014-03-17 12:45:40 . init_openspin_done - parsing fullDuplexSerial4port.spin ...
2014-03-17 12:45:40 . call_openspin_done - ... finished all parsing. calling compiler.
2014-03-17 12:45:41 w Writing NetBios.spin to MemFS.
2014-03-17 12:45:42 w Writing W5100.spin to MemFS.
2014-03-17 12:45:42 w put_file: already there : W5100.spin
2014-03-17 12:45:44 w Writing Socket.spin to MemFS.
2014-03-17 12:45:44 w put_file: already there : Socket.spin
2014-03-17 12:45:45 w Writing Dhcp.spin to MemFS.
2014-03-17 12:45:45 w Writing W5100.spin to MemFS.
2014-03-17 12:45:46 w put_file: already there : W5100.spin
2014-03-17 12:45:47 w Writing Socket.spin to MemFS.
2014-03-17 12:45:47 w put_file: already there : Socket.spin
2014-03-17 12:45:48 w Writing Socket.spin to MemFS.
2014-03-17 12:45:49 w put_file: already there : Socket.spin
2014-03-17 12:45:49 w Writing W5100.spin to MemFS.
2014-03-17 12:45:50 w put_file: already there : W5100.spin
2014-03-17 12:45:50 w Writing S35390A_SD-MMC_FATEngineWrapper.spin to MemFS.
2014-03-17 12:45:51 w Writing S35390A_SD-MMC_FATEngine.spin to MemFS.
2014-03-17 12:45:51 w Writing S35390A_RTCEngine.spin to MemFS.
2014-03-17 12:45:52 w Writing S35390A_RTCEngine.spin to MemFS.
2014-03-17 12:45:52 w put_file: already there : S35390A_RTCEngine.spin
2014-03-17 12:45:53 w Writing fullDuplexSerial4port.spin to MemFS.
2014-03-17 12:45:54 w Running openspin main
2014-03-17 12:45:54 + Propeller Spin/PASM Compiler 'OpenSpin' (c)2012-2013 Parallax Inc. DBA Parallax Semiconductor.
2014-03-17 12:45:55 + Compiled on Mar 6 2014 19:58:09
2014-03-17 12:45:56 + Compiling...
2014-03-17 12:45:56 + WebServer_W5100_RTC.spin
2014-03-17 12:45:57 + |-fullDuplexSerial4port.spin
2014-03-17 12:45:57 + |-S35390A_RTCEngine.spin
2014-03-17 12:45:58 + |-S35390A_SD-MMC_FATEngineWrapper.spin
2014-03-17 12:45:58 + |-S35390A_SD-MMC_FATEngine.spin
2014-03-17 12:45:59 + |-S35390A_RTCEngine.spin
2014-03-17 12:45:59 + |-W5100.spin
2014-03-17 12:46:00 + |-Spi5100CounterPasm.spin
2014-03-17 12:46:00 + |-Socket.spin
2014-03-17 12:46:01 + |-W5100.spin
2014-03-17 12:46:01 + |-Spi5100CounterPasm.spin
2014-03-17 12:46:02 + |-Dhcp.spin
2014-03-17 12:46:02 + |-Socket.spin
2014-03-17 12:46:03 + |-W5100.spin
2014-03-17 12:46:03 + |-Spi5100CounterPasm.spin
2014-03-17 12:46:04 + |-W5100.spin
2014-03-17 12:46:04 + |-Spi5100CounterPasm.spin
2014-03-17 12:46:05 + |-NetBios.spin
2014-03-17 12:46:05 + |-Socket.spin
2014-03-17 12:46:05 + |-W5100.spin
2014-03-17 12:46:06 + |-Spi5100CounterPasm.spin
2014-03-17 12:46:06 + |-W5100.spin
2014-03-17 12:46:07 + |-Spi5100CounterPasm.spin
2014-03-17 12:46:07 + |-Dns.spin
2014-03-17 12:46:08 + |-Socket.spin
2014-03-17 12:46:08 + |-W5100.spin
2014-03-17 12:46:09 + |-Spi5100CounterPasm.spin
2014-03-17 12:46:09 + |-W5100.spin
2014-03-17 12:46:09 + |-Spi5100CounterPasm.spin
2014-03-17 12:46:10 + |-Sntp.spin
2014-03-17 12:46:10 + |-HttpHeader.spin
2014-03-17 12:46:11 + Done.
2014-03-17 12:46:11 + Program size is 31308 bytes
2014-03-17 12:46:11 w Finished openspin main.
2014-03-17 12:46:12 . call_openspin_done - now retrieving result...
2014-03-17 12:46:12 w Reading WebServer_W5100_RTC.binary from MemFS.
2014-03-17 12:46:13 . Result file size 31308 bytes.
2014-03-17 12:46:13 . Worker terminated itself correctly.
In both cases you can see that the parser just send a message to the webworker to upload the file.
after call_openspin_done - ... finished all parsing. calling compiler
the webworker still uploads files.
The upload local file thing is probably a result of me trying to upload multiple files. You may need to look at the upload stuff. there may be that NEW missing you where talking about in post #438.
Thinking a bit more about this -
after displaying call_openspin_done - ... finished all parsing. calling compiler.
the main thread should display all webworker results ( them with w in front) in the same order as issued by the parser.
I do not see that in your failing example.
it starts the LAST issued webworker message - and runs the compiler.
I found the issue with upload file ... I guess it is what you call closure. so I put () around the definition of the inline function and now local file upload works with multiple files.
So now you can simply upload all files (or some) of a directory at once to the editor and open them.
I am still thinking that we reinvent the wheel here, because CodeMirror has already a complete SpinParser written by @PhiPI. It also has a couple of other Parser for C, C++, C# JavaScript, Html, CSS and well even COBOL.
It needs and uses those Parser for syntax and background highlighting. Down to keywords, variables, comments, whatever. It would be nice to tap into that instead of using another parser.
There has been an interesting little "debate" on loader stuff. Thought I would ask about it here.
Two loader scenarios are possible, but I'm not convinced that both will work in a real network.
1. For one loader, the goal is to be able to program Propeller directly from a computer with Wifi modules in transparent serial mode.
2. For another loader, the goal is to be able to load a program to Propeller over a typical network using Javascript to send packets to a Propeller loader server.
Item 2 is desirable for the WebTool. Item 1 can be useful too. What do you think? Insight? Opinions?
1) Editing and compiling of Spin programs entirely within the browser.
2) Upload and download of Spin source and binaries from a webserver.
3) Reading and writing of source and binaries to the local file system.
What we cannot do, as we have no access to serial ports from the browser, is:
1) Program a Propeller
2) Talk to a running Propeller over a serial link into a terminal Window.
Enter the "Propeller Server" the Propeller server:
1) Has serial connection to one or more Propellers.
2) Accepts Propeller binaries from browsers.
3) Programs those binaries into Propellers, using whatever exiting loader programs we have.
4) Relays serial communication from Propeller to in browser terminal window,
Never mind sending packets from JavaScript to a loader server. I would never trust the Prop loader protocol to work reliably that way. We just upload the entire binary in a PUT/POST request and let the server do the programming.
The relaying of the serial terminal stream is going to be done with websockets. That's the only way to get a nice real-time communication stream.
Potential issue here is that I'm not sure we can upload binaries to a different server than the one the web page and Source files were served from. Perhaps they need to be the same server.
Having said that, I pretty sure a websocket connection can be made to a different server than the page source. So binaries could be uploaded for programming using the websocket connection.
Where so these servers run? Any where. On my local machine with a locally attached Propeller. On a class room machine with many attached Propellers.
I didn't get your option one. Is that in the context of the web IDE? Makes no odds, I have no idea how well serial loading might work of a "transparent serial mode" WIFI connection.
My WIFI module would be a Raspberry Pi with a USB WIFI dongle. Then it could run the Propeller Server as described above.
Don't forget option 3). The Chrome browser can access the serial ports of your machine if your web page is wrapped up as a Chrome App. I was about to start experimenting with that. I wonder how well we can get a loader to work after it has been translated to JS?
A Propeller load server would run on the Propeller to be loaded. At reset time the load server starts up and listens to the network device (XBee S6B or equivalent) for programming packets. On timeout or after programming, the load server would start a program if any from the 32KB eeprom. This could be used on the PropellerBOE or ActivityBoard.
How to cause a reset from the S6B device is not entirely worked out at the moment. It has GPIO bits but Parallax didn't bring them out to a header. RTS is available, but it's not clear if that can be used for a reset signal at the moment.
Are you saying the combination of Propeller + S6B is performing as a web server accessible over WIFI and that we can upload files to it (HTTP GET/PUT). Are those "files" the packets of which you speak that end up in EEPROM ?
I'm not familiar with these devices.
How are going to combat "bricking" the Propeller when the EEPROM is corrupted due to a disconnection during reprogramming?
Is it really so that a Propeller cannot reset or at least reload itself under program control?. Can't we just start COG 0 with the loader code in the ROM?
Are you saying the combination of Propeller + S6B is performing as a web server accessible over WIFI and that we can upload files to it (HTTP GET/PUT). Are those "files" the packets of which you speak that end up in EEPROM ?
How are going to combat "bricking" the Propeller when the EEPROM is corrupted due to a disconnection during reprogramming?
The boot loader in lower 32KB never gets written. Only the upper 32KB gets written. The boot loader reads upper 32KB into HUB RAM and starts the SPIN interpreter.
Is it really so that a Propeller cannot reset or at least reload itself under program control?. Can't we just start COG 0 with the loader code in the ROM?
It would be nice to have a Monitor COG that would act as a serial port and a reset command monitor. That would take a COG away from the user though.
This is GREAT. Please give me some link to your (techbasic?) sources showing how to talk to the S6B. I am really interested to assimilate that.
By now I have a compiled binary in the browser. If I can send it somewhere over http or whatever we are fine.
I may be able soon to use a spinneret as PropPlug and Program any board connected to the spinneret. (serial / no usb). work in progress....
@Heater claims local access to local serial/usb might be possible with chrome. Very interesting.
Anyways. I have Editor15 ready.
Slowly I am getting where I want to go. I finally figured out what was nagging me with @Heater.'s 'recursion'. I guess I fixed it. Now I walk the whole object tree.
Comments
s = "....a string which is an entire Spin source file..." editor.doc.setValue(s);All files in use are in memory. In fact they are in memory at least twice because the actual compiler is a "web worker", a totally separate process that needs to be sent it's own copies of the files. How much memory does your machine have. 4GB? 16GB, 32GB...Take it all if you like ! Yep. OK let me hit you hard with the awesome power and unholy majesty of JavaScript. Hold tight because there are things going on here that you may never have seen in any other low level programming language like VB, Java, C++...01 parseProject = function parse(fileName) { 02 var fileStack; 03 04 fs.readFile(fileName, encoding, function (err, data) { 05 if (err) { 06 throw err; 07 } 08 fileStack = parser.parse(fileName, data); 09 fileName = fileStack.pop(); 10 if (fileName) { 11 parseProject(fileName); 12 } else { 13 console.log(parser.prettyPrint()); 14 } 15 }); 16 }(topFileName);Lines numbered for explanation:01) Here we have a function "parseProject" that is going to fetch the file "fileName" read it's contents and produce a data structure containing all OBJs PUBs and PRIs within. It's also going to fetch all those sub-objects and do the same to them. And the sub-sub-sub-objects and so on.
02) "fileStack" keeps a stack of files to be processed initially undefined.
04) Call "readFile" to get the file "fileName" with some "encoding" (ASCII, UTF-8 etc). Notice the third parameter here is a function definition which in turn has parameters "err" and "data". What happens is that "readFile" returns immediately, continuing at line 16. Which is actually where "parseProject" also ends.
Nothing happens until the operating system has fetched the file from disk (or over the network). Could be milliseconds or minutes later. During that time your JS code can continue doing something else, like responding to user input. Then that strange function parameter actually gets called, it's like an event handler. When it is called the system gives it the "err" and "data" parameters. "data" is the content of the fetched file as a string.
08) Let's ignore errors, now we have the file content as a string in "data" and we call parser.parse with the file name and file content string. The parser does whatever it does and returns a list of sub-object file names in "fileStack". It also builds up an internal data structure containing all PUBs, PRIs etc.
09) Now we want to process any sub-objects so we pop a "fileName" off of the "fileStack".
11) If there was a "fileName" on the stack what do we do? It's a sub-project so obviously we call the "parseProject" function again to deal with it! Magic hey?
13) If there were no sub-objects returned on the "fileStack" we call a parser method that prints out it internal data structure of PUBs, PRIs etc. And we are done.
16) Now here is a mind bender. Up to this point in the description no code has actually been executed. It's only a definition of the "parseProject" function. To get things started we could call it in a normal fashion "parseProject (topFileName)".
But notice we do not do that. Instead we just add "()" after the function definition. Now, the function definition is an expression that happens to return a function so adding "()" calls that function. This thing calls itself. Both internally (line 08) and externally (line 16) !!
Phew, that's a lot for a little harmless looking function.
Hmm - good points.
Well, first thing I guess is what you are doing is open source. So if there is a framework for a compiler, it should not be too hard to add another tab/menu item. That can add the drag and drop part of the program.
Icons. Well, they are pretty easy to design. Make them 32x32 pixels and bitmaps. Paintshop pro or any graphics program can do this. I have a really old version of paint shop pro, I set it for screen capture, go searching in Images on google, then capture what I want, resize it, save it as a bitmap.
Now the next bit might need to be a bit cunning. Let's define the icon in text. So, rather than having associated icons, then having to keep them in the same folder etc, how about we take that 32x32 pixel icon, and we convert it to base64. Now the icon is text, and you can include that in a Spin (or C) program as a comment block. And any program parsing this can look for such a block and extract it.
Next little thing, say we want to make it a bit easier to work with the .start method. Usually that is a line of numbers, or sometimes a line of constants that link to numbers in the CON section. So maybe that needs an explanatory series of comment lines that explains what each parameter is, so it can appear in a format the IDE can display and the user can edit. So instead of
You might have
So it is a bit more verbose, but those comments would be useful anyway. Just need to use the same format.
Maybe this is a bit pointless for Serial and Keyboard objects, but they would be the easy ones to code, and then you can use the same framework for real GUI objects, like a button. There are VGA and TV objects that can display buttons and text boxes, even if they are a bit chunky on only in monochrome. And touchscreens can look just as good as a PC screen.
Where this would really start to get useful is where you drop a textbox and a button onto a screen, and you click the button and the textbox says 'Hello World'. In pseudo code that many languages use
and then to be able to run this on a PC, debug it on a PC, and then download and run the same program to a Propeller.
This doesn't have to need a display on the propeller. Say I want to turn on a pump when my solar panels are producing more than 4kw. Ok, I can write that in Spin. But I can also write it in a graphical sort of language, and instead of worrying about the complexities of talking to an ADC device via the I2C bus, I can abstract that to an icon that reads an analog voltage. Then I drag a relay icon onto the screen as well. I can then write one line of code to link them together. The propeller need not have a display at all, maybe in this case it just has a LED that lights up.
Anyway, getting something working in text is the first priority. But it is nice to know that javascript can be used to build graphical things too.
Addit - cross posts. Reading your recursive post now. Feeling a bit weird...
Opensource for sure. Nothing would make me happier than to see the results of this effort get taken up and extended. Perhaps not even the code itself. We have only demonstrated what can be done. Ken at parallax already has his eye on this..
Icons are easy. What I was hinting at is the huge job it might be to maintain the whole infrastructure. In the extreme I want to be able to drag and drop any object from OBEX into program and have it do all that start up code editing and stuff. No messing with searching obex, downloading unpacking etc etc, No. Ideally such icons and "meta-data" would just be files included in the object package. Ideally of course all the objects would be kept in a source code repository where such things can be easily managed. Nice. But to do that the descriptions of the parameters have to be included in the objects Spin source as comments in a format that an IDE or other tool can use. There are such systems in use for JavaScript itself for example:
/** * Queries a Baz for items. * @param {number} groupNum Subgroup id to query. * @param {string|number|null} term An itemName, * or itemId, or null to search everything. */ goog.Baz.prototype.query = function(groupNum, term) { ... };The formalized comment there is designed to be readable by a tool to generate documentation or otherwise process the source code. It tells the parameter types and describes them. It can also specify the return types.Who is going to document every method in OBEX like that so that the GUI builder can use it?
JavaScript does that to people. It looks like a simple harmless stupid scripting language. And has been used as such in web pages for years. But when you start to look at it hard it starts to look very weird. That's because it is not the simple language it appears. It has features that other high level languages are only now starting to adopt. "First class functions", "nested functions", "anonymous functions (lambdas)", "proto-typical object inheritance", "closures", "self invocation", and so on.
Here is a fun one:
(function () { setInterval (function() { console.log ("Hello!"); }, 1000); })();That is an expression which returns a function, which has no name, which is immediately called.That nameless function sets up a timer that calls the inner, also nameless, function every 1000ms.
The inner function just prints "Hello".
So this program sits there printing "Hello" every second.
But here is the thing. This code has not defined any function we can call or any variables we can use. Nothing. But still that unnamed, unaccessible thing continues to print "Hello" in the background!
Weird.
I did study your test project for the parser, but you are missing some stuff while doing recursion.
consider this:
... [SIZE=2] parseProject = [/SIZE][SIZE=2][COLOR=#020fc0][SIZE=2][COLOR=#020fc0]function[/COLOR][/SIZE][/COLOR][/SIZE][SIZE=2] parse(fileName) {[/SIZE] [SIZE=2][COLOR=#020fc0][SIZE=2][COLOR=#020fc0] var[/COLOR][/SIZE][/COLOR][/SIZE][SIZE=2] fileStack;[/SIZE] [SIZE=2] fs.readFile(path + fileName, [/SIZE][SIZE=2][COLOR=#a31515][SIZE=2][COLOR=#a31515]'utf8'[/COLOR][/SIZE][/COLOR][/SIZE][SIZE=2], [/SIZE][SIZE=2][COLOR=#020fc0][SIZE=2][COLOR=#020fc0]function[/COLOR][/SIZE][/COLOR][/SIZE][SIZE=2] (err, data) {[/SIZE] [SIZE=2][COLOR=#020fc0][SIZE=2][COLOR=#020fc0] if[/COLOR][/SIZE][/COLOR][/SIZE][SIZE=2] (err) {[/SIZE] [SIZE=2][COLOR=#020fc0][SIZE=2][COLOR=#020fc0] throw[/COLOR][/SIZE][/COLOR][/SIZE][SIZE=2] err;[/SIZE] [SIZE=2] } fileStack = parser.parse(fileName, data); fileName = fileStack.pop(); [/SIZE][SIZE=2][COLOR=#020fc0][SIZE=2][COLOR=#020fc0]if[/COLOR][/SIZE][/COLOR][/SIZE][SIZE=2] (fileName) {[/SIZE] [SIZE=2] parseProject(fileName); [/SIZE][SIZE=2] } [/SIZE][SIZE=2][COLOR=#020fc0][SIZE=2][COLOR=#020fc0]else[/COLOR][/SIZE][/COLOR][/SIZE][SIZE=2] {[/SIZE] [SIZE=2] console.log(parser.prettyPrint()); } }); }; parseProject(topFile); [/SIZE]compared to this
... [SIZE=2] parseProject = [/SIZE][SIZE=2][COLOR=#020fc0][SIZE=2][COLOR=#020fc0]function[/COLOR][/SIZE][/COLOR][/SIZE][SIZE=2] parse(fileName) {[/SIZE] [SIZE=2][COLOR=#020fc0][SIZE=2][COLOR=#020fc0] var[/COLOR][/SIZE][/COLOR][/SIZE][SIZE=2] fileStack;[/SIZE] [SIZE=2] fs.readFile(path + fileName, [/SIZE][SIZE=2][COLOR=#a31515][SIZE=2][COLOR=#a31515]'utf8'[/COLOR][/SIZE][/COLOR][/SIZE][SIZE=2], [/SIZE][SIZE=2][COLOR=#020fc0][SIZE=2][COLOR=#020fc0]function[/COLOR][/SIZE][/COLOR][/SIZE][SIZE=2] (err, data) {[/SIZE] [SIZE=2][COLOR=#020fc0][SIZE=2][COLOR=#020fc0] if[/COLOR][/SIZE][/COLOR][/SIZE][SIZE=2] (err) {[/SIZE] [SIZE=2][COLOR=#020fc0][SIZE=2][COLOR=#020fc0] throw[/COLOR][/SIZE][/COLOR][/SIZE][SIZE=2] err;[/SIZE] [SIZE=2] } fileStack = parser.parse(fileName, data); [SIZE=2][COLOR=#020fc0][SIZE=2][COLOR=#020fc0] while[/COLOR][/SIZE][/COLOR][/SIZE][SIZE=2] (fileStack.length>0) {[/SIZE] fileName = fileStack.pop();[/SIZE] [SIZE=2] parseProject(fileName); [/SIZE][SIZE=2] }[/SIZE][SIZE=2] }); }; parseProject(topFile); [/SIZE]see what I stumbled upon? you are missing all sub-sub objects exept the first. Where now to put prettyPrint?
without async calls I can put it behind parseProject(topFile)
Enjoy!
Mike
Ok, well there is always the 'standard' way. For icons, include a file with the same filename as the spin code, and the extension .ico
For autocompletion, the parser simply collects up all the PUBs in the object. Let's grab a random object, say the Keyboard, and look at its Start PUB. It is
As soon as you type "keyboard." it can give the suggestion of this line. Maybe leave out the : okay bit. So you get dpin and cpin as suggestions. It should be possible to take a reasonable guess what those two variables mean. But if not, maybe you have to take a look at the code. Or make a slight change to the object to make the variables a bit more descriptive.
What I can anticipate with autocompletion suggestions is a new way of coding - instead of having multiple objects open in tabs, most of the time the only object open would be the main object. Using objects would become easier.
The following is the way auto-complete (spinnysense?) works in the PropellerIDE project.
I'm trying to hold several concepts in my head at the same time.
1) Reading about html 5 http://www.w3schools.com/html/default.asp - things like web workers, local storage (5Mb) and drag and drop.
2) Learning javascript http://www.w3schools.com/js/default.asp and integrating this with html.
3) Testing and trying to understand the code on this thread
4) Thinking about whether it is possible to write spin code to understand html (or at least a much simpler subset of html)
5) Thinking about whether it is possible to interpret javascript with spin
6) Thinking about more graphical ways of programming using icons and links
7) Wondering whether Spin is too limited to do any of this in terms of code size, and whether overlays are the answer or whether running C with external memory is the best solution
8) Thinking if C is a better solution, whether simple html and javascript interpreters have been written in C already and whether these could be ported to the propeller
9) Wondering if the web based framework could then run on the propeller too.
I'm still trying to climb out of heater's recursive code from yesterday - it might need more caffeine!
My test example works just fine.
In the webserver program there are 13 spin files. Containing 329 PUB's and 186 PRI's. A total of 515 methods. Including commented out ones because I was counting with a simple grep.
If you look in the output of the parser test it includes 13 spin files. Containing 515 PUB and PRI (If I don't use the comment removal).
How is it that it works without that loop you added?
fileStack is what it says, a stack. The parser pushes an entry on the stack every time it finds a sub-object.
Now, every time we come out of parser.parse we pop one more file from the stack to process. We process it by calling "parseProject". Which calls parser.parse and pops another file from the stack. And around we go again.
The "loop" is already there in that recursion. Cunning hey?
I put that stack there exactly so that it would process all files even if it is doing async file fetching.
Now. If there is one thing missing from the parser it is that it does not build a "tree" structured output refelcting the nesting of sub-objects within the program. I was about to tackle that next.
I firmly believe that async file fetching should be used. Otherwise the browser is going to freeze while loading all the files. Else you put the parsing/fetching into the loader which I think is messy and backwards.
Attached is a new parser. I added detection of curly comments "{..}". There is also a new bit of test code (Commented out) that simply lists the sub-objects required by a single given spin file.
When I said "web worker process in the background" I meant a "Web Worker" process. This is actually another totally separate JavaScript program running as a sub process spawned by the browser. It's a real OS process with it's own memory space and all. It listens for messages from the web page containing Spin source and sends messages back containing the compiled binary.
Web workers are a recent HTML5 standard thing. Ask Google about them.
They are intended for doing that long winded compute intensive work without hanging up the main user interface.
Items 1) and 2) (and 3)) are of course huge by themselves. My plan in life, for now at least, is to avoid 1) as much as possible. All that HTML/CSS page layout stuff is a mess.
I believe it's a good idea to tackle JS as a language by itself initially. Like you might learn the working of VB or C++ syntax and semantics before getting bogged down in the GUI building business.
The best way to do that is get node.js installed on your machine and experiment with JS with nothing but an editor and the command line. Then you can write programs, run them, save them, experiment with them, all without all that browser hassle. JS is a sophisticated language it deserves study by itself like any other.
Item 4) is no doubt possible. But a lot more work than I would dream of taking on. Spin really is not suited to all that string handling.
Item 5) is clearly impossible. It would be terrible slow anyway. JS works with strings a lot and all numbers are 64 bit floating point!
Item 8) amazingly has already been done. Well at least there is TinyJS that is a very small JS engine that has been compiled with propgcc and run on a Propeller with external memory. It's very crude and limited and has been superseded by Espruino. A very complete JS engine by the same author used on micro-controllers. Hopefully we will get that running on the P2 one day! Here is a new twist for you. That code is not actually recursive. Not in the traditional sense of a function calling itself and running up a huge stack of return addresses.
Notice how the outer function "parseProject" returns immediately after kicking off "readFile". The inner function, passed to "readFile" is called some time later by the JS event loop system when data from the file has arrived from disk.
It's a bit of a brain twister. It depends on JavaScripts "event loop". That's rather like an "event loop" in Visual Basic or whatever. That inner function passed to readFile is rather like a mouse click or keyboard press handler you might set up on the code for a GUI widget. In this case the function is called when data arrives from disk or from the network. It's kind of subtle because that inner event handler is nameless, unlike having something like "blabla.onClick" defined in "normal languages".
I even spend some hours to build a test harness to prove that I am wrong and @Heater. is right.
So I included the spinparser into the editor and am able now to load all needed files before calling the compiler. Sweet.
Next on the list is the jumplist for PRI and PUB.
Its weekend! Propellertime! Update to Editor14 soon.
Enjoy!
Mike
It's time you installed node.js, Instead of spending hours building that test harness you could have just typed "node spin-parser.js" and seen the result immediately
spin_parser still has issues. For example I have totally ignored strings. If somebody puts a "{" or "'" in a string then a lot of following code will be ignored when it should not.
I'm starting to wonder if it's safer not to use the curly comment remover and accept some commented out PUBs and PRIs some times. It's only one line of code to comment out.
Looking forward to Editor15. I might not have much time to play the weekend.
Consider the following program:
var greet = function () { console.log("Hello"); } setInterval (greet, 1000); console.log("Started...");That creates a function which prints "hello", and assigns that function to the variable "greet".Then it sets up a timer that will call "greet" after one second.
The program does not exit immediately at the bottom, after printing "Started...". Rather it waits for the pending timer event. When that fires, one second later, the "greet" function is called "Hello" is printed and the program terminates.
We could wrap all that up in a function, in case we want to reuse it, like so:
function makeGreeting () { var greet = function () { console.log("Hello"); } setInterval (greet, 1000); } makeGreeping(); console.log("Started...");That program can be rewritten like so:function makeGreeting () { setInterval (function () { console.log("Hello"); }, 1000); } makeGreeping(); console.log("Started...");Now we don't have a named variable or named function "greet", we just define it on the fly as a parameter to "setInterval". It works the same way. "Started" is printed first. After one second "Hello" is printed and the program terminates.But what if we do this:
function makeGreeting () { setInterval (function () { console.log("Hello"); makeGreeping(); }, 1000); } makeGreeping(); console.log("Started...");Now, that timer event fires after one second, "Hello" is printed, and then we call makeGreeting again which sets up a new timer and creates a new function to call on the timer event. So "Hello" gets printed every second. Forever.There is no recursion here in the traditional sense. It's just a timer event firing off and creating a new timer event for the next second ahead.
Now, instead of using a timer to generate events we can use something like "fs.readFile".
"readFile" returns immediately, just like the "setInterval". But it sets up an event that will fire off sometime later, when the data arrives from that slow old disk. When the event fires the data is used and we go round and call "readFile" again which sets up a whole new event for more data in the future. And so on until all the data is read.
There we have it, non-recursive recursion
This is some of the magic of JS. Your program could have many things going on at the same time. Something is waiting for data from disk, something is waiting for data from the network. In the browser you are waiting on the users mouse and keyboard actions.
It's like having a lot of threads but without the complexity of using threads.
Slowly I get a grip on this event loop thing. JavaScript is a really fun language.
Some things still confuse me.
when to use === instead of == and why?
functions defined inside of functions can access all variables of parent? or copy of?
I do have changed your parser to fit my needs. I had to make remove Brackets optional.
It messes up all line numbers. And I need the line number to jump to.
will update tomorrow. tired.
Enjoy!
Mike
Thing is "==" is comparing values. Well, JS being a dynamically typed language has a habit of changing the types of things. For example:
console.log("Hello prisoner number " + 6);Does not explode complaining that it cannot add a number to a string. No, it does what you want, convert the number to a string "6" and then concatenate the strings and print it.So what about:
if ("6" == 6)...Well that number 6 is going to be converted to a string and then the two string compared and the result will be true.Fine, but that is often not what you want. It's also a common source of programmer error. A number is not a string. Enter "===".
if ("6" === 6)...That will check that the types of the things being compared are the same. A string is not a number so this returns false.Bottom line: Always, Always use ===. You will save yourself a lot of head scratching some days.
Whilst we are at it. Always put "use strict"; at the top of your JS files. And always check your code with jslint or jshint. Those tools will point out all kind of gotchas like using "==". They will drive you mad because they are very fussy about formatting. Take their advice and format your code as they say. There are gotchas in the way you format JS. See this video about that: http://www.youtube.com/watch?v=_EANG8ZZbRs Now we are talking about "closures". Consider:
function makeGreeter (message) { var date = new Date(); return function greet () { console.log(message + " from " + date); } } var g1 = makeGreeter ("Hello"); var g2 = makeGreeter ("Good evening"); g1(); g2();That inner function can see "message" and "data" in the surrounding scope. When the function is returned by makeGreeter it is actually returning a new function object that captures the values of "message" and "date" at the time.So g1 and g2 are different function objects with different values of "message" and "date", they just happen to share the same code.
This is a very powerful feature and one that is only recently coming to C++ as Java (in a very clumsy and crude way).
Be warned. This works for numbers and strings and such. What if "message" were a complex object? Then it would actually be a reference to the same object rather than a new copy. In that case you need "new" in there somewhere to create a new message instance for each new greeter to grab.
Ah, poop, I forgot about the line numbers.
So I have a new Version. Editor14. attached as zip.
This is now using the JavaScript parser to find embedded objects and provide a jumplist to PRI and PUB methods.
It also provides a jumplist to functions. Handy if you edit js or htm files...
updated link for testing live http://parallax.msrobots.net/Editor14.htm
Enjoy!
Mike
A little issue. When I load WebServer_W5100_RTC.spin from my local machine, and all the files in depends on, and try to compile it it fails complaining that fullDuplexSerial4port.spin is missing. Even though it has it and has sent it to the web worker.
I can reproduce it when I have a missing file. Example: http://parallax.msrobots.net/Editor14.htm?a=http://parallax.msrobots.net/propeller-w5200-driver/WebServer_W5200_RTC.spin
Then your 'recursive' parser stops parsing the files and returns, starting the compiler before all files are there.
I can not reproduce it when all files are there. Example: http://parallax.msrobots.net/Editor14.htm?a=http://parallax.msrobots.net/propeller-w5200-driver/Spinneret-msrobots/WebServer_W5100_RTC.spin
In both cases you can see that the parser just send a message to the webworker to upload the file.
after call_openspin_done - ... finished all parsing. calling compiler
the webworker still uploads files.
The upload local file thing is probably a result of me trying to upload multiple files. You may need to look at the upload stuff. there may be that NEW missing you where talking about in post #438.
occupied by paid work...
Enjoy!
Mike
after displaying call_openspin_done - ... finished all parsing. calling compiler.
the main thread should display all webworker results ( them with w in front) in the same order as issued by the parser.
I do not see that in your failing example.
it starts the LAST issued webworker message - and runs the compiler.
I found the issue with upload file ... I guess it is what you call closure. so I put () around the definition of the inline function and now local file upload works with multiple files.
So now you can simply upload all files (or some) of a directory at once to the editor and open them.
I am still thinking that we reinvent the wheel here, because CodeMirror has already a complete SpinParser written by @PhiPI. It also has a couple of other Parser for C, C++, C# JavaScript, Html, CSS and well even COBOL.
It needs and uses those Parser for syntax and background highlighting. Down to keywords, variables, comments, whatever. It would be nice to tap into that instead of using another parser.
I just have not figured out how.
will update soon.
Enjoy!
Mike
Two loader scenarios are possible, but I'm not convinced that both will work in a real network.
1. For one loader, the goal is to be able to program Propeller directly from a computer with Wifi modules in transparent serial mode.
2. For another loader, the goal is to be able to load a program to Propeller over a typical network using Javascript to send packets to a Propeller loader server.
Item 2 is desirable for the WebTool. Item 1 can be useful too. What do you think? Insight? Opinions?
I'm not sure I understand the options.
What we have achieved here is:
1) Editing and compiling of Spin programs entirely within the browser.
2) Upload and download of Spin source and binaries from a webserver.
3) Reading and writing of source and binaries to the local file system.
What we cannot do, as we have no access to serial ports from the browser, is:
1) Program a Propeller
2) Talk to a running Propeller over a serial link into a terminal Window.
Enter the "Propeller Server" the Propeller server:
1) Has serial connection to one or more Propellers.
2) Accepts Propeller binaries from browsers.
3) Programs those binaries into Propellers, using whatever exiting loader programs we have.
4) Relays serial communication from Propeller to in browser terminal window,
Never mind sending packets from JavaScript to a loader server. I would never trust the Prop loader protocol to work reliably that way. We just upload the entire binary in a PUT/POST request and let the server do the programming.
The relaying of the serial terminal stream is going to be done with websockets. That's the only way to get a nice real-time communication stream.
Potential issue here is that I'm not sure we can upload binaries to a different server than the one the web page and Source files were served from. Perhaps they need to be the same server.
Having said that, I pretty sure a websocket connection can be made to a different server than the page source. So binaries could be uploaded for programming using the websocket connection.
Where so these servers run? Any where. On my local machine with a locally attached Propeller. On a class room machine with many attached Propellers.
I didn't get your option one. Is that in the context of the web IDE? Makes no odds, I have no idea how well serial loading might work of a "transparent serial mode" WIFI connection.
My WIFI module would be a Raspberry Pi with a USB WIFI dongle. Then it could run the Propeller Server as described above.
How to cause a reset from the S6B device is not entirely worked out at the moment. It has GPIO bits but Parallax didn't bring them out to a header. RTS is available, but it's not clear if that can be used for a reset signal at the moment.
I'm not familiar with these devices.
How are going to combat "bricking" the Propeller when the EEPROM is corrupted due to a disconnection during reprogramming?
Is it really so that a Propeller cannot reset or at least reload itself under program control?. Can't we just start COG 0 with the loader code in the ROM?
Yes.
The boot loader in lower 32KB never gets written. Only the upper 32KB gets written. The boot loader reads upper 32KB into HUB RAM and starts the SPIN interpreter.
It would be nice to have a Monitor COG that would act as a serial port and a reset command monitor. That would take a COG away from the user though.
This is GREAT. Please give me some link to your (techbasic?) sources showing how to talk to the S6B. I am really interested to assimilate that.
By now I have a compiled binary in the browser. If I can send it somewhere over http or whatever we are fine.
I may be able soon to use a spinneret as PropPlug and Program any board connected to the spinneret. (serial / no usb). work in progress....
@Heater claims local access to local serial/usb might be possible with chrome. Very interesting.
Anyways. I have Editor15 ready.
Slowly I am getting where I want to go. I finally figured out what was nagging me with @Heater.'s 'recursion'. I guess I fixed it. Now I walk the whole object tree.
attached Editor15. Live Link here http://parallax.msrobots.net/Editor15.htm
Enjoy!
Mike