Seairth,
The compiler Heater is talking about is an emscripten conversion of OpenSpin. It was done a while ago, and I don't know if he/they have kept it updated with changes/fixes to OpenSpin.
Can the VS Code extensions call external executable files? In any case, I am pretty sure VS Code can be configured to run different external compilers for given source extensions.
Seairth,
The compiler Heater is talking about is an emscripten conversion of OpenSpin. It was done a while ago, and I don't know if he/they have kept it updated with changes/fixes to OpenSpin.
Can the VS Code extensions call external executable files? In any case, I am pretty sure VS Code can be configured to run different external compilers for given source extensions.
Yes, VSCode can easily run external code.
The reason I want to embed it (or a variation of it) is to get an AST to use for things like intellisense, hover, etc. I may have to roll my own, though, since it's unlikely that the emscriptened version will be very hackable.
...what is this compiler's name and is it open source?
As Roy says it is actually OpenSpin compiled to Javascript using Emscripten.
I can dig out the build script that does that if you like, but that will have to wait till I get back to my PC at home. Might be a week. You'll have to remind me.
Question is do you want to write a new Spin compiler from the ground up in Javascript, perhaps using pegjs?
If you do you'll have to be prepared to track changes in OpenSpin as it evolves in the future.
I'm not looking to develop a compiler. There are (or will be, for P2) plenty of those. I just want to parse the file enough to extract some of the information: constant values, label references, etc.
When I played with pegjs I found parsing Spin and PASM a bit of a challenge.
I forget the correct terminology for grammars and parsers but Spin/PASM is not a nice regular syntax like those evolved from ALGOL. It has that white space delimiting and everything is line based.
Perhaps I just did not think about it hard enough.
When I played with pegjs I found parsing Spin and PASM a bit of a challenge.
I forget the correct terminology for grammars and parsers but Spin/PASM is not a nice regular syntax like those evolved from ALGOL. It has that white space delimiting and everything is line based.
Perhaps I just did not think about it hard enough.
No, I think you are right about the parsing complexity of the language. From what I've seen of your PEG files, it looks like you've put quite a bit of thought into it!
I spent a long while getting to grips with peg and then the Spin/PASM syntax. And then that project timed out, half baked.
That is my second attempt at parsing Spin/PASM. Years back I wrote a mostly complete parser in Ada. Would you believe. Then I decided Ada really is a dead end, no matter that I am attracted to a lot of it's features.
As for syntax, this kind of thing in a DAT section is a doozy:
I don't know if anybody here uses Arduino, but this guy makes an extension for VS Community. Hopefully Parallax can support something similar with Prop chips.
I would rather they did not. That is for Microsoft Visual Studio which only runs on Windows. It's better that Parallax use their resources in creating cross platform tools that any one can use anywhere.
What we have been discussing here is Visual Studio Code. This is an entirely different product. It written in Javascript (Well, Typescript). It runs on the Google V8 Javascript engine and uses Electron for the GUI. It is open source and cross platform.
VS Code is something worth investing time in.
I don't say this lightly, as a life long critic of all things Microsoft.
. . .[snip]. .
What we have been discussing here is Visual Studio Code.
. . .[snip]. .
I don't say this lightly, as a life long critic of all things Microsoft.
.
I guess I didn't explain that I knew the difference. I feel VS Community is too heavy to program a microchip on a daily basis. My mistake for not making it clearer. I was voicing support that I think Parallax should transition to supporting a Prop extension FOR VS CODE and move away from SimpleIDE. Not because SimpleIDE is a bad product, but because they can focus on the Prop details and leave the editor portion to the community. I was only pointing out the VisualMicro extension for VS Community as an example of what has been done before.
Perhaps everyone should list their parochialisms in their signature.
As far as parsers go, spin2cpp has a Yacc parser. I know Yacc isn't everyone's cup of tea, but it is pretty widely supported. The tricky bits of Spin are mostly in the lexer, which is hand coded to handle the indentation (when indent level increases it emits an INDENT token, when it decreases it emits UNDENT) and state dependent stuff (when we're in DAT sections some differently rules are used than in PUB/PRI).
. . .[snip]. .
What we have been discussing here is Visual Studio Code.
. . .[snip]. .
I don't say this lightly, as a life long critic of all things Microsoft.
.
I guess I didn't explain that I knew the difference. I feel VS Community is too heavy to program a microchip on a daily basis. My mistake for not making it clearer. I was voicing support that I think Parallax should transition to supporting a Prop extension FOR VS CODE and move away from SimpleIDE. Not because SimpleIDE is a bad product, but because they can focus on the Prop details and leave the editor portion to the community. I was only pointing out the VisualMicro extension for VS Community as an example of what has been done before.
Perhaps everyone should list their parochialisms in their signature.
You and I are on the same page, a few of us have brought this up in the past here. VS Code is not the first program to come out that allows one to build an IDE on top of an editor. Parallax could have done that from the start with SimpleIDE, and probably as far back as the Prop Tool. From what I've been able to glean from the forums, it was a deliberate choice to start from scratch. Parallax want something tailored for their exact requirements which they believe will serve the education community best. And SimpleIDE is certainly far more simple than VS Code would be - they hit that nail on the head!
I think there are different types of users with different desires for IDE.
P1 has a big hobbyist/education lobby, so I think something simple, like Prop Tool is fine.
It also has tiny memory, so any program is going to be relatively simple for it.
But, P2 may attract more needy users who want to write big C/C++ programs.
For them, I think the full blown Visual Studio is hard to beat.
I would love to see a Prop2 extension for it.
For them, I think the full blown Visual Studio is hard to beat.
I would love to see a Prop2 extension for it.
.
I've been programming my Arduinos using the VisualMicro extension for the "full-blown" VS Community edition. While this works well, VS Community is a Kitchen Sink tool and I always feel I'm sifting thru all the phone and cloud programming options to blink an LED.
.
So I guess it's up to us . .
.
So... I've been slowly(!) working on the P2 extension. I hope to have something useful for people to play with within the next week. A significant portion of my time has been taken in learning modern Javascript (and Typescript), node.js, PEG (Parsing Expression Grammars), and writing VSCode Language Servers. Learning PEG (and, more specifically, PEGJS) has been very time-consuming. And the grammar I have so far (~460 lines) is still missing a few language elements. But, if I can keep going with it, I think it's got some great potential! Frankly, if I (or someone else) were enterprising enough, I could probably write an entire assembler with PEGJS. For now, though, I am simply creating an AST (and a fairly flat one, at that)...
Anyhow, right now, all I've got working is:
* Syntax coloring (not much different than what I provided before)
* Parsing and displaying the first encountered error
* Completion suggestions for constants and labels
* Hover information for constants (the expression) and labels (the line)
Obviously, there is a lot more that could be added (e.g. intellisense for the instructions, constant expression evaluation, address calculations, snippets, etc). The question is: what would be most useful (and hopefully easy to implement)?
Aside: I think some of this would have gone a bit more smoothly if we had access to the pnut source, which I think would be about as close as we're going to get to a formal grammar. Has Chip talked about opening that up at all? Also, did he ever get a CLI implemented for pnut? It would be nice to call it directly from VSCode...
Seairth, the PASM code is written in '386 and it compiles into a Delphi app. Roy Eltham has gotten into it and rewritten it in C. His code would be what someone would want, I'd think. Mine is cryptic and I'm surprised Roy figured it out.
I haven't worked on the command-line interface yet. Just been getting back to documenting the architecture.
Seairth, the PASM code is written in '386 and it compiles into a Delphi app. Roy Eltham has gotten into it and rewritten it in C. His code would be what someone would want, I'd think. Mine is cryptic and I'm surprised Roy figured it out.
I haven't worked on the command-line interface yet. Just been getting back to documenting the architecture.
That's fine. I might look at Roy's work then. Actually, this exercise in writing a PEG for the P2 assembly language has been an interesting exercise. It's the first time I've written a grammar as complex as this one. And I'm still learning to do some parts of it properly. I'll be sure to publish it when I release the extension...
As for the CLI, I'll take it when I can get it. No hurry.
If you want a formal grammar for Spin, there's a YACC description in spin2cpp. It seems to be pretty accurate (it compiles everything I've thrown at it so far).
If you want a formal grammar for Spin, there's a YACC description in spin2cpp. It seems to be pretty accurate (it compiles everything I've thrown at it so far).
This is hurting my brain. So, one of the down sides to using PEG is that it's an all-or-nothing approach. If I encounter an error, I get nothing back, except the error. But, I think I've got a way around it... instead of parsing the entire file at once, I'll parse it one line at a time. That way, even if one line can't be parsed (e.g. you are in the middle of editing it), I should still be able to parse all valid lines above and below it.
Before I go down this path, is anyone aware of any multi-line expressions that I might encounter? The only thing that comes to mind is comment blocks. That complicates things slightly, but if it's the only one I encounter, I can deal with that easily enough. Can anyone think of any others?
It's a while since I played with peg and parsing Spin. I do recall that it gave me headache.
My impression is that PEG likes a nice clean syntax that can be parsed in a recursive decent fashion, which Spin is certainly not. Spin is kind of line based like BASIC, but then it is block structured, with the cursed white space deliminting of Python. Then you basically have to change syntax from block to block, DAT, PUB, PRI, CON... It's a mess.
A few thoughts:
Perhaps it's better to pre-process the source and deal with the comments first. The output of that process would have no comments,
Rather than reduce the parser to parsing one line at a time do it a block at a time. If I understand correctly PUB, PRI, DAT etc have to start at the beginning of a new line. So if anything before that is in error you can always move forward, line by line, and find a new PUB, PRI or whatever to start parsing from again.
On the other hand why are you fighting this problem? If PEG detects a syntax error and bails then that is it, the user has to fix that error to move on. For example when I write Javascript in MS Visual Studio Code or Atom or Sublime editors I have JS lint running that will highlight all suspect code with red squiggles or whatever. But it will bail out if I I have a real syntax error. The rest of the file does not get checked.
It's a while since I played with peg and parsing Spin. I do recall that it gave me headache.
My impression is that PEG likes a nice clean syntax that can be parsed in a recursive decent fashion, which Spin is certainly not. Spin is kind of line based like BASIC, but then it is block structured, with the cursed white space deliminting of Python. Then you basically have to change syntax from block to block, DAT, PUB, PRI, CON... It's a mess.
A few thoughts:
Perhaps it's better to pre-process the source and deal with the comments first. The output of that process would have no comments,
Rather than reduce the parser to parsing one line at a time do it a block at a time. If I understand correctly PUB, PRI, DAT etc have to start at the beginning of a new line. So if anything before that is in error you can always move forward, line by line, and find a new PUB, PRI or whatever to start parsing from again.
On the other hand why are you fighting this problem? If PEG detects a syntax error and bails then that is it, the user has to fix that error to move on. For example when I write Javascript in MS Visual Studio Code or Atom or Sublime editors I have JS lint running that will highlight all suspect code with red squiggles or whatever. But it will bail out if I I have a real syntax error. The rest of the file does not get checked.
Yeah, I was thinking along the same lines, somewhat. Start by detecting the block type (con, dat, etc.), then go from there. The issue with a failed parse is that I don't have access to the output from the part that did parse correctly. For instance, even if the entire CON section parsed, I wouldn't get back the AST for that section if I hit an error in a later section. But, as you are typing in new instructions, you will constantly be switching between a valid and invalid document (e.g. as you enter a constant expression, it's likely to be invalid for some number of keystrokes). So... how do I provide auto-complete of constants or labels, if the line your editing just happens to be invalid?
By switching to per-line parsing, I think I can get around a lot of these issues. Even when I eventually add support for Propeller 1 Spin (e.g. PUB and PRI sections), I believe that the block structures won't be too much of an issue. But, right now, I am only concerned about P2 assembly, which is somewhat less complicated.
Pre-processing the comments sounds like a good idea too.
So is there any work done to integrate P1/P2 Spin and PASM into Visual Studio Code?
Looking at the VS code site there is a generator able to import TextMate, Sublime uses TextMate like files and someone on this forum posted about Sublime-Support for Spin/Pasm.
I personally have zero experience with Yacc/Bision/Pegis/TextMate nor VS code. I just spend some time with @Heater to build some JavaScript based IDE based on some HtmlEditor. But there I just had to add new rules and keywords, basically the rest was there, But that Editor18 we worked on, was really just experimental and for fun.
Editor18.htm was a single 1.5xx MB file mainly JavaScript, but able to edit itself!
But integration in VS code would be really nice. So is there some way to use the Yacc stuff out of Spin2cpp and create a basic TextMate Syntax to import into VS code language generator?
Comments
The compiler Heater is talking about is an emscripten conversion of OpenSpin. It was done a while ago, and I don't know if he/they have kept it updated with changes/fixes to OpenSpin.
Can the VS Code extensions call external executable files? In any case, I am pretty sure VS Code can be configured to run different external compilers for given source extensions.
Yes, VSCode can easily run external code.
The reason I want to embed it (or a variation of it) is to get an AST to use for things like intellisense, hover, etc. I may have to roll my own, though, since it's unlikely that the emscriptened version will be very hackable.
I can dig out the build script that does that if you like, but that will have to wait till I get back to my PC at home. Might be a week. You'll have to remind me.
I hope Mrs. Heater. doesn't clean up your (her new) PC like she has your projects in the past!!
Oh, shoot, now you have put the fear of God into me.
That JS version of OpenSpin is one of the few things that never made it into github.
What do you know! I just came across http://pegjs.org/, which brought me back to your post on the very subject!
This seems like a good place to start.
Question is do you want to write a new Spin compiler from the ground up in Javascript, perhaps using pegjs?
If you do you'll have to be prepared to track changes in OpenSpin as it evolves in the future.
I'm not looking to develop a compiler. There are (or will be, for P2) plenty of those. I just want to parse the file enough to extract some of the information: constant values, label references, etc.
When I played with pegjs I found parsing Spin and PASM a bit of a challenge.
I forget the correct terminology for grammars and parsers but Spin/PASM is not a nice regular syntax like those evolved from ALGOL. It has that white space delimiting and everything is line based.
Perhaps I just did not think about it hard enough.
No, I think you are right about the parsing complexity of the language. From what I've seen of your PEG files, it looks like you've put quite a bit of thought into it!
I spent a long while getting to grips with peg and then the Spin/PASM syntax. And then that project timed out, half baked.
That is my second attempt at parsing Spin/PASM. Years back I wrote a mostly complete parser in Ada. Would you believe. Then I decided Ada really is a dead end, no matter that I am attracted to a lot of it's features.
As for syntax, this kind of thing in a DAT section is a doozy:
name byte "Hello" + "World"
You know what that compiles to ?
I don't know if anybody here uses Arduino, but this guy makes an extension for VS Community. Hopefully Parallax can support something similar with Prop chips.
What we have been discussing here is Visual Studio Code. This is an entirely different product. It written in Javascript (Well, Typescript). It runs on the Google V8 Javascript engine and uses Electron for the GUI. It is open source and cross platform.
VS Code is something worth investing time in.
I don't say this lightly, as a life long critic of all things Microsoft.
I guess I didn't explain that I knew the difference. I feel VS Community is too heavy to program a microchip on a daily basis. My mistake for not making it clearer. I was voicing support that I think Parallax should transition to supporting a Prop extension FOR VS CODE and move away from SimpleIDE. Not because SimpleIDE is a bad product, but because they can focus on the Prop details and leave the editor portion to the community. I was only pointing out the VisualMicro extension for VS Community as an example of what has been done before.
Perhaps everyone should list their parochialisms in their signature.
You and I are on the same page, a few of us have brought this up in the past here. VS Code is not the first program to come out that allows one to build an IDE on top of an editor. Parallax could have done that from the start with SimpleIDE, and probably as far back as the Prop Tool. From what I've been able to glean from the forums, it was a deliberate choice to start from scratch. Parallax want something tailored for their exact requirements which they believe will serve the education community best. And SimpleIDE is certainly far more simple than VS Code would be - they hit that nail on the head!
P1 has a big hobbyist/education lobby, so I think something simple, like Prop Tool is fine.
It also has tiny memory, so any program is going to be relatively simple for it.
But, P2 may attract more needy users who want to write big C/C++ programs.
For them, I think the full blown Visual Studio is hard to beat.
I would love to see a Prop2 extension for it.
I've been programming my Arduinos using the VisualMicro extension for the "full-blown" VS Community edition. While this works well, VS Community is a Kitchen Sink tool and I always feel I'm sifting thru all the phone and cloud programming options to blink an LED.
.
So I guess it's up to us . .
.
Anyhow, right now, all I've got working is:
* Syntax coloring (not much different than what I provided before)
* Parsing and displaying the first encountered error
* Completion suggestions for constants and labels
* Hover information for constants (the expression) and labels (the line)
Obviously, there is a lot more that could be added (e.g. intellisense for the instructions, constant expression evaluation, address calculations, snippets, etc). The question is: what would be most useful (and hopefully easy to implement)?
Aside: I think some of this would have gone a bit more smoothly if we had access to the pnut source, which I think would be about as close as we're going to get to a formal grammar. Has Chip talked about opening that up at all? Also, did he ever get a CLI implemented for pnut? It would be nice to call it directly from VSCode...
I haven't worked on the command-line interface yet. Just been getting back to documenting the architecture.
That's fine. I might look at Roy's work then. Actually, this exercise in writing a PEG for the P2 assembly language has been an interesting exercise. It's the first time I've written a grammar as complex as this one. And I'm still learning to do some parts of it properly. I'll be sure to publish it when I release the extension...
As for the CLI, I'll take it when I can get it. No hurry.
Thanks! I'll check that out.
Before I go down this path, is anyone aware of any multi-line expressions that I might encounter? The only thing that comes to mind is comment blocks. That complicates things slightly, but if it's the only one I encounter, I can deal with that easily enough. Can anyone think of any others?
My impression is that PEG likes a nice clean syntax that can be parsed in a recursive decent fashion, which Spin is certainly not. Spin is kind of line based like BASIC, but then it is block structured, with the cursed white space deliminting of Python. Then you basically have to change syntax from block to block, DAT, PUB, PRI, CON... It's a mess.
A few thoughts:
Perhaps it's better to pre-process the source and deal with the comments first. The output of that process would have no comments,
Rather than reduce the parser to parsing one line at a time do it a block at a time. If I understand correctly PUB, PRI, DAT etc have to start at the beginning of a new line. So if anything before that is in error you can always move forward, line by line, and find a new PUB, PRI or whatever to start parsing from again.
On the other hand why are you fighting this problem? If PEG detects a syntax error and bails then that is it, the user has to fix that error to move on. For example when I write Javascript in MS Visual Studio Code or Atom or Sublime editors I have JS lint running that will highlight all suspect code with red squiggles or whatever. But it will bail out if I I have a real syntax error. The rest of the file does not get checked.
Yeah, I was thinking along the same lines, somewhat. Start by detecting the block type (con, dat, etc.), then go from there. The issue with a failed parse is that I don't have access to the output from the part that did parse correctly. For instance, even if the entire CON section parsed, I wouldn't get back the AST for that section if I hit an error in a later section. But, as you are typing in new instructions, you will constantly be switching between a valid and invalid document (e.g. as you enter a constant expression, it's likely to be invalid for some number of keystrokes). So... how do I provide auto-complete of constants or labels, if the line your editing just happens to be invalid?
By switching to per-line parsing, I think I can get around a lot of these issues. Even when I eventually add support for Propeller 1 Spin (e.g. PUB and PRI sections), I believe that the block structures won't be too much of an issue. But, right now, I am only concerned about P2 assembly, which is somewhat less complicated.
Pre-processing the comments sounds like a good idea too.
So is there any work done to integrate P1/P2 Spin and PASM into Visual Studio Code?
Looking at the VS code site there is a generator able to import TextMate, Sublime uses TextMate like files and someone on this forum posted about Sublime-Support for Spin/Pasm.
I personally have zero experience with Yacc/Bision/Pegis/TextMate nor VS code. I just spend some time with @Heater to build some JavaScript based IDE based on some HtmlEditor. But there I just had to add new rules and keywords, basically the rest was there, But that Editor18 we worked on, was really just experimental and for fun.
Editor18.htm was a single 1.5xx MB file mainly JavaScript, but able to edit itself!
But integration in VS code would be really nice. So is there some way to use the Yacc stuff out of Spin2cpp and create a basic TextMate Syntax to import into VS code language generator?
just asking if there is already something to use?
Mike
I'm not aware of anyone working on such a thing.
There is a P1Spin+Pasm highlighter as a TextMateBundle, whatever that is.
Let see if I can get VS code to accept it
Weekend coming...
Mike