Spin compiler

dfletch · 2008-03-03 05:34

Hmm I suppose you're right rokicki and it's cool for the parser to live there. I think SF can support multiple related sub-projects under one main one.

rokicki, please add me as a dev and I'll figure out how to make a sub project. SourceForge account name: fletch (user ID 440)

▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔
Join us in the Un-Official Propeller IRC channel: irc.freenode.net #propeller
Newbies, oldies, programmers, math professors, and everyone in-between welcome!
Propeller IRC howto

rokicki · 2008-03-03 18:41

Okay, fletch is now a dev (and an admin so he can add other devs if he wants). Anyone else want to be a developer, either post here or
send a PM to me or dfletch. (My email address is my username [noparse][[/noparse]rokicki] on gmail.)

dfletch, it might be easier not to make it a subproject. Just give it a directory name (parser) or some such and that's good for now.
Note that we can have different binary and source releases under one project without giving them separate project or subproject
names. As long as no one starts breaking files or builds, there should be no trouble.

The project is currently under the GPL license (version 2, I hope); it's your code so you get to decide, but we do need to put it under
an open source license to use the code. I recommend GPL 2 to start with.

-tom

dfletch · 2008-03-03 18:45

Oh, I thought we were using LGPL? GPL is actually fine with me but I thought the library might get more use if it's a shade more commercial friendly. Hmm yeah the SF page says LGPL too. Was that what you meant?

This evening I'll get the code to compile again (broke it last night [noparse];)[/noparse] and import.

Thanks rokicki!

Cheers,

--fletch

▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔
Join us in the Un-Official Propeller IRC channel: irc.freenode.net #propeller
Newbies, oldies, programmers, math professors, and everyone in-between welcome!
Propeller IRC howto

dfletch · 2008-03-04 02:07

From now on you can download the latest version from SourceForge CVS. Here's the project page:

sourceforge.net/projects/spinc

I'll still keep you guys posted on milestones here though!

Cheers,

--fletch

UPDATE: View CVS repo directly here: spinc.cvs.sourceforge.net/spinc/spinparser/

▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔
Join us in the Un-Official Propeller IRC channel: irc.freenode.net #propeller
Newbies, oldies, programmers, math professors, and everyone in-between welcome!
Propeller IRC howto
spinc - open source Spin compiler

Post Edited (dfletch) : 3/4/2008 2:38:46 AM GMT

dfletch · 2008-03-11 07:25

It now parses a nice chunk of spin. I'm sure there's loads of issues still, but it parses more every day. The DAT section I'm sure is particularly weak, I haven't even tried it.

You can help out by pointing out specific issues here in this thread.

$ cvs -d\:\pserver:anonymous@spinc.cvs.sourceforge.net:/cvsroot/spinc co spinparser
$ cd spinparser
$ export CXXFLAGS="-Wall -DDEBUG_LEXER -DDEBUG_PARSER"
$ ./configure
$ make
$ cat test/simpletest.spin | src/SpinParserTest

Darn I'm sorry the forum software seems to not like the sequence (colon, p) treating it as one of the smilies. Please remove those two backslashes near the start of the CVS command.

Should build on any automake/autoconf friendly platform. Still good in Cygwin. Has been compiled and tested a bit on a Mac (install your dev tools CD). If you want to get "down & dirty" and mess with the parser and lexer (src/spinparse.ypp and src/spinlex.lpp), you should install bison and flex too (that Apple dev cd installs both).

I look forward to hearing all about how broken my parser is!

Cheers,

--fletch

UPDATE: Just remembered, need to mention: there is absolutely no object / library / library path support. Still hammering out the basics here! So currently you can only parse "self-contained" Spin files.

UPDATE2: If you post issues to the forum, please include as small a sample as you can give that triggers the error, and maybe the output of the test program (but I can get that too, so the Spin is more important).

▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔
Join us in the Un-Official Propeller IRC channel: irc.freenode.net #propeller
Newbies, oldies, programmers, math professors, and everyone in-between welcome!
Propeller IRC howto
spinc - open source Spin compiler

Post Edited (dfletch) : 3/11/2008 7:50:14 AM GMT

stevenmess2004 · 2008-03-11 07:38

dfletch said...
I look forward to hearing all about how broken my parser is!

Really? Give me a few minutes

stevenmess2004 · 2008-03-11 07:46

Just a note for those on macs. bash doesn't like it when you have more than 1 argument after the = sign. Don't have a workaround yet so I'm just running it with -DDEBUG_LEXAR at the moment.

dfletch · 2008-03-11 07:50

steven: My bad. Please put quotes around the whole thing. Just updated my post.

▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔
Join us in the Un-Official Propeller IRC channel: irc.freenode.net #propeller
Newbies, oldies, programmers, math professors, and everyone in-between welcome!
Propeller IRC howto
spinc - open source Spin compiler

dfletch · 2008-03-11 07:52

Oh and you won't get far with LEXAR. Is that a British/Australian spelling? My book calls it "lexer".

▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔
Join us in the Un-Official Propeller IRC channel: irc.freenode.net #propeller
Newbies, oldies, programmers, math professors, and everyone in-between welcome!
Propeller IRC howto
spinc - open source Spin compiler

stevenmess2004 · 2008-03-11 07:59

Yep, the quotation marks work. And since I copied and pasted the command into the terminal window it wasn't a problem. Only a problem when I type it into the forum from memory

Don't know about the spelling being British/Australian. I've just got it that way in my head and its hard to change

Well it appears to have gone through a lot more of my spin file than before so I'll post some results in a bit.

stevenmess2004 · 2008-03-11 08:07

Okay, here are the contents of the spin file

{{
Object for making windows
by Steven Messenger
}}
  
[b]VAR[/b] 'most of these will be replaced by pointers to dynamically allocated memory
  'word definition[noparse][[/noparse]5&#093;

  [b]byte[/b] elementCounter

  [b]word[/b] nextW
  [b]word[/b] xp
  [b]word[/b] yp
  [b]word[/b] w
  [b]word[/b] h
  [b]word[/b] wTitle

  [b]word[/b] data 

[b]PUB[/b] getStart
  [b]return[/b] @nextW '<<<<<<<<<<<<<<

[b]PUB[/b] setPosition(x,y)
  xp:=x
  yp:=y

[b]PUB[/b] setSize(width,height)
  w:=width
  h:=height

[b]PUB[/b] setTitle(titlePtr)
  wTitle:=titlePtr

[b]PUB[/b] register(pointer)|i
  [b]if[/b] data==0
    data:=pointer
    [b]return[/b]

  i:=data

  [b]repeat[/b]
    [b]if[/b] [b]word[/b][noparse][[/noparse]i&#093;==0
      [b]word[/b][noparse][[/noparse]i&#093;:=pointer
      [b]return[/b]

    i:=[b]word[/b][noparse][[/noparse]i&#093;

It gets all the way to the place I have marked which is a big improvement.

dfletch · 2008-03-11 09:13

Well now it gets a little farther!

I've been trying to implement PhilPi's idea about counting the start of line spaces in the lexer and passing "block start / end" to the parser. It works pretty well, but what I ran into is that the indent stack needs to be fully popped when a PUB or PRI block ends, or at end of file.

Not sure what's wrong now, but getting really late, time for bed.

Cheers,

--fletch

▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔
Join us in the Un-Official Propeller IRC channel: irc.freenode.net #propeller
Newbies, oldies, programmers, math professors, and everyone in-between welcome!
Propeller IRC howto
spinc - open source Spin compiler

stevenmess2004 · 2008-03-11 09:32

The indent only matters sometimes. For example this

PUB foo
temp:=1
   temp:=3
temp:=9
            temp:=1

doesn't care about indenting. Only things like if, case and repeat do. Don't know if this will make it easier or harder.

hippy · 2008-03-11 14:12

PhiPi mentioned the indentation rules earlier. In the current Spin compiler the start of block ( PUB, IF, REPEAT ) sets how far in the following block statements' indentation must be ( => +0 for PUB/PRI, => +1 for the rest ? ) but then indentation is allowed to drift in and out.

I'd add a command line switch so the indentation of the first line of a block were taken and all statements within that block should subsequently line up with it. I'd also required the block after PUB/PRI to be indented. Don't worry about this for now though.

dfletch · 2008-03-11 17:09

The way hippy describes how a command line switch should change the behavior - that's what it's dong now by default. So for example:

PUB foo
  a := 1      ' indent stack is pushed with value "2"
  if b == 2   ' indent is still "2", no stack change
    b := 5    ' indent is now "4", push "4"
PUB bar       ' pop "4" and "2" before we can parse this...
  ...

Each time the stack is pushed a BLOCK_START token is passed to the parser. Upon pop, BLOCK_END is sent.

I'm not quite sure how to deal with the randomness of stevenmess2004's post. Is that really right? I'm a little confused! If true, I could have this stack pushing and popping only in repeat / if statements, not across all of PUB/PRI.

Cheers,

--fletch

▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔
Join us in the Un-Official Propeller IRC channel: irc.freenode.net #propeller
Newbies, oldies, programmers, math professors, and everyone in-between welcome!
Propeller IRC howto
spinc - open source Spin compiler

Phil Pilgrim (PhiPi) · 2008-03-11 18:23

Indentation is only significant for control constructs, not for section headers. The algorithm I suggested takes "drift" into consideration, permitting it when it's not syntactically significant.

-Phil

dfletch · 2008-03-11 18:32

Phil, by "control constructs" do you mean if, repeat, etc? Here's what I'm not seeing:

PUB Start
  a := 1
    a := 2
 a :=3

  if a == 1
    a := 2

      b := 2 ' what the heck do we do in this case?

    c := 2 ' now back to sane indenting - does the IF continue?

 d := 3 ' this doesn't align with "if", is it enough to close the block?

I guess my issue here is how do I detect when a block like that IF statement ends? Any non-empty line with less indents than the original IF?

Cheers,

--fletch

▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔
Join us in the Un-Official Propeller IRC channel: irc.freenode.net #propeller
Newbies, oldies, programmers, math professors, and everyone in-between welcome!
Propeller IRC howto
spinc - open source Spin compiler

Phil Pilgrim (PhiPi) · 2008-03-11 18:41

If you follow my algorithm to the letter, you'll see what's supposed to happen. Random indentation changes that don't align with, or to the left of, a prior control construct (IF, REPEAT, CASE, etc.) are okay and don't mean anything. Try it; you'll see what I mean.

-Phil

Chuck Rice · 2008-03-11 18:47

dfletch said...
I guess my issue here is how do I detect when a block like that IF statement ends? Any non-empty line with less indents than the original IF?

I have not followed this whole thread, so I may be missing some info, but if I understand the question, I think that you want to save the column when you start a block, then test each line for greater or equal to that column. When you see a line that is less than, you end the block. Of corse you need to stack each block to account for nested control blocks.

dfletch · 2008-03-11 18:49

Ok was having trouble finding that OP until a moment ago, was being too lazy to scan the whole thread

Yeah, it sounds like I'm pretty close, got it mostly right from memory. My code goes a little over the top though, pushing the stack when it finds more indentation than the top of stack, as opposed to only pushing when hitting a control construct.

Thanks Phil, this is a slick way to handle the whitespace issue

Update coming tonight with this change and more.

▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔
Join us in the Un-Official Propeller IRC channel: irc.freenode.net #propeller
Newbies, oldies, programmers, math professors, and everyone in-between welcome!
Propeller IRC howto
spinc - open source Spin compiler

dfletch · 2008-03-13 07:30

Gets quite a bit further on stevenmess2004's code after that indent stack change. Committed.

I think it's pretty darn close to parsing the whole thing, but now Flex is being weird... the regex:

[noparse][[/noparse]a-z_][noparse][[/noparse]a-z0-9_]*

Seems to be matching a newline at the end for some reason. That regex is pretty simple so I'm guessing it's something I'm doing wrong in Flex.

So now it chokes on the line that says "i:=data" because it doesn't know that i is a local var - when it spied |i in the method declaration, the \n came along with it, giving me a value of "i\n" for the CNAME. Weird, right?

I'll look closer tomorrow. G'night.

Cheers,

--fletch

▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔
Join us in the Un-Official Propeller IRC channel: irc.freenode.net #propeller
Newbies, oldies, programmers, math professors, and everyone in-between welcome!
Propeller IRC howto
spinc - open source Spin compiler

stevenmess2004 · 2008-03-13 07:46

That great dfletch. I'll have a look tomorrow night.

dfletch · 2008-03-16 05:18

Woohoo!

It now parses stevenmess2004's entire test.

Not 100% positive that it's *correctly* parsing it, but no more syntax errors

The last issue I was having turned out to be totally simple once I thought about it. The lexer reuses it's buffer, right? Otherwise it would be allocating and freeing all the time. Anyway, so when any of the terminal productions in the parser used the string this was no problem - it had just come from the lexer immediately before that piece of code. If I used the string in a nonterminal parser production, it is matched later - the lexer has already moved on to other stuff, so of course the string has changed! Very easy solution: copy the string before passing it to the parser.

Fixed a couple of minor probs with the parser - we had all forgotten about the VariableExpression Size[noparse][[/noparse]VariableExpression] (e.g. WORD[noparse][[/noparse] i ] ). There was a couple extra unneeded T_LINE_ENDs.

So who's going to be the next one to break it?

Cheers,

--fletch

▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔
Join us in the Un-Official Propeller IRC channel: irc.freenode.net #propeller
Newbies, oldies, programmers, math professors, and everyone in-between welcome!
Propeller IRC howto
spinc - open source Spin compiler

stevenmess2004 · 2008-03-16 06:46

Okay, it isn't seeming to recognize DAT variables as variables. This code generates a syntax error

''datTest.spin
CON
  abc=123

DAT
inDAT long 1234

PUB aPUB
  inDAT:=abc

This also generates a syntax error so it might be more of a problem just with DAT sections

CON
  abc=123

DAT
inDAT long 1234

VAR
long inVAR

PUB aPUB
  inVAR:=abc

Good work so far. I've looked through some of the code for it and am starting to get an idea of how it works. Slowly

dfletch · 2008-03-16 07:08

Yes, Steven the DAT section is where there's still a couple of parser conflicts that I don't quite understand yet. I will focus on this next.

Cheers,

--fletch

▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔
Join us in the Un-Official Propeller IRC channel: irc.freenode.net #propeller
Newbies, oldies, programmers, math professors, and everyone in-between welcome!
Propeller IRC howto
spinc - open source Spin compiler

dfletch · 2008-03-16 19:42

OK!

It parses your DAT sections and recognizes DAT variables now!

Your code exposed another bug too: constant names were not recognized (e.g. inDAT:=abc) but it's working now.

Moving right along here! I think I'll start on the object tree next.

Cheers,

--fletch

▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔
Join us in the Un-Official Propeller IRC channel: irc.freenode.net #propeller
Newbies, oldies, programmers, math professors, and everyone in-between welcome!
Propeller IRC howto
spinc - open source Spin compiler

dfletch · 2008-03-16 19:50

Oh, here's a snapshot if anyone else wants to get involved but doesn't want to mess with CVS. You'll still need automake/autoconf/libtool/gcc at least for now.

If anyone wants to create a VC++ project file, that would be great. I don't know how to do it and make it relative and have the project in a separate dir (e.g. ./proj at the top, but it looks in ../src for sources and headers). If anyone is interested in doing that, you might want to pop on IRC to discuss. If someone makes this VC++ project, I'll add it to CVS.

Again, sorry for the .txt extension. Attachment manager doesn't like .tgz Please remove .txt from the name.

Cheers,

--fletch

▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔
Join us in the Un-Official Propeller IRC channel: irc.freenode.net #propeller
Newbies, oldies, programmers, math professors, and everyone in-between welcome!
Propeller IRC howto
spinc - open source Spin compiler

dfletch · 2008-03-25 19:04

Just floating the thread again to see if I can stir up a couple more beta testers.

I've been ignoring it for a week or so working on my console but I plan on getting back to this tonight.

If there's no major issues in the next 8 hours, I'm going to build out the object support, which will make it easy to test the core Propeller library code.

If that goes well, on to building the output tree!

Cheers,

--fletch

▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔
Join us in the Un-Official Propeller IRC channel: irc.freenode.net #propeller
Newbies, oldies, programmers, math professors, and everyone in-between welcome!
Propeller IRC howto
spinc - open source Spin compiler

Harrison. · 2008-03-26 02:51

I have created a visual c++ 2005 project/solution for spinparser. This means we now have windows binaries for testing!

Attached is the source with the vs2005 project files. All the visual studio related stuff, including binaries, are in the spinparser/vs2005 folder so there will be no confusion between linux and windows stuff. I've also attached a screenshot of the environment for those who are curious to see what it looks like.

**EDIT: Updated project properties to only dump parser/lexer debug information when in the Debug build configuration.
**EDIT #2: Now builds a dll and an exe.

Harrison

Post Edited (Harrison.) : 3/27/2008 1:28:29 AM GMT

dfletch · 2008-03-26 03:04

OK great, thank you Harrison.

The relevant project files have been added to the CVS repo.

Cheers,

--fletch

UPDATE: CVS updated with the above changes and fixed a missing return value that GCC didn't mind but VC hates.

UPDATE2: There's a weird issue "error C3861: 'read': identifier not found". I'm replacing this code in a minute though with fread.

UPDATE3: It builds in both GCC and VC++ now. Thanks again Harrison! Committed to CVS.

▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔
Join us in the Un-Official Propeller IRC channel: irc.freenode.net #propeller
Newbies, oldies, programmers, math professors, and everyone in-between welcome!
Propeller IRC howto
spinc - open source Spin compiler

Post Edited (dfletch) : 3/26/2008 4:27:55 AM GMT

Spin compiler

Comments