New PNut-TS compiler for P2 development on Windows, Mac, Linux (yes, RPi)

Stephen Moraco

PNut-TS is the result of porting PNut for Windows to a platform-agnostic language (in this case, TypeScript). I collaborated with Chip Gracy from Mid-Jan to now porting this compiler. We're taking a small break while Chip completes the Structure work in the upcoming PNut_v45. Then we'll port the structure changes to PNut-TS.

We will officially release PNut-TS when we've incorporated the Structure support, appearing in PNut_v45. Meanwhile, we are making this v1.43.0 version available for installation and testing against PNut_v43. We have over 100 .spin2 files in our Regression Test Suite, we have coverage testing, and we've each compiled and tested our more significant projects against this version. Even with all our testing, we'd like a few more eyes on this to ensure we haven't missed any typical code shape. We believe that with our coverage testing, we haven't missed code generation issues, but there still might be things like code-flow-control or file handling issues that we haven't yet found. We don't believe this is likely, but we want to feel more comfortable after more projects have been compiled with v1.43.0 and found to be correct.

(Need v43? Then download PNut_v43 here.

PNut-TS is found at its new Public Repository, while the latest Releases are also found there at the Releases Page

We have written a process for testing for you. Please follow these instructions.

We are looking for someone running Linux on an Intel CPU to test the packaging for Linux X64. If you have such a machine, please download and install PNut-TS per our instructions. Then, test it by simply running pnut_ts—- help once you have it installed. If the help command succeeds, that's all we need to know!

We look forward to hearing what you find. Thank you for helping us test this early version!

• Stephen & Chip

Wuerfel_21

@"Stephen Moraco" said:
We are looking for someone running Linux on an Intel CPU to test the packaging for Linux X64. If you have such a machine, please download and install PNut-TS per our instructions. Then, test it by simply running pnut_ts—- help once you have it installed. If the help command succeeds, that's all we need to know!

very cool.

(Though technically this is an AMD cpu. Do not call x86_64 "Intel", that's Mac brain poisoning.)

EDIT: It would appear that you have erroneously uploaded the ARM version

Wuerfel_21

Also, you should probably have the version number in the download file name, otherwise it get confusing quickly.

Stephen Moraco

@Wuerfel_21 said:
Also, you should probably have the version number in the download file name, otherwise it get confusing quickly.

Ahh, good idea! I will update it in a day or so.

Stephen Moraco

FYI- new .zip files with version suffix have been uploaded to release page. Also fixed content of Linux-x64-...zip file.

Wuerfel_21

Some very initial feedback:

• Output filename is .bin. It should really either be .binary (matching PropTool and flexspin) or .bix (short extension for native P2 executables on FAT filesystem). .bin implies a generic data file (It is also used for P1 executables in VentilatorOS, but that's moreso historically motivated and it should really not do that - note here that I share the same SD cards between P1 and P2 boards)
• Every command regardless of success prints out this bulls‌hit:

pnut-ts: Done
pnut-ts: ERROR- Error: Command failed: /home/ada/Downloads/pnut_ts/pnut_ts -v
node:internal/modules/cjs/loader:949
  throw err;
  ^

Error: Cannot find module '/home/ada/Downloads/pnut_ts/-v'
    at Module._resolveFilename (node:internal/modules/cjs/loader:946:15)
    at Function._resolveFilename (pkg/prelude/bootstrap.js:1951:46)
    at Module._load (node:internal/modules/cjs/loader:787:27)
    at Function.runMain (pkg/prelude/bootstrap.js:1979:12)
    at node:internal/main/run_main_module:17:47 {
  code: 'MODULE_NOT_FOUND',
  requireStack: []
}

Node.js v18.5.0

• Typo in the help command: "optons"
• Time for assembling a simple PASM-only program is over an order of magnitude worse than flexspin. 540ms vs 25ms, or rather "noticable delay" vs "instant". Will need to check how that scales to larger programs. (Have to go look for a large-ish program I have that doesn't need flexspin-only features...)

Wuerfel_21

Also, preprocessor's comment parsing is mega busted:

{
Crustaceous crustaceans encrust thy crusty code
'}
#define HMMM
DAT
org
jmp #$

It seems to think the #define afterwards is still part of the comment and leaves it in the code and then the compiler chokes on it.
This pattern happens very frequently.

Stephen Moraco

Ada, thank you for this early feedback!
Look for a 1.43.1 release in the next couple of days that "finishes" (my oops) the intended PreProcessor behavior and fixes the issues you've cited.

-Stephen

Wuerfel_21

It seems the slow compile time is mostly correlated with including large FILEs in a DAT section. So I guess there's some horrible per-byte loop that runs on either the FILE includes or the whole output binary.

Stephen Moraco

NEWS

Excerpt from our Latest Release page:

v1.43.1
Update Preprocessor logic

• Finish implementation of PreProcessor (Oops!) Ada, my apologies! Maybe I should have completed this before asking you to test it? ;-)
• Clean up output under error conditions
• Compatible with PNut_v43.exe

If you see any issues with PNut-TS please let us know by filing new issues at our code repo. (or email me directly.)

You can always review the PNut-TS ChangeLog from within your install directory or at github for more details.

Stephen Moraco

@Wuerfel_21, thank you, as always, for your detailed feedback. Our tools are always improving because of your help!

@Wuerfel_21 said:

• Output filename is .bin. It should really either be .binary (matching PropTool and flexspin) or .bix (short extension for native P2 executables on FAT filesystem). .bin implies a generic data file (It is also used for P1 executables in VentilatorOS, but that's moreso historically motivated and it should really not do that - note here that I share the same SD cards between P1 and P2 boards)

On the one hand, I don't disagree with you. But on the other hand... I chose .bin specifically because it differs from flexspin, as I'm running both compilers all the time during our development. I thought it was a reasonable choice since, as you know, this suffix has been used for decades (since the 70's and 80's) for embedded systems and microcontrollers. Can we put up with this for the time being?

@Wuerfel_21 said:
It seems the slow compile time is mostly correlated with including large FILEs in a DAT section. So I guess there's some horrible per-byte loop that runs on either the FILE includes or the whole output binary.

Um, good to know and you are correct. Our port was 90% or so algorithm as written. Your suggestion is correct as this is a byte loop, and there are more than one that could/should be addressed to improve performance. I've filed this as Issue #2, so we track this as an upcoming improvement. I suggest we continue to file these issues as we discover which areas are most impactful if we fix them (such as the one you cite here.)

JonnyMac

I tried to use PNut-TS as an external tool for Spin Tools IDE and it failed.

Here's the report

C:\Users\jmcph\Documents\pnut_ts\pnut_ts.exe -I C:\Users\jmcph\Documents\Propeller_Tool\Library jm_flasher.spin2
pnut-ts: (DBG) foundJest=(false), runningCoverageTesting=(false)
pnut-ts: * Propeller Spin2/PASM2 Compiler 'pnut_ts' (c) 2024 Iron Sheep Productions, LLC., Parallax Inc.
pnut-ts: * Version 1.43.1, Build date: 9/17/2024
pnut-ts: * pnut_ts -I C:\Users\jmcph\Documents\Propeller_Tool\Library jm_flasher.spin2
arguments: [ [length]: 0 ]
combArguments: [
  'C:\\Users\\jmcph\\Documents\\pnut_ts\\pnut_ts.exe',
  'C:\\snapshot\\PNut-TS\\out\\pnut-ts.js',
  '-I',
  'C:\\Users\\jmcph\\Documents\\Propeller_Tool\\Library',
  'jm_flasher.spin2',
  [length]: 5
]
options: {
  Include: [
    'C:\\Users\\jmcph\\Documents\\Propeller_Tool\\Library',
    'jm_flasher.spin2',
    [length]: 2
  ],
  filename: undefined
}
pnut-ts: ERROR- [31mMissing filename argument[0m
pnut-ts: Aborted!

Here's my PNut-TS setup for Spin Tools

I wondered if space in the path might be a problem so I removed it.

PNut uses -r for RAM and -f for flash. I don't see either of those options for PNut-TS.

Wuerfel_21

@"Stephen Moraco" said:
@Wuerfel_21, thank you, as always, for your detailed feedback. Our tools are always improving because of your help!

@Wuerfel_21 said:

• Output filename is .bin. It should really either be .binary (matching PropTool and flexspin) or .bix (short extension for native P2 executables on FAT filesystem). .bin implies a generic data file (It is also used for P1 executables in VentilatorOS, but that's moreso historically motivated and it should really not do that - note here that I share the same SD cards between P1 and P2 boards)

On the one hand, I don't disagree with you. But on the other hand... I chose .bin specifically because it differs from flexspin, as I'm running both compilers all the time during our development. I thought it was a reasonable choice since, as you know, this suffix has been used for decades (since the 70's and 80's) for embedded systems and microcontrollers. Can we put up with this for the time being?

Well the right thing to do there would really be to override the output filenames, i.e. compile with flexspin -2 whatever.spin2 -o whatever.flex.binary and equivalent for the other compilers. Maybe annoying if you're not scripting it... .bin being used by everyone is the problem, you can't tell it apart! (To be fair,.binary is only better because I've never seen it outside of Parallax.) E.g. I have a terminal autocomplete set up so it automatically finds .binary files when completing loadp2 commands. If I added .bin, it would find any .bin file in the directory. Now imagine developing something like my emulators where you might have a non-P2 .bin file in the directory because that's the ROM you're loading into it (at initial bringup I just use FILE to hardcode it - SD loading happens afterwards)... annoying!

@Wuerfel_21 said:
It seems the slow compile time is mostly correlated with including large FILEs in a DAT section. So I guess there's some horrible per-byte loop that runs on either the FILE includes or the whole output binary.

Um, good to know and you are correct. Our port was 90% or so algorithm as written. Your suggestion is correct as this is a byte loop, and there are more than one that could/should be addressed to improve performance. I've filed this as Issue #2, so we track this as an upcoming improvement. I suggest we continue to file these issues as we discover which areas are most impactful if we fix them (such as the one you cite here.)

I guess whatever native I/O function Chip's version used is vastly quicker than what node.js offers. I posted the particular program those timings are from into the issue.

Stephen Moraco

@JonnyMac said:
I tried to use PNut-TS as an external tool for Spin Tools IDE and it failed.

@JonnyMac, geez, sorry to hear. Could you please try one thing for me? Add a '--' before the filename? I suspect my command-line parser is having issues without it. I'll fix this.
Also, if you are willing, could you .zip up your project (or archive it)? Then email it to me? I'll test your code before I push another update.

Actual test findings (cause of the problem): Any option that takes an argument causes a misparse of the following non-arguments options, like the filename. The workaround is to place '--' before the filename. (If possible, I will add detection of this case and handle it internally.)

@JonnyMac said:
PNut uses -r for RAM and -f for flash. I don't see either of those options for PNut-TS.

Yes, you found that! ;-) Okay, long story short, this version does not have a downloader built in. After I removed it, I did further work on getting USB Serial to distribute with a typescript application (in the Spin2 extension), and in doing so, I discovered the solution. I can add the downloader back into the compiler in a future update.

JonnyMac

Tried. No joy.

Project is single file -- no objects needed. It's attached.

I can add the downloader back into the compiler in a future update.

Okay, I'm going to need that for use with Spin Tools. I assume the -r and -f will apply to the downloader.

JonnyMac

I would suggest you install Spin Tools IDE on your system so that you can test locally. Marco has Windows, Mac, and Linux versions.
-- https://www.maccasoft.com/spin-tools-ide/

Note: The Parallax standard path has a space in "Propeller Tool" so I'm putting the path in quotes. The output doesn't seem to indicate a problem with the space, but you may want to check on that. I replaced the space with and underscore and Propeller Tool simply made a new Propeller Tool folder and library!

JonnyMac

I tried several more iterations, including forcing the full path of the Spin2 file -- still not working.

C:\Users\jmcph\Documents\pnut_ts\pnut_ts.exe -I C:\Users\jmcph\Documents\Propeller Tool\Library --D:\documents\consulting\elte\_code_\testing\p2_flash\binary_files\jm_flasher.spin2
PNut-TS: [31merror: unknown option '--D:\documents\consulting\elte\_code_\testing\p2_flash\binary_files\jm_flasher.spin2'
[0mpnut-ts: (DBG) foundJest=(false), runningCoverageTesting=(false)
pnut-ts: * Propeller Spin2/PASM2 Compiler 'pnut_ts' (c) 2024 Iron Sheep Productions, LLC., Parallax Inc.
pnut-ts: * Version 1.43.1, Build date: 9/17/2024
pnut-ts: * pnut_ts -I C:\Users\jmcph\Documents\Propeller Tool\Library --D:\documents\consulting\elte\_code_\testing\p2_flash\binary_files\jm_flasher.spin2
pnut-ts: ERROR- [31mFile [--D:\documents\consulting\elte\_code_\testing\p2_flash\binary_files\jm_flasher.spin2] does not exist or is not a .spin2 file![0m
pnut-ts: Aborted!

The file does exist -- it works with Propeller Tool and Spin Tool's internal compiler.

Stephen Moraco

@JonnyMac said:
Tried. No joy.

@JonnyMac not quite right. The -- is used to end option processing before the next text. The double dash must have a space on both sides. (before the filename in your example.)

Stephen Moraco

@JonnyMac And yes, once I fix the command-line parsing, I'll test locally with that tool and let you know when it's working and released.

JonnyMac

@"Stephen Moraco" said:
@JonnyMac not quite right. The -- is used to end option processing before the next text. The double dash must have a space on both sides. (before the filename in your example.)

Still not working.

C:\Users\jmcph\Documents\pnut_ts\pnut_ts.exe -I C:\Users\jmcph\Documents\Propeller Tool\Library -- D:\documents\consulting\elte\_code_\testing\p2_flash\binary_files\jm_flasher.spin2
pnut-ts: (DBG) foundJest=(false), runningCoverageTesting=(false)
pnut-ts: * Propeller Spin2/PASM2 Compiler 'pnut_ts' (c) 2024 Iron Sheep Productions, LLC., Parallax Inc.
pnut-ts: * Version 1.43.1, Build date: 9/17/2024
pnut-ts: * pnut_ts -I C:\Users\jmcph\Documents\Propeller Tool\Library -- D:\documents\consulting\elte\_code_\testing\p2_flash\binary_files\jm_flasher.spin2
D:\documents\consulting\elte\_code_\testing\p2_flash\binary_files\jm_flasher.spin2:1:error:Expected "}}"
[31mD:\documents\consulting\elte\_code_\testing\p2_flash\binary_files\jm_flasher.spin2:1:error:Expected "}}"[0m
pnut-ts: Done
pnut-ts: ERROR- [31mError: Command failed: C:\Users\jmcph\Documents\pnut_ts\pnut_ts.exe -v
node:internal/modules/cjs/loader:949
  throw err;
  ^
Error: Cannot find module 'D:\documents\consulting\elte\_code_\testing\p2_flash\binary_files\-v'
    at Module._resolveFilename (node:internal/modules/cjs/loader:946:15)
    at Function._resolveFilename (pkg/prelude/bootstrap.js:1951:46)
    at Module._load (node:internal/modules/cjs/loader:787:27)
    at Function.runMain (pkg/prelude/bootstrap.js:1979:12)
    at node:internal/main/run_main_module:17:47 {
  code: 'MODULE_NOT_FOUND',
  requireStack: []
}
Node.js v18.5.0
[0m

Wuerfel_21

That actually does mean that it's working, it said pnut-ts: Done. The nonsense afterward is just there because ???

Rayman

I’m curious how you did this @"Stephen Moraco" …

Isn’t pnut written in 80x86 assembly ?
The compiler part…

Stephen Moraco

@Rayman said:
I’m curious how you did this @"Stephen Moraco" …

Isn’t pnut written in 80x86 assembly ?
The compiler part…

Chip and I translated all the assembly and Pascal by hand into TypeScript. Line by line... ;-)

Stephen Moraco

Note to all my early testers.

It looks like I've fallen victim to my own scripted testing techniques (not enough variation of command lines I use and not enough testing on all the supported platforms.)

It appears that we've found that the command-line parser library I'm using behaves differently on various platforms. While I'm not seeing parse issues on macOS, many of you are seeing parse issues on Linux and Windows.

So, I'm off testing the parser I'm using on all platforms. I'll post a new point update when it all behaves the same (1.43.2?).

More soon.

Stephen Moraco

NEWS

Excerpt from our Latest Release page:

v1.43.2
Fix Windows/Linux command-line processing logic and Fix Regression caused by Preprocessor Upgrade

• Repair command-line option parsing (on Windows/Linux)
• BUGFIX fixed elementizer issues caused by preprocessor changes
• Compatible with PNut_v43.exe

If you see any issues with PNut-TS, please let us know by filing new issues at our code repo (or emailing me directly).

You can always review the PNut-TS ChangeLog from within your install directory or at github for more details.

JonnyMac

Seems to have solved the compile issue I was having when using it with Spin Tools, though it seems irritated by empty VAR sections.

When can we expect loading the .BIN file into the P2 from PNut-TS?

Just a comment: You seem to be using ANSI control codes in your output. Do you want to do that given you don't know what terminal the user will have?

Stephen Moraco

@JonnyMac said:
Seems to have solved the compile issue I was having when using it with Spin Tools, though it seems irritated by empty VAR sections.

Good to hear. Does PNut have the same issue with Empty VAR sections? I haven't tested that...

JonnyMac

Good to hear. Does PNut have the same issue with Empty VAR sections? I haven't tested that...

No -- not that I'm aware of, anyway. I don't know that the empty section is stopping the compilation, but when I see reports I tend to react. I've never had an issue with my code in Propeller Tool, FlexProp, or Spin Tools.

dgately

On macOS (Sequoia 15.0.1 on M2 MacBook Pro)... pnut_ts (Version 1.43.2, Build date: 9/22/2024) does not compile a simple .spin2 code file if a VAR (var) section has no variable defined... spin2 template files (such as the one within Spin Tools IDE) can have VAR sections that are named but include no defined types.

''The code with defined variable:
con
  CLK_FREQ = 200_000_000
var
  byte someVar      '' a defined variable
pub start()
  ' initialization
  repeat
    ' loop

pnut_ts compile:

~ % pnut_ts -l -O exampleWithVAR.spin2
pnut-ts: * Propeller Spin2/PASM2 Compiler 'pnut_ts' (c) 2024 Iron Sheep Productions, LLC., Parallax Inc.
pnut-ts: * Version 1.43.2, Build date: 9/22/2024
pnut-ts: * pnut_ts -l -O exampleWithVAR.spin2
pnut-ts: Wrote exampleWithVAR.obj (41 bytes)
pnut-ts: Wrote exampleWithVAR.lst
pnut-ts: Wrote exampleWithVAR.bin (5284 bytes)
pnut-ts: Done

''The code with NO defined variable:
con
  CLK_FREQ = 200_000_000
var      '' no defined variable
pub start()
  ' initialization
  repeat
    ' loop

pnut_ts compile:

~ % pnut_ts -l -O exampleNoVAR.spin2
pnut-ts: * Propeller Spin2/PASM2 Compiler 'pnut_ts' (c) 2024 Iron Sheep Productions, LLC., Parallax Inc.
pnut-ts: * Version 1.43.2, Build date: 9/22/2024
pnut-ts: * pnut_ts -l -O exampleNoVAR.spin2
pnut-ts: Wrote exampleNoVAR.obj (53 bytes)
exampleNoVAR.spin2:4:error:Expected a unique variable name, BYTE, WORD, LONG, ALIGNW, or ALIGNL (m0E0)
exampleNoVAR.spin2:4:error:Expected a unique variable name, BYTE, WORD, LONG, ALIGNW, or ALIGNL (m0E0)
pnut-ts: Done

dgately