Shop OBEX P1 Docs P2 Docs Learn Events
A retromachine Basic interpreter [beta] — Parallax Forums

A retromachine Basic interpreter [beta]

pik33pik33 Posts: 2,387
edited 2023-09-16 15:21 in Propeller 2

The goal is to turn P2-EC32 into something like 8-bit Atari on steroids :)

This of course means a standalone Basic interpreter.

I never wrote any interpreter before, except PC-Softsynth, but the music language to interpret there was much simpler than normal Basic. No functions, no expressions, only commands with parameters. This means I have a lot to learn during the process.

There were several bugs in Flexprop that prevented me to go further with the project, but now these bugs are gone and the intepreter started to interpret something:

As the first home computer I used was 8-bit Atari, I want this "look and feel" and that's why I used these colors as default on the screen. Also, the keyboard generates Atari 8-bit style clicks - instant nostalgia.

Of course these things will be configurable... as in real Atari :)

The project repository is here: https://gitlab.com/pik33/P2-retromachine/-/tree/main/Propeller/Basic https://github.com/pik33/P2-Retromachine-Basic.
As it is now, it can only interpret and execute several graphics related commands, but at least the line processing and tokenization work including multiple commands separated by a colon. All the rest is to do..

The project uses HDMI at P0, AV at P8 and USB at P16

«134567

Comments

  • pik33pik33 Posts: 2,387

    Variables now work. To do next: programming mode (now it is still only immediate)

  • mparkmpark Posts: 1,305

    Awesome!

  • Christof Eb.Christof Eb. Posts: 1,214

    @pik33 said:
    The goal is to turn P2-EC32 into something like 8-bit Atari on steroids :)

    This of course means a standalone Basic interpreter.

    I never wrote any interpreter before, except PC-Softsynth, but the music language to interpret there was much simpler than normal Basic. No functions, no expressions, only commands with parameters. This means I have a lot to learn during the process.

    There were several bugs in Flexprop that prevented me to go further with the project, but now these bugs are gone and the intepreter started to interpret something:

    As the first home computer I used was 8-bit Atari, I want this "look and feel" and that's why I used these colors as default on the screen. Also, the keyboard generates Atari 8-bit style clicks - instant nostalgia.

    Of course these things will be configurable... as in real Atari :)

    The project repository is here: https://gitlab.com/pik33/P2-retromachine/-/tree/main/Propeller/Basic

    As it is now, it can only interpret and execute several graphics related commands, but at least the line processing and tokenization work including multiple commands separated by a colon. All the rest is to do..

    The project uses HDMI at P0, AV at P8 and USB at P16

    Hi,
    nice project!

    As you are doing this from scratch, perhaps you might include two methods to speed up?
    I have studied some basic interpreters. One of the most interesting, I found, was for HP-85.

    1. Instead of searching for lines at every GOTO, they did a complete compile scan before executing and stored addresses for target lines and for variables.
    2. They compiled every line to some sort of RPN language. Decompile for LIST. As far as I understand, the speed is improved, because you need less instructions.

    Christof

  • pik33pik33 Posts: 2,387

    I am doing this from scratch, and I did think about a some kind of a "precompiler", but at this stage it cannot even run a program. It only executes immediate commands. The first program... maybe today, but I will be using a slow and simple way at first to make it simply work and then upgrade it to do things faster.

  • MicksterMickster Posts: 2,698

    @"Christof Eb."

    I think I have something similar. Bypic is no-longer supported but that doesn't bother me because it simply works. I duplicated the site and have a copy of the PIC32MX hex file. All that's required is the MCU and a capacitor and we're up and running.

    Rgds,

    Craig

  • pik33pik33 Posts: 2,387
    edited 2023-07-17 13:53

    I started to think about "how to write a precompiler"

    The interpreter's expression evaluator algorithm was based on this Basic interpreter: https://github.com/adamdunkels/ubasic. It implements tree-like call hierarchy to do higher priority operator first and then lower level (2 levels implemented). It tries to retrieve arguments first, then do the operation.

    But the argument-argument-operator is something that Forth uses, called the reverse Polish notation

    To check this I addes a simple logger to the line interpreter to see how it actually works.

    Got this:

    and yes, it is!. So instead of doing these function at once I need to implement a stack and then, for the second operation on the picture, compile something like this (pseudocode):

    push #0
call getintvar
push result
push #0
call greintvar(0)
push result
push constant (the interpreter already retrieved the constant)
push constant
call div 
push result (maybe these functions should push it automatically)
call mul (and of course push)
call plus
push #1
call assigntoint

    So the precompiled program would execute in Forth style...

    ... we have a Forth in P2 ROM....... can I call it from the inside of Flexprop compiled code and can I use it do to the work?... that are the questions. To experiment :)

  • MicksterMickster Posts: 2,698

    Brilliant idea :+1:B)

    Craig

  • pik33pik33 Posts: 2,387
    edited 2023-07-26 14:54

    First program run.

    The line, after it is entered, is precompiled to the reverse Polish notation style. Then the result and the line itself is written to PSRAM.

    The result is an array of tokens. These tokens are indexes to a function table. Some of them contains also a parameter

    "run" starts from PSRAM address #0, reads the precompiled line to the buffer in the HUB RAM, then executes it. The execution is done like this:

    sub execute_immediate_line 

dim cmd as asub

for lineptr_e=0 to lineptr-1
cmd=commands(compiledline(lineptr_e).result_type)
cmd
next lineptr_e

end sub

    The line contains a pointer to the next line in PSRAM, so the run procedure does a loop, reading lines until it encounters $FFFFFFFF, an end of program flag.

    The interpreter is now a big mess than needs cleaning before I can go further, and also while it can do something like this as the immediate line:

    color 40: plot 100,100

    in the program mode only executes the first command. This is to do also. And stilll no if, goto, for, etc commands to make loops and branches.

  • MicksterMickster Posts: 2,698

    Is the goal to have a resident, interactive BASIC onboard?

    :)

    Craig

  • pik33pik33 Posts: 2,387
    edited 2023-07-26 17:33

    @Mickster said:
    Is the goal to have a resident, interactive BASIC onboard?

    :)

    Craig

    Yes.

    And it can expose what I have in my drivers already. Graphics, sprites and audio. And also USB HID via Wuerfel's new USB driver (so I can use a keyboard without problems, but also mice, joysticks and gamepads)
    It is intended for PSRAM based systems, as we have to have a proper framebuffer, so it now works with P2-EC32 and Eval board with 4-bit PSRAM attached. These are platforms I have and I can test this with. So I also use PSRAM as a program memory, while variables are kept in the hub.

  • pik33pik33 Posts: 2,387
    edited 2023-07-27 14:44

    The retromachine type Basic has to allow to inserting lines into the existing code via their line numbers, so they are listed and executed in the order determined by these line numbers and not in the order in which they are entered.

    That was not very easy to do: I had to implement a 2-way list so every line has pointers to previous and next one.
    Now it partially works. There are 2 procedures that compile the line, one compiling the command (eg. print) and another compiling the assigning to a variable. The second one has to be upgraded by the Copy-Paste method :)

    The list allows positions of lines in the memory to not change. That enables to determine the line address for goto at the interpret/compile time, so the goto itself will go fast while the program runs.

    The way it is done now causes a memory leak when the line is replaced by the new line with the same number. In the future I have to do a memory manager to avoid this, but at this pre-alpha stage let it leaks, I have 8 or 32 MB of it. The screen was taken when the program was running on Eval with 4-bit PSRAM.

  • pik33pik33 Posts: 2,387
    edited 2023-07-27 19:44

    Goto is tricky.

    To do it fast, it should compute the pointer to the line at precompile time and use it later to simply jump.

    Problems are:

    • the destination line for goto may not yet exist
    • the destination line may be overwritten with a new line of the same number, but a different pointer

    To solve this, a temporary table of goto source and destination can be used and while a line that is the destination for already existing goto is entered, a table can be searched to adjust the pointers

    There is however more problems if an argument of goto is an expression that involves variables, like this:

    goto 5*(x+1)+y

    There is no way to know at the precompile time, where that goto will go to.

    I have to introduce 2 functions, slow goto and fast goto. If the argument is a constant, use fast version. If it is an expression, use slow (go through the list, find a line with a desired number)

    And the interpreter as it is now doesn't check if the expression contains variables. If the expression ha no variables in it, it can be converted to a constant while interpreting, and not while running.

  • roglohrogloh Posts: 5,837


    Almost 8MB that's awesome. I remember using BASIC with around 16 to 32kB of RAM when I first started to code and at that time that was considered quite a lot compared to older systems. RAM cost so much back then.

  • pik33pik33 Posts: 2,387
    edited 2023-07-28 06:10

    This is Eval+4bit PSRAM... on EC32 it reports near 32 MB... PSRAM size minus the 576 kB framebuffer that is placed on top.
    However the memory usage is higher in this interpreter as it keeps both the source line to list, and the precompiled line to execute. The precompiled line is not suitable to list, as it is precompiled to RPN/Forth style. so instead of

    print (2+3)*4

    there is

    2 3 add 4 mul print
  • hinvhinv Posts: 1,255

    This is excellent! Forth under the hood can make for a nice way to learn forth if you already know basic as well.

  • Wuerfel_21Wuerfel_21 Posts: 5,105

    If you have the RPN style expression, you could also easily turn that into assembly instructions... :3

  • pik33pik33 Posts: 2,387
    edited 2023-07-29 21:10

    @Wuerfel_21 said:
    If you have the RPN style expression, you could also easily turn that into assembly instructions... :3

    That I hope I can do. Now I am still fighting with basics of Basic. Already added save/load and it started to work, but not yet 100% (it seems I have to save;load variable tables too, to enable further editing of the loaded program)

    The way the code is saved now is not efficient, as a class element, that has 3 longs, is used for everything, a token, a variable, this ends with a lot of unnesessary zeros. The line "plot 10,10" is translated to 10-push-converttoint-10-push-converttoint-plot. That's 5 elements, 15 longs. (10-push is one element, constant token in the compiled stream means "push it")

    If using proper asm, it may look like this (the compiler should recognize a constant and don't call unnecessary convert function - to do even in the current token based precompiler)

                mov pr0,#10
            call pushintvar
            mov pr0,#10
            call pushintvar
            call plot

    5 longs and no unnecessary bloat
    Fast, eficient, but this involves calling to varptr of functions. Current FlexBasic allows this but I was warned to not do it. This is another argument to left varptr of a function in FlexBasic as it is now.

  • Wuerfel_21Wuerfel_21 Posts: 5,105
    edited 2023-07-29 21:51

    Calling a high-level function directly should actually work as long as you stay inside the same object (i.e. don't call instance methods of other objects). Problem is that there isn't really a way to figure out which registers the compiler decided to use, so you need to use prx for everything. You can also use ptrb, which is useful for a stack. Or alternatively, move execution into a separate cog that you control entirely (possibly using a mailbox to communicate with the high-level code. This would also allow some things to happen in parallel.). If you have enough free registers, you can actually eliminate any stack operations. Keep the stack in registers. When going through the expression, since you know how many values each operation consumes/produces, you know which registers it will use and can hardcode those. Super fast.

    You could then have something like this for 2 3 add 4 mul print

    mov 0, #2
mov 1, #3
add 0, 1
mov 1, #4
qmul 0,1
getqx 0
mov pa, #0 ' <- Location of parameters (callee will need to use ALTS to get them)
callpb #PRINT_ID, remote_call_ptr ' <- remote call routone would pass function ID in pb and parameters pointed to by pa into a mailbox. Needs to be called through pointer to be position independent.

    if it was floats you'd of course have a floating point library, with the same ALTS and indirect pointer caveat

    mov 0, #2
mov 1, #3
callpa #0, fadd_ptr
mov 1, #4
callpa #0, fmul_ptr
mov pa, #0
callpb #PRINT_ID, #remote_call

    But in the integer version, you can notice the obvious optimization where you push a constant into a slot that immediately gets reused in an instruction S slot? That is easy to detect...

    mov 0, #2
add 0, #3
qmul 0,#4
getqx 0
mov pa, #0
callpb #PRINT_ID, remote_call_ptr

    Just a random brainworm.

  • pik33pik33 Posts: 2,387
    edited 2023-07-30 17:05

    Today, for saving and loading programs, we use modern media, such as SD cards. So my interpreter can now save and load from SD. But a nostalgic, retromachine Basic has to be able to use a cassette.

    So I found and bought this

    so after it arrives (and I make a contraption to interface it to a P2) I can test if this works:

  • Wuerfel_21Wuerfel_21 Posts: 5,105

    Is that a bootlegged Commodore C2N?

    Could just use a regular tape deck connected to audio jacks, Sinclair style.

  • pik33pik33 Posts: 2,387
    edited 2023-07-30 17:32

    Yes it could. I used a simple sinclair style modulation, although on a higher frequency (2/4 kHz). I have a tape deck, but I don't want to disconnect it from the rest of the stereo system. The retro datassette will be simpler and more convenient. Costed me an equivalent of $25.
    Commodore cassette units as far as I know have only signal shapers inside that converts anything to a TTL level square wave (while Atari type units have a lot more complex stuff to make loading slower).

    I want to play with loading a P2 binary from the tape. Needs some more speed (at least 3 kbps) to fit on the cassette side if full HUB RAM has to be filled. Simply nostalgic useless playing :)

  • pik33pik33 Posts: 2,387
    edited 2023-07-30 21:06

    A riddle/problem to solve

    There is do_run(). It is called when "run" command is entered. It starts from the first line, loads it from PSRAM to HUB and call execute_line(). Execute_line is as simple as it now can be without using direct function addresses and asm calling:

    sub execute_line 

dim cmd as asub

for lineptr_e=0 to lineptr-1
cmd=commands(compiledline(lineptr_e).result_type)
cmd
next lineptr_e
end sub

    It goes through the RPN precompiled line, token by token, calling the proper function for every token

    Now, "do_run" can be called in the interactive mode, but also in the program mode. That should restart the program. As it is now, it does this, but the new instance of do_run() is called from inside old instance of do_run(). And then the next do_run() is called.... until after (about 4000) iterations the program runs out of stack.

    I have to solve this. A first idea is to set a variable that will tell the do_run() that it is not the first instance, so instead starting to execute_line() it should find a first line and goto there, then immediately exit, so the proper (first instance) of do_run() will do its job from the start. Runptr variable has to be made global, so it may be changed outside do_run().

    Another solution is to compile something like do_run2() if the precompiler recognizes non-interactive mode (and it can do it). Still run pointer variable has to be global. Commands like goto, gosub or return will change this outside do_run()

    Goto waits for implementing. For..next and do..loop are, internally, goto.

  • roglohrogloh Posts: 5,837
    edited 2023-07-30 23:34

    @pik33 said:
    I want to play with loading a P2 binary from the tape. Needs some more speed (at least 3 kbps) to fit on the cassette side if full HUB RAM has to be filled. Simply nostalgic useless playing :)

    Oh the joys of loading a 30 minute program!

    I once remember doing a 300 bps BBS download of a 200kB text adventure game direct to a floppy disk. Took a couple of hours! We even watched a VHS movie during this time as I recall. We were worried it would fail or timeout etc but it didn't and we got the game in the end. This connection used an old acoustic coupler (which was ancient even then).

  • pik33pik33 Posts: 2,387
    edited 2023-07-31 13:48

    First run with rnd and (unfinished) goto (that can only go to the line that is already in the program, the rest of cases are yet to implement) . Running on Eval+4bit PSRAM.

    Edit: i cannot play these video clips I placed here on the forum, except "play in YouTube". Do you have the same problem with these clips? What may I do wrong?

  • MicksterMickster Posts: 2,698

    Plays perfectly, right here in the thread :+1:

    This is looking fantastic :)

    Will this BASIC be limited to retro-games or will it be able to drive I/O and trigger compiled code in other cogs, etc.?

    Craig

  • pik33pik33 Posts: 2,387
    edited 2023-07-31 16:58

    Of course it will drive i/o and load a compiled code to cogs, although I think the code to load to the cog will have to be compiled with something else (Flexprop) first and placed on SD and/or flash. Then a command like cogload 2,"vga.drv" or something like this will upload it.

    The precompiled code is placed in, and running from PSRAM :)

    I have several cogs already used: PSRAM, USB, video, audio, main - that's 5 of them, but 3 are still free

    The basics: syntax analyzer, expression evaluator, program control (if/for/do) are most difficult to do, adding a new command that do something is easy. I can add basic pin driving even now, but I still have no for and no if.

  • ke4pjwke4pjw Posts: 1,169

    This is great! I am going to have to play with this!

  • pik33pik33 Posts: 2,387
    edited 2023-07-31 18:23

    @ke4pjw said:
    This is great! I am going to have to play with this!

    The repository link is in the first post
    Needs P2-EC32 with HDMI on P0, AV on P8 and USB on P16. The current version in the repository is basic13.bas, needs a bleeding edge Flexprop to compile. Set #define PSRAM16, comment out #define PSRAM4.
    If #define PSRAM4, then it runs on P2-Eval with 4-bit PSRAM connected as DATABUS = 48, CLK_PIN = 55, CE_PIN = 54

    Still very early pre-alpha without the most important language features. Recognized keywords in 0.13 are: cls, new, plot, draw, print, fcircle,color,list, run,goto (limited), csave, save, load, rnd. Also arithmetic, div, mod, shl, shr and ^

    Variables are integer (without suffix), uint (var%), float-single (var!) and string (var$)

    Goto will only run if the target line for goto is already in the program when the goto line was entered. Lines are ordered by their number, don't need to enter sequentially.

    ctrl-c or f12 stops running the program

    Many commands separated with : work only in the immediate line.

  • pik33pik33 Posts: 2,387
    edited 2023-07-31 21:14

    At the end of the day I added waitms and pinwrite to the command list so blinking the led is now possible:

    10 pinwrite 38,0
20 waitms 500
30 pinwrite 38,1
40 waitms 500
50 goto 10

    ... also added waitvbl....

  • MicksterMickster Posts: 2,698

    Man, I can't wait for next week when we'll have functions, labels, etc.....Just kidding :D

    Craig

Sign In or Register to comment.