Shop OBEX P1 Docs P2 Docs Learn Events
Catalina - ANSI C for the Propeller 1 & 2 - Page 3 — Parallax Forums

Catalina - ANSI C for the Propeller 1 & 2

1356711

Comments

  • RossHRossH Posts: 5,336
    edited 2022-11-13 23:23

    New version of yymodem - various bug fixes and improvements, and now tested and working on Propeller 1 and Propeller 2 (but see the README.TXT file about Propeller 1 limitations)

    YMODEM FOR CATALINA/CATALYST
    ============================
    
    This folder contains an implementation of the YMODEM file transfer protocol 
    for Catalina and Catalyst. It is derived from a free public domain version 
    of YMODEM written by Fredrik Hederstierna.
    
    YMODEM can be used to send and receive both text and binary files using the
    serial connection from the Propeller to a host PC, or vice-versa. Transfers 
    are done in 128 (YModem) or 1024 byte (YModem-1K) packets, and each packet
    is protected by a 16 bit CRC, and retried if it fails, so transfers are very 
    reliable and transmission errors are extremely rare.
    
    The impetus for this program was that I was damaging my Propeller boards and
    also my PC by inserting and removing SD Cards so often - typically because I 
    had to update the software on them. Both the Host and the Propeller SD Card 
    sockets can eventually be damaged or become unreliable, so I wanted a simple 
    file transfer mechanism that would help me reduce the need for doing this - 
    and YMODEM (although it dates from the days of modems and bulletin boards) 
    does the job nicely!
    
    To execute on the Propeller, the program must be compiled to use a serial 
    plugin and library (e.g. using the tty256 serial plugin on the Propeller 1,
    or the 2 port serial plugin on the Propeller 2) and cannot also use a serial
    HMI option (if this is the default, specify -C NO_HMI). If the plugin supports 
    multiple ports then by default the program uses serial port 0.
    
    There is only one YMODEM program file (fymodem.c) plus a serial support file 
    (rs232.c) which allows it to be compiled using gcc, but the program can be 
    compiled as either a sender or a receiver by defining the symbols YMODEM_SEND
    or YMODEM_RECEIVE in the compile commmand. For example, the following commands
    might be used to compile a YMODEM receive and YMODEM send program:
    
       catalina -p2 fymodem.c -D YMODEM_RECEIVE -C NO_HMI -lcx -lserial2 -o receive
       catalina -p2 fymodem.c -D YMODEM_SEND -C NO_HMI -lcx -lserial2 -o send
    
    The programs can be compiled for the Host PC using MinGW under Windows, or 
    gcc under Linux. Note that you need to include the rs232.c serial interface
    code. For example:
    
       gcc fymodem.c rs232.c -D YMODEM_RECEIVE -o receive
       gcc fymodem.c rs232.c -D YMODEM_SEND -o send
    
    There is a build_all script provided that does all this for you. For example,
    you can just say:
    
       build_all P2_EDGE
    or
       build_all C3 FLASH CACHED_1K
    
    This will build two PC executables:
    
       send.exe                 <-- or just 'send' for Linux
       receive.exe              <-- or just 'receive' for Linux
    
    which will build two Propeller executables:
    
       send.bin                 <-- or 'send.binary' for the Propeller 1
       receive.bin              <-- or 'receive.binary' for the Propeller 1
    
    The format of the commands under Windows or Linux is:
    
      send    [-h] [-v] [-d] [-x] [-p PORT] [-b BAUD] [-t TIMEOUT] [-s DELAY] file
      receive [-h] [-v] [-d] [-x] [-p PORT] [-b BAUD] [-t TIMEOUT] [file]
    
    The format of the commands on the Propeller is similar, except for the baud
    rate option. On the Propeller the baud rate cannot be specified on the command
    line - it must be pre-configured in the platform configuration files (on the 
    Propeller 1, this is the file Extras.spin, on the Propeller 2 it is in the 
    platform.inc file - e.g. P2_EDGE.inc). Also note that the Makefile provided 
    adds -C NO_HMI when compiling the program for the Propeller, which makes the 
    -h, -v and -d options a bit useless. This is because ymodem generally uses 
    the same port as a serial HMI (which is often the default HMI), so the HMI 
    cannot be used at the same time. However, if your platform can use a
    non-serial HMI option (such as VGA) you can edit the Makefile to remove 
    the -C NO_HMI option and add your preferred HMI option (e.g. -C VGA).
    
    The default baud rate is 230400 baud. This is fine on the Propeller 2 but is
    probably too fast on the Propeller 1, where a baud rate of 115200 is generally
    the default. Also, the Propeller 1's smaller serial buffer sizes and slower 
    serial plugins means the -s option must usually be specified for the sender 
    (the -s option applies ONLY to the sender). This option does two things:
    
       1. Tells the sender to only send 128 byte blocks, not 1024 byte blocks -
          i.e. use YModem and not YModem-1K.
    
       2. Adds a delay of the specified number of milliseconds between each
          character sent - often -s0 will work, but if not try -s5, -s10 etc. 
    
    Currently, only one file can be transferred at a time. This may change in
    a future release.
    
    The send command must include the name of the file to be sent, but in the 
    receive command this is optional. It will be used if specified, but if not 
    then the received file will be given the same name as the sent file.
    
    You can terminate an executing receive or send program by entering two
    successive [CTRL-X] characters.
    
    From Windows you can use the stand-alone ymodem programs on both the Propeller
    and the Host PC, or you can use them only on the Propeller and use payload 
    (which now supports YMODEM) on the PC. Start payload as normal in interactive
    mode, even if the Propeller uses a non-serial HMI option. Execute the ymodem 
    send or receive program on the Propeller first, and then within payload, press
    CTRL-A and you will see a menu appear. Press Y for YModem and then either S 
    for a YModem Send, T for a YModem-1K Send or R for a YModem Receive. You must
    enter a file name when Sending, but this is optional when Receiving.
    
    A terminal emulator program that supports YMODEM may also be used, such as 
    Tera Term or ExtraPuTTY (on Windows) or minicom or picocom (on Linux). 
    However, note that the Propeller 1 can only support YModem transfers, not 
    YModem-1K, and some terminal emulators (e.g. Tera Term and ExtraPuTTY) assume 
    they can always use YModem-1K.
    
    When the YMODEM program is started and you are using a terminal emulator, 
    you may see 'C' characters being printed repeatedly in the terminal window - 
    this is normal, and is how the YMODEM send and receive programs synchronize 
    with each other. 
    
    IMPORTANT NOTE WHEN USING WINDOWS:
    ==================================
    
    Windows recently introduced a new console mode that is very buggy. Programs 
    such as payload that use console mode can fail unexpectedly if the new console 
    mode is used. This requires more investigation, but until Microsoft fixes the 
    issues, it is recommended that the "legacy console mode" be used with payload
    and other Catalina programs. To ensure this, select Properties from the system 
    menu of any console window and in the Options tab, ensure that the option
    to "Use legacy console" is selected.
    
    

    EDIT: Updated to match the version included in Catalina 5.5

  • RossHRossH Posts: 5,336
    edited 2022-11-13 23:25

    Deleted - this YMODEM issue has now been fixed in the above post, and also in Catalina 5.5

  • how about using the sd/host filesystem @ersmith is using in Flex?

    It is a nice implementation, it is neat integrated to use from C and Basic, just the Spin support is a bit lacking.

    Having that in Catalina/Catalyst would be fantastic.

    Mike

  • RossHRossH Posts: 5,336

    @msrobots said:
    how about using the sd/host filesystem @ersmith is using in Flex?

    It is a nice implementation, it is neat integrated to use from C and Basic, just the Spin support is a bit lacking.

    Having that in Catalina/Catalyst would be fantastic.

    Mike

    I'm not familiar with Flex. Does it support transferring files between the host PC and the Propeller SD Card? That's what I needed, and now that I have yymodem working reliably at 230400 baud, it's easy to use when using Catalyst via a terminal emulator, since many of them already have support for it built in. Catalina's own terminal emulator (payload) currently does not, but it will be easy to add, so I will probably do that for the next release.

    Ross.

  • The FAT filesystem in FlexProp is elm-chan.org/fsw/ff/00index_e.html one, which is pretty widely used in the microcontroller world. For the host PC file system I used a Plan 9 protocol server on the PC side and wrote my own code on the Prop side; it's a very clean protocol and easy to implement (not surprising considering where it came from!). The PC side code is in both loadp2 and my version of proploader, so it works on both P1 and P2. To copy files from host to SD I just run a shell that mounts both the host FS and FAT FS, and then use "copy /host/file.txt /sd/file.txt".

    Having said all that, YMODEM is a fine solution for file copies as well, it just serves a slightly different function (although you may well be able to put a file system interface on top of the YMODEM protocol?)

  • The nice thing about the Plan 9 protocol is that in principle it gives a standard way to talk to any file or device, so we could (theoretically) expose I2C, SPI, and any kind of disk drive (including FAT) via a 9P interface, which would make it easy for code to interoperate between Catalina, FlexProp, Spin2, or whatever. 9P does impose some overhead compared to raw device accesses though, so it isn't widely used in the embedded space.

  • RossHRossH Posts: 5,336

    Hi @ersmith

    Sounds useful. Does the Plan 9 system mount the PC disk as a file system on the Propeller, or the Propeller SD Card as a file system on the PC? Or perhaps it can do either? Could it also be used to mount one Propeller SD Card file system on another Propeller?

    I did something similar using my proxy SD card driver on the P1 - i.e. you can mount an SD card from one Propeller as the file system on another, using a simple 2-pin serial connection. This was used in the various multi-Propeller systems such Cluso's and Bill Henning's boards. There was a craze for these multi-Propeller P1 systems before the P2 came along, when people were desperate for more pins, more RAM and more speed! But I have not bothered porting it to the P2 yet because there are no multi-Propeller 2 systems yet (AFAIK!). But give it time, and I'm sure some new lunatic will come along ... :)

    The Plan 9 software might be better than mine if it is written in C - mine was written in P1 assembler, so it will need porting and re-testing for the P2. But on the other hand mine also supports other proxy devices such as the mouse, screen and keyboard. I'll look into the Plan 9 stuff when I get some time. For the moment, yymodem solves the problem I needed solved, and already had software support on the PC side. The current version now works at full speed with all the terminal emulators I have tried so far - e.g. minicom, PuTTY, Tera Term.

    Ross.

  • RossHRossH Posts: 5,336
    edited 2022-11-08 04:04

    All,

    I've updated the post above (here) with a new version of ymodem. The version has various bug fixes and improvements, and has now been now tested and is working on both the Propeller 1 and the Propeller 2.

    This version will also be included in the next release of Catalina.

    Ross.

  • RossHRossH Posts: 5,336

    All,

    YModem support has now been added to the payload loader. This is convenient on Windows, but it is required for Linux - on Windows you can easily use the stand-alone YModem send and receive programs, but Linux toggles DTR (which resets the Propeller) whenever you close and re-open a serial port, so you have to use a terminal emulator that supports YModem so that it does not have do that before the YModem transfer can be started. This version of payload will be included in the next release, but it is not worth doing a release just for this.

    I have never found a general solution for this Linux/DTR issue - if anyone knows of one please let me know!

    Ross.

  • ElectrodudeElectrodude Posts: 1,614
    edited 2022-11-10 04:22

    @RossH said:
    I have never found a general solution for this Linux/DTR issue - if anyone knows of one please let me know!

    Ross.

    Linux normally does this because serial ports were traditionally all connected to modems, for which this is the right thing to do. You can turn it off with stty -hup -F /dev/whatever, but turning it off causes DTR to transition, resetting the P2, so make sure you run that command as soon as you plug in the device, before you try programming it.

    It should be possible to write a udev rule to make stty -hup get called automatically whenever you plug in a serial device.

    I wouldn't be surprised if some of the more "helpful" GUI terminal emulators turn HUP back on for you, but I just confirmed that the one I usually use, picocom, leaves the setting alone (unless you explicitly ask it to change it for you).

    EDIT: It's probably also a good idea to add -ixon -ixoff to stty's options to make the terminal not interpret Ctrl-S and Ctrl-Q as flow control characters.

  • RossHRossH Posts: 5,336

    @Electrodude said:

    @RossH said:
    I have never found a general solution for this Linux/DTR issue - if anyone knows of one please let me know!

    Ross.

    Linux normally does this because serial ports were traditionally all connected to modems, for which this is the right thing to do. You can turn it off with stty -hup -F /dev/whatever, but turning it off causes DTR to transition, resetting the P2, so make sure you run that command as soon as you plug in the device, before you try programming it.

    It should be possible to write a udev rule to make stty -hup get called automatically whenever you plug in a serial device.

    I wouldn't be surprised if some of the more "helpful" GUI terminal emulators turn HUP back on for you, but I just confirmed that the one I usually use, picocom, leaves the setting alone (unless you explicitly ask it to change it for you).

    EDIT: It's probably also a good idea to add -ixon -ixoff to stty's options to make the terminal not interpret Ctrl-S and Ctrl-Q as flow control characters.

    Thanks, but I've tried all the stty and ioctl tricks that are reported by various people to solve this problem and none of them make any difference for me. At least not on my version of Linux (Ubuntu 18). I don't think there is a general solution to this problem. I know some people have resorted to writing their own kernel drivers in desperation, but that's difficult and also version-specific.

    Ross.

  • @RossH said:

    @Electrodude said:

    @RossH said:
    I have never found a general solution for this Linux/DTR issue - if anyone knows of one please let me know!

    Ross.

    Linux normally does this because serial ports were traditionally all connected to modems, for which this is the right thing to do. You can turn it off with stty -hup -F /dev/whatever, but turning it off causes DTR to transition, resetting the P2, so make sure you run that command as soon as you plug in the device, before you try programming it.

    It should be possible to write a udev rule to make stty -hup get called automatically whenever you plug in a serial device.

    I wouldn't be surprised if some of the more "helpful" GUI terminal emulators turn HUP back on for you, but I just confirmed that the one I usually use, picocom, leaves the setting alone (unless you explicitly ask it to change it for you).

    EDIT: It's probably also a good idea to add -ixon -ixoff to stty's options to make the terminal not interpret Ctrl-S and Ctrl-Q as flow control characters.

    Thanks, but I've tried all the stty and ioctl tricks that are reported by various people to solve this problem and none of them make any difference for me. At least not on my version of Linux (Ubuntu 18). I don't think there is a general solution to this problem. I know some people have resorted to writing their own kernel drivers in desperation, but that's difficult and also version-specific.

    Ross.

    That's very strange. That's the standard way of doing it - it should work. What USB-serial converter are you using? What terminal emulator have you been using on Linux?

  • RossHRossH Posts: 5,336
    edited 2022-11-10 22:00

    @Electrodude said:
    That's very strange. That's the standard way of doing it - it should work. What USB-serial converter are you using? What terminal emulator have you been using on Linux?

    I am just using a Prop Plug. I generally use my own terminal emulator (payload) and stty hup or stty -hup etc makes no difference. But I have just tried picocom and interestingly it can open the port without resetting the Prop. Again, stty hup or stty -hup makes no difference to it, but clearly it is possible. There must be a combination of ioctl options I have not tried which overrides the port settings.

    I'll take a look at the picocom source code to see how it does it.

    Thanks.

  • RossHRossH Posts: 5,336

    @RossH said:
    I'll take a look at the picocom source code to see how it does it.

    Thanks.

    Ha! Comparing the picocom code to the code I use, everything looked fine - except that the code I use contains some ADDITIONAL code specifically designed to avoid toggling DTR! When I take that code out, payload behaves the same way as picocom.

    It is possible I added that code in the early days of Catalina when I was using a different flavor of Linux. Or perhaps the Linux serial driver changed at some point. It's not worth going back to find out. I will just add a suitable note, make it configurable, and include the change in the next release.

    Ross.

  • @RossH said:

    @RossH said:
    I'll take a look at the picocom source code to see how it does it.

    Thanks.

    Ha! Comparing the picocom code to the code I use, everything looked fine - except that the code I use contains some ADDITIONAL code specifically designed to avoid toggling DTR! When I take that code out, payload behaves the same way as picocom.

    It is possible I added that code in the early days of Catalina when I was using a different flavor of Linux. Or perhaps the Linux serial driver changed at some point. It's not worth going back to find out. I will just add a suitable note, make it configurable, and include the change in the next release.

    Ross.

    Are you sure you ever actually needed that code to begin with, or is it possible you just put it in because, say, you thought it'd fix a problem that was really somewhere else? Why don't you just tear it out, with no configuration option, until somebody complains, and only then consider putting some configuration option in. Less code means less room for future bugs...

  • RossHRossH Posts: 5,336

    @Electrodude said:
    Are you sure you ever actually needed that code to begin with, or is it possible you just put it in because, say, you thought it'd fix a problem that was really somewhere else? Why don't you just tear it out, with no configuration option, until somebody complains, and only then consider putting some configuration option in. Less code means less room for future bugs...

    I have wrestled with this problem several times over the years, and so has Linux (evident from looking at various old Linux bug reports about DTR/DSR). I just went back through old versions of Catalina, and it looks like I did indeed write this particular code. So I think it was probably both necessary and worked at the time - but I would probably have been using Red Hat/Fedora at that point, not Debian/Ubuntu.

    In any case I have effectively removed it by putting #if ... #endif around it and also putting a note in the source file so it will be obvious in future for anyone who looks that enabling it might offer a solution.

  • RossHRossH Posts: 5,336
    edited 2022-11-13 23:38

    I have just posted Catalina 5.5 to SourceForge here.

    This is a full release, primarily to update Catalyst and Payload to add YMODEM support, and also fix the Payload DTR issue on Linux. Plus a few minor things.

    Here is the relevant extract of the README.TXT file:

    RELEASE 5.5
    
    New Functionality
    -----------------
    
    1. A version of the YModem serial file transfer protocol has been added to 
       Catalyst. When Catalyst is built, there will be two more Propeller programs
       added to the demos\catalyst\bin directory:
    
          send.bin - send a file from the Propeller to the host.
          receive.bin - receive a file on the Propeller sent from the host.
    
       Also, in the demos\catalyst\fymodem directory, there will be two
       corresponding host binaries (on Windows these will be .exe files):
    
          send - send a file from the host to the Propeller
          receive - receive a file on the host sent from the Propeller 
    
       See the README.TXT file in demos\catalyst\fymodem for more details.
    
    2. A version of the YModem serial protocol has been integrated into payload's 
       interactive mode. This can be used for file transfers even if the Propeller
       does not use a serial HMI option. 
    
    3. Payload now supports configuring the line termination mode from within
       interactive mode as well as on the command line (i.e. using the -q command
       line option).
    
    4. Payload now has menus. Pressing the "attention" key (CTRL-A by default) 
       in payload's interactive mode now brings up a menu from which you can 
       select either Terminal Configuration, YModem Transfer, or to Exit from 
       payload (as an alternative to pressing CTRL-D twice).
    
    5. The ex_leds_1.c and ex_leds_2.c example programs in demos\examples now 
       support the P2_EDGE board.
    
    6. Payload now responds correctly to the DEC VT100 Cursor Position Requests 
       (CUP) and responds to a Device Status Report (DSR) specifying parameter 6 
       with a Cursor Position Report (CPR). Taken together, this allows programs
       expecting a VT100 compatible terminal to determine the actual terminal 
       window size, rather than simply assuming it will always be 80x24.
    
    7. The Catalyst vi program now uses the DEC VT100 CUP and DSR requests to 
       elicit a CPR, which allows it to determine the actual window size. 
       This means that vi is no longer limited to 80x24 windows when a serial
       HMI option is used. In particular, when payload is used, the actual window 
       size will now be reported correctly to vi on startup, and vi will use it
       rather than defaulting to 80x24 whenever a serial HMI is used. The default
       VT100 window size (80x24) will still be used if VT100 is specified in the
       build and the terminal emulator cannot respond to CUP and DSR requests.
    
       For example, to use vi in a terminal window of 50 lines and 120 columns, 
       and assuming you are using a serial HMI and specified the VT100 option
       when vi was built, then you might start payload as follows (assuming N is 
       the comms port connected to your Propeller):
    
          payload -i -g120,50 -b230400 -pN
    
       Then within payload just start vi as normal and it will use the correct
       window size - e.g:
    
          vi catalyst.txt 
    
    Other Changes
    -------------
    
    1. The files s2_serial.h, s4_serial.h and s8_serial.h were not correctly
       defining the symbols s2_getc, s4_getc and s8_getc respectively. These 
       names were only intended to be used to emulate the original Spin names - 
       the actual underlying C functions (i.e. s2_rxcheck, s4_rxcheck and
       s8_rxcheck respectively) were all fine.
    
    2. The Catalina Optimizer may have reported undefined symbols when optimizing
       programs that used the serial2 or serial8 libraries.
    
    3. The header files for the tty, tty256, serial2, serial4 and serial8
       libraries now check and issue a message (using #error) if the libraries
       they represent are not supported on the Propeller type for which the 
       program is being compiled:
    
       - The serial2 and serial8 libraries are only supported on the Propeller 2.
    
       - The tty, tty256 and serial4 libraries are only supported on the Propeller
         1 (but note this has nothing to do with the TTY HMI option, which applies
         to both the Propeller 1 and Propeller 2 - the TTY HMI option does not 
         use these libraries).
    
    4. The rs232 interface module (rs232.h, rs232.c) used by payload and fymodem
       included code intended to prevent DTR being toggled when opening a serial
       port. This code no longer works correctly on Debian Linux and has been 
       disabled. If it is required on other platforms, it can be re-enabled by 
       setting rs232_DTR_FIX to 1 in the file rs232.h. Note that copies of this 
       file exist in both the folder source/catalina as well as the folder 
       demos/catalyst/fymodem folder, and will need to be modified in both places.
    
    5. The help text for payload's -q command line option had the values for
       enabling CR_TO_LF and LF_TO_CR translation the wrong way around.
    
    

    There is not much new stuff in this release, but I needed to get it out because I won't as have much time for Catalina for the next few months - it is coming into our "busy" season here. Also, I think this may well be the last "monolithic" release. The overheads of doing this are now much too high. I will think about the best way to break it down into more manageable chunks in future - at the very least, I will probably have separate releases for Catalina and Catalyst.

    Ross.

  • RossHRossH Posts: 5,336

    I have just posted Catalina 5.5.1 to SourceForge here.

    This is a patch release, and there are no changes to Catalina itself. The purpose of the release is to introduce a new Windows-based terminal emulator, and to update payload to use it.

    Payload's built-in interactive mode is still available, but I got tired of fighting with Microsoft's console mode, so I decided it was best to stop using it. I dusted off a terminal emulator I developed a while ago - for essentially the same reason - and added a few Propeller-friendly specifics (basically, the ability to control DTR, support for additional baud rates common on the Propeller, and YModem support).

    Note that the new payload option allows any terminal emulator to be used. Examples of using it to invoke PuTTY (on Windows) or minicom (on Linux) are included. Also note that the new terminal emulator is completely separate from Catalina, and can also be used stand-alone (it is also available here.

    Here is the relevant extract of the README.TXT file:

    RELEASE 5.5.1
    
    New Functionality
    -----------------
    
    1. The Windows version of Catalina now includes a new serial comms program
       with a full-function terminal emulator, which can be used as a replacement
       for payload's very simple interactive mode by specifying "-I terminal" 
       instead of just "-i" on the payload command line - see below for more 
       details). 
    
       The binary for the program (comms.exe) is in Catalina's bin folder,
       and the source is in source\comms, but note that the source is not built 
       by default if Catalina is rebuilt, since it requires GNAT (the GNU Ada 
       Translator) and the GNAT Studio development environment to be installed. 
       See the BUILD.TXT file in source\comms for details on how to rebuild the 
       comms program from source.
    
    2. Payload now allows an external terminal emulator program to be executed in
       place of the simple internal one.  The -i option still enables the internal
       emulator, but a new -I option (i.e. upper case) calls an external terminal
       emulator program. The -I option accepts one parameter which specifies the 
       command to be used. The port name (not the port number) and baud rate will
       be passed to this program on startup. 
    
       For example, if the payload command executed was:
    
          payload program.bin -p11 -b230400 -I vt100
    
       Then after loading the program.bin file, the command executed (on Windows)
       would be:
    
          vt100 COM11 230400
    
       On Linux, the command would be something like:
    
          vt100 /dev/ttyUSB0 230400
    
       A suitably named script (i.e. on Windows a batch file called vt100.bat, 
       or on Linux a shell script called vt100) can be used to specify any other 
       required parameters. 
    
       The following Windows scripts are provided (each one is a batch file):
    
          pc                  <-- start comms, specifying PC emulation
          vt52                <-- start comms, specifying VT52 emulation
          vt100               <-- start comms, specifying VT100 emulation
          vt101               <-- start comms, specifying VT101 emulation
          vt102               <-- start comms, specifying VT102 emulation
          vt220               <-- start comms, specifying VT220 emulation
          vt320               <-- start comms, specifying VT320 emulation
          vt420               <-- start comms, specifying VT420 emulation
          vt100_putty         <-- start PuTTY, specifying VT100 emulation
    
       The following Linux scripts are provided (each one is a shell script):
    
          ansi               <-- start minicom, specifying ANSI emulation
          vt100              <-- start minicom, specifying VT100 emulation
          vt102              <-- start minicom, specifying VT102 emulation
    
       Note that these scripts can be used independently of payload. They all 
       accept two parameters - the com port to use, and the baud rate. 
       For example:
    
          vt100 COM11 230400
          vt102 /dev/ttyUSB1 115200
    
       Note that the comms program executable is provided as part of the 
       Windows version of Catalina, but PuTTY and minicom executable will have 
       to be installed separately. Ensure the appropriate executable is in the 
       PATH.
    
    Other Changes
    -------------
    
    1. Payload now interprets either ESC [ ? 6 n or ESC [ 6 n as a DEC DSR
       request and responds with a DEC CUP. Strictly speaking, ESC [ ? 6 n 
       is not a correct DEC DSR command, and would not be recognized by 
       other terminal emulators, so the Catalyst xvi program now uses the
       more correct ESC [ 6 n
    
    

    Ross.

  • RossHRossH Posts: 5,336
    edited 2022-11-30 06:37

    A change to the clock and SD Card plugin code (made in release 5.4) means that the Catalyst Super Star Trek demo now needs to be explicitly built with the -C CLOCK option.

    I will fix this in the next release. but in the meantime, just edit the file demos\catalyst\sst\MAKEFILE.CAT and find the following line:

    CFLAGS += -lcx -lma -C NO_MOUSE

    and add -C CLOCK so that it reads as follows:

    CFLAGS += -lcx -lma -C NO_MOUSE -C CLOCK

    Then rebuild.

    EDIT: This has been fixed in patch release 5.5.2.

    Ross.

  • RossHRossH Posts: 5,336

    Another quick patch - release 5.5.2, available here.

    Again, no changes to Catalina itself. Mainly just an update for the vi text editor.

    Here is the relevant portion of README.TXT:

    RELEASE 5.5.2
    
    New Functionality
    -----------------
    
    1. The Catalyst vi editor (xvi) demo program has been updated to version 2.51,
       which is the latest version and contains some new functionality and some
       bug fixes. See the documents in demos\catalyst\xvi-2.51\doc for details.
    
    2. The Catalyst vi editor now processes the keypad keys PgUp, PgDown, Home, 
       End, Ins & Del appropriately.
    
    2. The Catalyst vi editor can now be compiled to use colour when the VT100 HMI 
       option is used, since colour is supported by the new external terminal 
       emulator (comms.exe). Just define the Catalina symbol USE_COLOR in addition
       to VT100, either on the Catalina command line or by adding it as a parameter
       to the build_all command. For example:
    
          cd demos/catalyst/xvi-2.51
          build_all P2_EDGE CR_ON_LF VT100 USE_COLOR 
    
       Note: Only foreground colours can be set within vi itself - the current
       background colour (typically the background colour in use before vi was 
       started) will be used. The default colours are that text will be grey, 
       the status line will be green (normal files) or red (read-only files). 
       Try the vi command ":help" to see an example. 
    
       You can manually set the colours (for the text, status line, or read-only
       status line) in vi as follows:
    
          :set colour=n
          :set statuscolour=n
          :set roscolour=n
    
       where n is:
    
          0  : black
          1  : red
          2  : green
          3  : yellow
          4  : blue
          5  : magenta
          6  : cyan
          7  : grey
          9  : default foreground colour
          10 : bold and bright black (dark grey)
          11 : bold and bright red
          12 : bold and bright green
          13 : bold and bright yellow
          14 : bold and bright blue
          15 : bold and bright magenta
          16 : bold and bright cyan
          17 : bold and bright grey (white)
    
       Note that when vi is compiled to use colour, the default vi colours will
       override the current foreground colour on startup. To get it back again, 
       use the vi command ":set colour=9"
    
       Any value other than those specified will set the colour to the default
       foreground colour (typically, the foreground colour before vi was started). 
       If you are using comms.exe then you can also select the default colours 
       using the Format menu (remember to select the Format->Set All To Colours 
       menu item after choosing your new colours, or the changes will apply only 
       to new characters, not existing ones).
    
       Note that you can also have other effects, such as inverse video -
       (i.e. black text on white bg). To do this, start vi and then set the 
       foreground colour to black (i.e. ":set colour=0") - note that any text 
       on the screen may temporarily disappear. Then use the Format->Bg Color 
       menu item to select a white background, then use the Format->Set All 
       to Colour menu item to refresh the colours of the characters already
       on display. You can choose to have any background colour you want.
    
    Other Changes
    -------------
    
    1. A change to the clock and SD Card plugin code (made in release 5.4) means
       that the Catalyst Super Star Trek demo now needs to be explicitly built 
       with the -C CLOCK option. The Makefile has been modified accordingly.
    
    2. When the external terminal emulator's YModem dialog box is open, it is now 
       always on top. Otherwise, it could get hidden by other windows and it was 
       not obvious why the main comms window would not respond.
    
    3. The external terminal emulator's YModem dialog box "Start" button is no 
       longer styled as if it was a default button, since it was not. There is no 
       default button for this dialog box.
    
    4. The new external terminal emulator now locks the screen and view. This 
       makes it easier to set the initial screen size (can now be done just using
       the /ScreenSize option, instead of also requiring the /ViewSize option).
       This is more appropriate when emulating a physical terminal, where the 
       screen size would normally be the same as the view size. They can be
       unlocked again (if required) via the Options->Advanced menu.
    
    5. The scripts that call the external terminal emulator (vt100.bat etc) now 
       add /Wrap to enable wrap mode on startup. This option can also be changed 
       via the Options->Advanced menu. Setting it on by default makes the external
       terminal emulator behave more like the internal terminal emulator.
    
    6. The payload internal terminal emulator now limits the number of restart 
       retries on ymodem_receive, to avoid locking up permanently in case the 
       other end has not been started.
    
    7. The file P2_EVAL.ZIP has been renamed P2_DEMO.ZIP, and the README_P2.TXT
       file has been updated to indicate that this demo version of Catalyst
       should work on the P2 EVAL, P2 EDGE or any other P2 board with a similar
       SD Card configuration.
    
    
  • RossHRossH Posts: 5,336

    Just a quick note about Catalyst on the Propeller 2 ...

    I discovered recently that the P2 EDGE and P2 EVAL boards (which are the only P2 boards I have) both boot programs much more reliably from FLASH than from the SD Card. Not sure why this is, but to take advantage of it Catalyst can be booted from FLASH just as easily as it can from the SD Card.

    Compile Catalyst as usual and copy the results from the demos\catalyst\bin directory to an SD Card as usual, but then set the FLASH dip switch ON (and the P59^ and P59v dip switches OFF ) and use flash_payload to program Catalyst into FLASH. For example, if you are using the serial interface and also want to use the new vt100 terminal emulator, you would use commands like:

    cd demos\catalyst\bin
    flash_payload catalyst.bin -I vt100
    

    Catalyst and all its demo programs should function as usual. Turn the FLASH dip switch OFF to revert to booting from the SD Card.

    A note to this effect will be included in the next release.

    Ross.

  • RossHRossH Posts: 5,336
    edited 2022-12-17 10:49

    Okay ... I said I wouldn't do it ... I said it was a mistake on the Propeller 1 ... but then I came down with Covid-19 and couldn't do much of anything else for the past week, so ...

    I now have Catalina executing C programs using PSRAM as XMM on the Propeller 2. This means Catalina C programs can now be up to 16MB.

    I currently only support the PSRAM on the P2 EDGE, but I am using the driver written by @rogloh so while I don't have one of the HyperRAM boards, I believe it would also work on that with just a few tweaks.

    The P2 EDGE has 32MB of PSRAM, but there are some internal design decisions from the Propeller 1 days (when anything bigger than 32kb was a big deal) that limit programs to 16MB - but you can use the other 16MB as extra storage and access it from C.

    My test suite executes correctly, and I just tried a simple benchmark program. The early results are quite promising:

    EXECUTION MODE     TIME    CODE SIZE 
                       (ms)     (bytes)
    ================   ====    =========
    NATIVE (HUBEXEC)    705      20584
    LMM                1308      23736
    CMM (COMPACT)      1551      12612
    XMM (PSRAM)        3009      26960
    

    So executing the code using XMM mode (i.e. from PSRAM) is about 4 times slower than executing the same code using NATIVE mode (aka P2 HUBEXEC mode), or about 2 times slower than the LMM or COMPACT modes (which are identical to the same modes on the Propeller 1). You can also have different cogs executing the different modes simultaneously, so you can have those parts of the program that need speed executing in NATIVE mode and the rest executing in XMM mode.

    There is still a heap more to do and test and tidy up, but I hope to get at least a beta release out before Christmas.

    Merry Christmas!

    Ross.

  • roglohrogloh Posts: 5,122
    edited 2022-12-17 07:22

    Hell froze over! LOL. You've seen the light! Well done. :smile: Merry Xmas.

    It's great that you used my driver as other COGs can still share the memory with QoS guarantees, like video drivers, etc - the executed code will slow down a bit if/when they do, but in many cases that's still okay. And as you say there are other memory types that would be supportable. I did HyperRAM and some smallish 2MB SRAM and still plan to do one for (Q)SPI FLASH which could give you some XIP (probably rather slow unless cached).

  • RossHRossH Posts: 5,336

    @rogloh said:
    Hell froze over! LOL. You've seen the light! Well done. :smile: Merry Xmas.

    It's great that you used my driver as other COGs can still share the memory with QoS guarantees, like video drivers, etc

    Yes, the fact that I can use your driver from multiple cogs was a key factor. Eventually I hope to have XMM programs executing in multiple cogs (currently only one cog can be executing in XMM mode, the others have to execute one of the other modes). If that had not been a possibility I probably would not have bothered. But that will have to come in a later release.

    So much to do, and so little time! :)

  • RossHRossH Posts: 5,336
    edited 2022-12-19 04:19

    Well, some bad news and some good news ...

    The bad news is that I got held up by a nasty bug (no, not Covid-19 - I mean the programming kind), so I will not have a release of Catalina that executes programs from external RAM ready by Christmas. The bug was because the shift instructions (in this case SHR) behave differently on the Propeller 2 than they did on the Propeller 1. I mean, who on earth would mess about with an instruction as fundamental as that? (no, you don't need to tell me, I know the answer :) )

    This put me behind schedule by a few days. And Christmas is now approaching fast!

    The good news is now that I have fixed that, everything seems to be working fine. I have the largest, most complex program I routinely use (i.e. the Lua language interpreter) now executing from XMM RAM (i.e. PSRAM) on the Propeller 2. This program exercises pretty much everything - memory, file systems, various plugins etc. It means that I can now execute much larger Lua programs than was previously possible (which was in fact the whole point of the exercise).

    Of course, just like on the Propeller 1, the performance of programs executing from external RAM on the Propeller 2 will never be anything to write home about. But as the Russian proverb says "The marvel is not that the bear dances well, but that the bear dances at all." But one reason is because I am still using just an 8kb cache to cache access to up to 16Mb of XMM RAM. This relatively tiny cache size is a hang over from the Propeller 1 days, when Hub RAM was so limited. On the Propeller 2 I will be able to make the cache much larger (say 32kb or 64kb) and that should improve the performance. I just haven't had time to do it yet.

    Instead of a full release, I may post a demo version of just one program (probably the Lua interpreter), Anyone with a suitable P2 EDGE (you need the one with 32MB of PSRAM, i.e. the P2-EC32MB) will be able to load it and execute it.

    Ross.

  • RossHRossH Posts: 5,336
    edited 2023-01-16 11:24

    Post deleted.

    The standalone Propeller 2 XMM Demo has been removed from SourceForge since an improved (but incompatible) version of Propeller 2 XMM support is now incorporated in Catalina release 5.6.

    Ross.

  • RossHRossH Posts: 5,336
    edited 2022-12-21 23:19

    Some more benchmark data.

    I had a suspicion my original benchmark was too optimistic, and I was right. The choice of benchmarks makes a huge difference tp the results. Also, I have now had time to expand the cache size to 64kb, and this improves things, but not as much as I had hoped.

    Here are some better benchmark results:

    BENCHMARK 1:
    ============
    
    MODE      MEMORY TYPE,    SIZE      TIME
              CACHE SIZE     (bytes)    (ms)
    ======    ==========     =======   =====
    NATIVE    HUB RAM          6304      998
    LMM       HUB RAM          6284     1000
    CMM       HUB RAM          3884     1997
    XMM       PSRAM,64K        6760     2001
    XMM       PSRAM,8K         6760     2998
    
    
    BENCHMARK 2:
    ============
    
    MODE      MEMORY TYPE,    SIZE      TIME
              CACHE SIZE     (bytes)    (ms)
    ======    ==========     =======   =====
    NATIVE    HUB RAM          8288       34
    LMM       HUB RAM          8392      166
    CMM       HUB RAM          4872      236
    XMM       PSRAM,64K        9136      625
    XMM       PSRAM,8K         9136      699
    
    
    BENCHMARK 3:
    ============
    
    MODE      MEMORY TYPE,    SIZE      TIME
              CACHE SIZE     (bytes)   (sec)
    ======    ==========     =======   =====
    NATIVE    HUB RAM        438176        1
    LMM       HUB RAM        442912        4
    CMM       HUB RAM        232736        7
    XMM       PSRAM,64K      476732       20
    XMM       PSRAM,8K       476732       23
    

    Benchmark 1 is too trivial to be much use - it is a variation of the original benchmark, and it understates the execution time of XMM. Benchmarks 2 and 3 are closer to reality - which is that executing from external RAM can be twenty times slower than executing natively from Hub RAM. I think this can be improved with a better cache - the current one - a hangover from the Propeller 1 - was primarily optimized for size, not speed, and just making it larger has not improved things as much as I had hoped. A redesign looks to be in order.

    Still, things are not as bad as all that. Benchmark 3 represents pretty much the largest program that can execute either in Native or LMM mode on the P2. For programs larger than that, it comes down to a choice between either CMM or XMM, and XMM is only three times slower than CMM, but allows programs thirty two times larger.

    EDIT: Updated final paragraph, which was overly pessimistic.

  • RossHRossH Posts: 5,336

    A quick post-Christmas update. I've now implemented an XMM SMALL mode in addition to the XMM LARGE mode on the Propeller 2 (the Propeller 1 already had both of these). The good news is that SMALL mode is significantly faster than LARGE mode.

    For those unfamiliar with XMM execution (which would probably be many Propeller 2 only users), Catalina offers several different program execution modes:

    NMM : Propeller 2 Native (HUBEXEC) mode. Combined Code, Data, Stack and Heap limited to 512kb (i.e. Hub RAM).
    LMM : A legacy mode carried over from the Propeller 1 (essentially, the Propeller 1's equivalent to HUBEXEC mode). Similar code sizes and limits to NMM.
    CMM : A compact version of LMM. Combined Code, Data, Stack and Heap still limited to 512kb, but Code size is generally about half that of LMM or NMM.
    XMM SMALL : Code in external RAM, Data, Stack and Heap in Hub RAM. Maximum code size is 16Mb, maximum combined Data, Stack and Heap is 512kb.
    XMM LARGE : Code, Data and Heap in external RAM, Stack in Hub RAM. Maximum combined Code, Data and Heap size is 16Mb, maximum Stack size is 512kb.

    I have also made a few other small improvements, so here are the most recent benchmark results, with values adjusted to make them all relative to NMM mode:

    NMM       = 1
    LMM       = 4
    CMM       = 6
    XMM SMALL = 13
    XMM LARGE = 18
    

    LMM is not particularly useful on the Propeller 2 (it was retained mainly because I knew it would provide an easy path to XMM). Once your program won't fit in Hub RAM and so you cannot use NMM any longer (which is obviously still the preferred option) you would first move to CMM and at that point you take a pretty large performance hit. Even so, CMM can still useful (on the Propeller 1, CMM is comparable in speed to Spin). For instance, many of Catalina's own utilities (such as Catalyst, and the Lua interpreter) are compiled in CMM mode. But once you need even more code or data space after that then you now have two further options - SMALL and LARGE mode. The latest addition (SMALL) allows code sizes up to 32 times larger than CMM but at the cost of executing at half the speed. You would move to LARGE mode only once your data size (i.e. not just code size) exceeds the 512kb of available Hub RAM. Also, keep in mind that if some parts of your program absolutely need speed, they can be executing in NMM mode on some cogs while other parts of your program are executing in one or more of the slower modes on other cogs.

    I have a few ideas about how to improve the performance of XMM further, but they will require a substantial amount of re-work (which implementing the existing XMM modes did not, since they were already supported on the Propeller 1) and I want to get the new Propeller 2 implementations bedded down properly first.

    The bad news is that this is still a little way off - it is so long since I did any XMM work on the Propeller that I had forgotten that most of the complexity is not in the execution itself, it is in all the supporting utilities (program loaders etc) that have to be updated :(

    Ross.

  • Hi @RossH,

    Happy New Year!

    I've been following your Prop2 XMM thread and see that you started with the Prop1 XMM capability as the baseline for it.

    To some extent you've already covered the following questions but I want to make sure I understand where the Prop2 XMM development stands right now:

    The Prop2 XMM code can only be executed from a single cog, correct?

    But the remaining cogs can execute code using any of the other non-XMM models (NMM,LMM,CMM)?

    Does the Prop2 XMM model use a C structure to share common variables among the other cogs like the Prop1 XMM?

    Do you eventually plan on allowing multiple cogs to simultaneously execute Prop2 XMM code?

    Do you anticipate any of the techniques used in the Prop2 XMM to backfeed into an enhanced Prop1 XMM? Like, for example, allowing more than one cog to execute XMM code on the Prop1?

    (Somewhere I thought I came across a writeup wherein the GCC developers claim to have enabled the Prop1 to execute XMM code on multiple cogs with as little as 2K cache for each cog).

    Obviously multi-cog XMM execution on either the Prop1 or Prop2 wouldn't be a speed demon and performance would depend upon what type of external memory is used (among other factors) but it might be an interesting capability nevertheless.

    But allowing multi-cog XMM execution seems to be quite the task as the caching cog(s) would have to prioritize and deconflict the access to external memory. A parallel dual port memory arrangement might help in that regard but it appears to be much more difficult attempting that with standard serial RAM.

    It would be interesting to see what your solution to that problem would be.

  • RossHRossH Posts: 5,336
    edited 2023-01-03 10:44

    @Wingineer19 said:
    Happy New Year!

    And to you!

    The Prop2 XMM code can only be executed from a single cog, correct?

    Yes. The same as on the Propeller 1. But this is largely a limitation of the current cache plugin, not the XMM plugin.

    But the remaining cogs can execute code using any of the other non-XMM models (NMM,LMM,CMM)?

    Yes. The same as on the Propeller 1.

    Does the Prop2 XMM model use a C structure to share common variables among the other cogs like the Prop1 XMM?

    Yes. The same as on the Propeller 1.

    Do you eventually plan on allowing multiple cogs to simultaneously execute Prop2 XMM code?

    Yes. This should be possible on the Propeller 2.

    Do you anticipate any of the techniques used in the Prop2 XMM to backfeed into an enhanced Prop1 XMM? Like, for example, allowing more than one cog to execute XMM code on the Prop1?

    Not likely. On the Propeller 1 both Hub space and cog space are too limited. But with the Propeller, you should never say never!

    (Somewhere I thought I came across a writeup wherein the GCC developers claim to have enabled the Prop1 to execute XMM code on multiple cogs with as little as 2K cache for each cog).

    Not with the cache architecture I am currently using, which was originally the same one GCC used. But I didn't really stay in touch with the GCC progress, because Catalina was always so much more capable and GCC could never do what I needed. But I'll check to see if they later came up with a better cache architecture that the one which I adopted. If so, I could perhaps use it on both the Propeller 1 and the Propeller 2. If you or anyone else has a reference or link, please let me know.

    Obviously multi-cog XMM execution on either the Prop1 or Prop2 wouldn't be a speed demon and performance would depend upon what type of external memory is used (among other factors) but it might be an interesting capability nevertheless.

    Yes it is definitely not a speed demon! My main purpose in implementing XMM on the Propeller 2 is to enable larger Lua programs, and eventually multi-processing Lua programs. Lua is my favorite language these days, and it is such a rich language that speed is not really a major issue. C and Lua are highly inter-operable - you use C where speed is necessary, and Lua where things are simply too difficult to code in C. Actually, you have just raised a good point - I don't yet have a means of invoking non XMM C from Lua. You can invoke C but it will always execute as XMM C. I should make it possible to invoke NMM C from Lua for when you need full speed. Just another thing to add to the "to do" list! :)

    But allowing multi-cog XMM execution seems to be quite the task as the caching cog(s) would have to prioritize and deconflict the access to external memory. A parallel dual port memory arrangement might help in that regard but it appears to be much more difficult attempting that with standard serial RAM.

    The PSRAM XMM plugin already allows multiple cog access, so a simplistic solution would be to simply have multiple independent caches all using the same XMM plugin. But this would consume too many cogs to be much use. It may be a valid approach if we ever see the 16 cog version of the Propeller 2, but I have another approach in mind. However, I haven't had time to investigate it yet.

    It would be interesting to see what your solution to that problem would be.

    Yes, indeed it would! :)

Sign In or Register to comment.