Catalina - 500 downloads and counting! Now tell me what you want next!

RossH · 2010-09-18 03:53

Okay! Today Catalina 2.6 has officially reached 500 downloads, so to celebrate I thought I'd try conducting a poll to see what Catalina improvements people wanted to see in the forthcoming months.

As you may know, I no longer have as much time as I would like to do Catalina work, so this is your chance to tell me what you would most like to see. I may be able to do only one, or I may be able to do more. Only time will tell.

Ross.

P.S. For those interested, the stats are 80 Linux downloads to 423 Windows downloads - or 16% Linux, 84% Windows. Of course, some people download both versions, so this doesn't necessarily mean 500 actual users - but then who's counting?

Heater. · 2010-09-18 04:17

Congratulations, that's quite a milestone.

Is that the figure for all version numbers or just the latest one?

I have accumulated a three or four versions for both Windows and Linux.

This prompted me to check the number of Zog downloads which is now a grand total of 56 for version 1.0. Zog has a way to go.

I won't vote for any more work for you to do, but if you could just squeeze C++ and Fortran support into Catalina that might give Zog a chance to catch up whilst you are busy:)

David Betz · 2010-09-18 04:55

RossH wrote: »

Okay! Today Catalina 2.6 has officially reached 500 downloads, so to celebrate I thought I'd try conducting a poll to see what Catalina improvements people wanted to see in the forthcoming months.

How about SPI SRAM support? I haven't been able to get any of my programs running in hub RAM. A little more breathing space would be nice. I can even offer to help.

RossH · 2010-09-18 05:02

Heater. wrote: »

Congratulations, that's quite a milestone.

Is that the figure for all version numbers or just the latest one?

The numbers are only for version 2.6.

I'll see what I can do about Fortran - there are several Fortran to C converters available.

RossH · 2010-09-18 05:04

David Betz wrote: »

How about SPI SRAM support? I haven't been able to get any of my programs running in hub RAM. A little more breathing space would be nice. I can even offer to help.

Hi David - I thought someone would ask for this :smhair: I'll keep working on it in the background, but I have my doubts it can ever be made fast enough to be useful for anything.

Ross.

Bill Henning · 2010-09-18 07:03

Congrats Ross!

It's only a matter of time before you hit 1,000...

Re/ SPI RAM:

I suspect that if you use VMCOG with a decent size working set, Catalina will slow down by about a factor of 2-3 compared to built-in XMM drivers - however it would allow large programs even on small cheap boards, and still be faster than Spin

You could try the VMCOG interface routines in heater's ZOG source, I think they would integrate into your Catalina LMM fairly easily.

Currently VMCOG only provides a 64KB VM, however I've started working on the large VM version which will support 512KB+++

RossH wrote: »

Hi David - I thought someone would ask for this :smhair: I'll keep working on it in the background, but I have my doubts it can ever be made fast enough to be useful for anything.

Ross.

Cluso99 · 2010-09-19 02:29

Congratulations Ross. Quite a milestone indeed.

RossH · 2010-09-19 16:21

Thanks for all the feedback!

There seem to be lots of requests for new device drivers - if this is one of your wishes, can you also tell me which specific devices you would like to see added?

Thanks.

P.S. I would edit the poll to add this to the poll question if I could - does anyone know if this is possible?

David Betz · 2010-09-19 17:03

I've just started using 2.7 and I want to say THANK YOU for fixing the macro bug that was forcing me to run everything through the GNU C preprocessor before compiling with Catalina. I can now use Catalina by itself. That cleaned up my build process considerably and also makes it easier to understand the error messages produced by Catalina since they now refer to the original source not some hacked up thing generated by CPP.

THANKS!!!

Invent-O-Doc · 2010-09-21 04:33

One really powerful capability would be to allow integration of spin objects into Catalina so that they can be instantiated and called from C. This would mean that there would be less pressure to maintain and expand a C-based object library.

I don't know if this is feasible, but if it happened, I would switch from SPIN to C.

RossH · 2010-09-21 06:33

Invent-O-Doc wrote: »

One really powerful capability would be to allow integration of spin objects into Catalina so that they can be instantiated and called from C. This would mean that there would be less pressure to maintain and expand a C-based object library.

I don't know if this is feasible, but if it happened, I would switch from SPIN to C.

Hi Invent-O-Doc.

Thanks for the suggestion. I have given some thought to allowing SPIN programs and C programs to coexist, but I never pursued it because once I could program the Prop using C there was not much that I needed to do in SPIN anymore. The existing programs I did need (mostly various device drivers) were generally written in PASM (for speed) with a simple SPIN "wrapper" module - and with these all I need to do is replace the SPIN wrapper with a C equivalent and then load the PASM code as a Catalina plugin. This conversion is a fairly simple and mechanical process, and is how most of the Catalina drivers originated.

Also, for a while it looked like Parallax themselves were going to go a fair way to making your suggestion unnecessary - their "Parallax Micro C" proposal would have added C-like syntax to SPIN. I've not heard how that's progressing recently, but in many ways it would have been a good answer for people who mostly want to use SPIN, but want to include a small amount of C code (e.g. a single function).

However, I'd be willing to investigate this a bit further in Catalina if there was sufficient demand for this feature - now that Catalina can load and execute multiple LMM kernels dynamically, it would be fairly easy to write a SPIN module to launch a Catalina LMM kernel and have it execute a specified precompiled Catalina C function.

However, that's only part of the story - making is possible for the C and SPIN programs interact is more problematic. Instead of allowing C to call SPIN directly (or vice-versa) I would probably instead write both a SPIN module and a C module that knew how to interact via the Catalina registry (i.e. via Hub RAM). I have some code already written to do this kind of thing anyway since I needed it for Catalyst. If you really wanted an interface that looked more like a direct procedure call you could overlay a simple RPC protocol on top of the registry primitives.

If anyone else thinks there is a need for this feature, by all means let me know by adding a comment to this thread (I don't think I can edit the poll itself, and in any case I think a poll can only have 10 options).

Ross.

David Betz · 2010-09-22 17:56

I've been reading the Catalina manual and I have a question about loaders. It seems that using the -x1 format is the only way to create a program that can use all of hub RAM for code but the only way that can be loaded is by programming it into the upper half of a 64k or larger EEPROM. Is there any way to extend the SD loader to allow it to load these larger hub RAM programs directly off of an SD card instead of from EEPROM?

Mike Green · 2010-09-22 17:58

Has anyone had any luck installing Catalina under the MacOS?

RossH · 2010-09-22 19:12

David Betz wrote: »

I've been reading the Catalina manual and I have a question about loaders. It seems that using the -x1 format is the only way to create a program that can use all of hub RAM for code but the only way that can be loaded is by programming it into the upper half of a 64k or larger EEPROM. Is there any way to extend the SD loader to allow it to load these larger hub RAM programs directly off of an SD card instead of from EEPROM?

Hi David,

I don't believe there is currently a way to do this - I'd have to write a new loader specifically to cover this situation. However, it shouldn't be too difficult, and I can see it would be a good option to have. I'll have a look at adding it in the next couple of days.

Ross.

David Betz · 2010-09-22 19:15

RossH wrote: »

Hi David,

I don't believe there is currently a way to do this - I'd have to write a new loader specifically to cover this situation. However, it shouldn't be too difficult, and I can see it would be a good option to have. I'll have a look at adding it in the next couple of days.

Ross.

Thanks! That would be great for those of us who don't have XMM support yet!

RossH · 2010-09-22 19:15

Mike Green wrote: »

Has anyone had any luck installing Catalina under the MacOS?

Hi Mike,

You will need to rebuild Catalina from source to run it under OSX. An earier release was successfully built by Infinity, but he no longer seems to be around on the forums.

If you try it, I'd appreciate it if you report any issues and I'll try to address them.

Ross.

RiJoRi · 2010-09-23 09:38

All the above are nice, but what I think we ALL want is a compiler that compiles what we want, not what we say!!

(OK, Rich, keep dreaming!

)

--Rich

RossH · 2010-09-23 16:04

RiJoRi wrote: »

All the above are nice, but what I think we ALL want is a compiler that compiles what we want, not what we say!!

--Rich

In the next release I plan to incorporate Beau's Neural Net Engine, so Catalina can debug your programs for you as they are loaded.

Ross.

Invent-O-Doc · 2010-09-24 12:24

RossH wrote: »

Hi Invent-O-Doc.

Thanks for the suggestion. I have given some thought to allowing SPIN programs and C programs to coexist, but I never pursued it because once I could program the Prop using C there was not much that I needed to do in SPIN anymore. The existing programs I did need (mostly various device drivers) were generally written in PASM (for speed) with a simple SPIN "wrapper" module - and with these all I need to do is replace the SPIN wrapper with a C equivalent and then load the PASM code as a Catalina plugin. This conversion is a fairly simple and mechanical process, and is how most of the Catalina drivers originated.

Also, for a while it looked like Parallax themselves were going to go a fair way to making your suggestion unnecessary - their "Parallax Micro C" proposal would have added C-like syntax to SPIN. I've not heard how that's progressing recently, but in many ways it would have been a good answer for people who mostly want to use SPIN, but want to include a small amount of C code (e.g. a single function).

Hi RossH. Thanks for the detailed and thoughtful reply. I see the approach you are considering; ie having a set of procedure/method tunnels to spin. I'd like to see this!

As far a PASM wrappers for some of the objects, thats a good idea I haven't considered.

Never heard of the Parallax micro C for prop? Wait I checked the forum. What an intriguing idea - A micro C - C-Spin translator would give me easy FLOATS, not worrying about variable passing value/address and access to the entire SPIN library! Wow. Comparing code between C and SPIN is really educational and also having a build in C will help with the educational market.

As far as the Catalina C, it is still needed because it offers advantages like full assembly to PASM and the like.

RossH · 2010-10-19 01:31

The two most requested features from this poll seem to be more device drivers and smaller/faster C libraries.

To address the latter, here is my proposal for a "minimal" subset of the standard ANSI C library functions that I think would be most useful in an embedded environment. My goal would be to have the entire libtiny library occupy around 4-6kb of Hub RAM (less if possible).

As much of the library as possible would be implemented using a single library cog - this is both for speed reasons and also because cog-based PASM tends to be smaller in code size than LMM PASM.

If anyone has any comments (or requests for additions) please let me know. Note that it would be possible to use this library in addition to parts of the standard C libraries (such as the maths functions, or malloc/free) - but the libtiny library is specifically intended for small embedded projects that would not normally require floating point or dynamic memory allocation. Of course, the basic floating point types are still available, as are the basic floating point operations (like +,-,*,/) - these are built into the kernel. I am talking here about leaving out the maths functions - like cos and exp).

#ifndef _TINY_H
#define _TINY_H
/*
 * tiny.h - a tiny C library for embedded applications. All functions are
 *          written to reduce the size of code, sometimes at the expense of
 *          execution speed. All the functions are compatible with the ANSI 
 *          functions of the same name, except for printf and sscanf which
 *          only support a very limited number of formats. 
 *
 *          Notable exceptions from this list of functions are floating point,
 *          file operations and memory management functions - if these are 
 *          required, or any other function not listed, it would be possible 
 *          mix and match libtiny with the standard ANSI versions of these 
 *          functions. However, libtiny is mainly intended to be "stand-alone".
 * 
 *          The libtiny functions make use of a dedicated Propeller cog to 
 *          implement as much of the functions as posible - this enhances
 *          speed and also minimizes the runtime code size, but at the 
 *          expense of a cog.
 *
 *          Note that this file is merely a convenience. The various standard 
 *          C include files can be used instead - but they may also define
 *          additional functions which are not included in libtiny.
 */

/*
 * include standard definitions (e.g. size_t, NULL, div_t, ldiv_t):
 */
#include <stddef.h>
#include <stdlib.h>

/*
 * ctype.h equivalents:
 */
int    isalnum(int _c);
int    isalpha(int _c);
int    iscntrl(int _c);
int    isdigit(int _c);
int    isgraph(int _c);
int    islower(int _c);
int    isprint(int _c);
int    ispunct(int _c);
int    isspace(int _c);
int    isupper(int _c);
int    isxdigit(int _c);
int    tolower(int _c);
int    toupper(int _c);

/*
 * string.h equivalents:
 */
void   *memcpy(void *_s1, const void *_s2, size_t _n);
void   *memmove(void *_s1, const void *_s2, size_t _n);
char   *strcpy(char *_s1, const char *_s2);
char   *strncpy(char *_s1, const char *_s2, size_t _n);
char   *strcat(char *_s1, const char *_s2);
char   *strncat(char *_s1, const char *_s2, size_t _n);
int    memcmp(const void *_s1, const void *_s2, size_t _n);
int    strcmp(const char *_s1, const char *_s2);
int    strncmp(const char *_s1, const char *_s2, size_t _n);
void   *memchr(const void *_s, int _c, size_t _n);
char   *strchr(const char *_s, int _c);
char   *strrchr(const char *_s, int _c);
char   *strstr(const char *_s1, const char *_s2);
void   *memset(void *_s, int _c, size_t _n);
size_t strlen(const char *_s);

/*
 * stdio.h equivalents:
 */
int    getchar(void);
char   *gets(char *_s);
int    putchar(int _c);
int    puts(const char *_s);
int    printf(const char *_format, ...);
int    scanf(const char *_format, ...);

/*
 * stdlib.h equivalents:
 */
int    atoi(const char *_nptr);
long   atol(const char *_nptr);
double strtod(const char *_nptr, char **_endptr);
long   strtol(const char *_nptr, char **_endptr, int _base);
unsigned long strtoul(const char *_nptr, char **_endptr, int _base);
int    rand(void);
void   srand(unsigned int _seed);
int    abs(int _j);
div_t  div(int _numer, int _denom);
long   labs(long _j);
ldiv_t ldiv(long _numer, long _denom);

#endif

Ross.

P.S. Catalina has now been downloaded 700 times!

Bill Henning · 2010-10-19 07:06

Looks like a very workable subset!

(although some provision for stderr would be nice)

vencejo · 2010-10-19 08:10

It would be wonderful to play around with code:: Blocks and Catalina.

Unfortunately, integration is very difficult if you do not have the same tree of files that marks the manual , who roots in C: \ Program Files \ Catalina.
If it is not there, you have to edit an improbable amount of files.

I would ask that would make easier an more flexible the integration of Code:: Blocks with Catalina.

David Betz · 2010-10-19 08:26

I'd like a small library that included at least minimal SD card file I/O including open/close/read/write of at least two files at a time. I guess that is unlikely in the LMM using only hub RAM though.

RossH · 2010-10-19 16:01

Hi David,

The "tiny" library is intended to contain only ANSI standard functions. The stdio file access functions are just too heavy because they not only need all the file system code, they also tend to pull in all of the additional stdio code as well - which I have deliberately omitted from the tiny library as it is so large.

However, file access in Catalina LMM mode is already possible if you're willing to do a bit of work - you just have to use the DOSFS library calls directly instead of stdio.

I just compiled a simple test program that uses these calls (e.g. it uses DFS_OpenFile and DFS_ReadFile instead of the standard stdio calls fopen() and fgetc() etc) - and it fits easily into 32k. But before you go crazy with them, see the attached dosfs.h header file - these calls are quite UGLY and hard to use. I'll see if I can come up with a file access library to simplify direct DOSFS access.

Hi Vencego,

You're right, the example Code::Blocks configuration files I provide assume Catalina is installed in the standard place (C:\Program Files\Catalina). I'll add some instructions on how to set it up Code::Blocks you have Catalina installed elswehere.

RossH · 2010-10-19 23:31

vencejo wrote: »

It would be wonderful to play around with code:: Blocks and Catalina.

Unfortunately, integration is very difficult if you do not have the same tree of files that marks the manual , who roots in C: \ Program Files \ Catalina.
If it is not there, you have to edit an improbable amount of files.

I would ask that would make easier an more flexible the integration of Code:: Blocks with Catalina.

Hi vencejo,

I just had a bit of a play with this, and you should not need to edit any files to make Code::Blocks work with Catalina even when Catalina is installed in a non-standard location.

After installing Catalina (to wherever you want) and then installing Code::Blocks (according to the documentation provided with Catalina) there are just two additional things you need to do:

Set your LCCDIR environment variable. To do this in Windows XP, enter Start->Settings->Control Panel->System and then select the Advanced tab. Press the Environment Variables button and then New. Add a new variable LCCDIR with the appropriate value (e.g. C:\Program Files\Catalina2). Then close the dialog box and restart Code::Blocks so it sees the change.
In Code::Blocks, select Settings->Compiler and Debugger..., then select Catalina as the compiler (I suggest making this the default). Then select the Toolchain executables tab and update the Compiler's installation directory to the location you have chosen for Catalina (e.g. C:\Program Files\Catalina2).

Of course, the example workspaces I provide also have the default path hardcoded into them - unfortunately there is nothing I can do about this - but you can simply create new workspaces from within Code::Blocks as required and everything should now work correctly.

I will update the documentation with these additional steps. Let me know if you have any problems.

Ross.

vencejo · 2010-10-21 12:44

Hi Ross, Thank you very much for your help.
At last I managed to integrate codeblocks and Catalina.
It was not so difficult,thanks to your instructions

RossH · 2010-10-22 19:51

More new (and smaller) libraries - this time a new file access library ...

The catalina_fs.h header file below defines a set of low level functions to access an SD card file system. Using these functions instead of the normal stdio functions (i.e. fopen, fputc, fgetc etc) allows C programs to use the SD card and still easily fit into the Propeller's 32k hub RAM.

I think it may be possible to make these new file access functions, plus the new libtiny library (described above) fit into 16kb - allowing a feature-rich C environment that still leaves 16kb for C application programs (and of course the size of the library functions will typically be less than 16kb because most programs won't need all the available functions).

The set of functions I have included is in fact the "minimal" set required to achieve ANSI compatibility (in fact, what they really do is expose the particular DOSFS functions that underly stdio).

One thing I have done to make DOSFS smaller is remove FAT12 support. Now only FAT16 and FAT32 are supported. If anyone really needs FAT12 support, it can be re-enabled - but will require that the libraries be re-compiled, and it will make the resulting executables a bit larger.

Another change is that there are in fact two sets of create/open/close functions - one that uses statically allocated memory that the application provides, and another that uses dynamically allocated memory that the routines manage using malloc & free. The advantage of the static functions is that programs that use them do not need to include the malloc and free library functions (which require quite a lot of Hub RAM).

These new library functions will be included in the next release of Catalina (along with the new multithreading support).

Ross.

#ifndef __CATALINA_FS_H
#define __CATALINA_FS_H

#include <sys/types.h> 
#include <dosfs.h>

extern PFILEINFO __fdtab[FOPEN_MAX];

extern uint32_t __pstart;  // will be -1 until an sd card is mounted

extern VOLINFO  __vi;      // information of mounted sd card

/*
 * mount must be called (once) before any file system access.
 * (unit and pnum are normally left as zero). Note that only
 * SD cards WITH AN MBR are supported. 
 */
extern int _mount(int unit, int pnum);

/*
 * unmount must be called (once) before another SD card can be mounted.
 */
extern int _unmount();

/*
 * open a file and return the file number (a pointer to a FILEINFO structure
 * must be provided).
 * 
 * The mode can be:
 *    0 - read only
 *    1 - write only
 *    2 - read and write
 */
extern int _open(const char *path, int flags, PFILEINFO fd);

/*
 * close an open file using the file number.
 */
extern int _close(int d);

/*
 * open a managed file and return the file number (managed files have the 
 * FILEINFO structure allocated and managed internally - which requires that
 * malloc be used by the program). The file must be closed using the managed 
 * close function. 
 *
 * The mode can be:
 *    0 - read only
 *    1 - write only
 *    2 - read and write
 */
extern int _open_managed(const char *path, int flags);

/* 
 * close a managed file (managed files have the FILEINFO structure allocated
 * and managed internally). The file must have been opened using the managed
 * open function.
 */
extern int _close_managed(int d);

/*
 * read from a file.
 */
extern int _read(int d, char *buf, int nbytes);

/*
 * write to a file.
 */
extern int _write(int d, const char *buf, int nbytes);

/*
 * seek within a file.
 */
extern off_t _lseek(int d, off_t offset, int whence);

/*
 * create and open a new file and return the file number (a pointer to a 
 * FILEINFO structure must be provided).
 *
 * The mode can be:
 *    0 - read only
 *    1 - write only
 *    2 - read and write
 */
extern int _create(const char *path, int mode, PFILEINFO fd);

/*
 * create and open a new managed file (managed files have the FILEINFO 
 * structure allocated and managed internally). The file must be closed 
 * using the managed close function.    
 *
 * The mode can be:
 *    0 - read only
 *    1 - write only
 *    2 - read and write
 */
extern int _create_managed(const char *path, int mode);
/*
 * rename a file.
 */
extern int _rename(const char *name1, const char *name2);

/*
 * unlink (delete) a file.
 */
extern int _unlink(const char *path);

#endif

RossH · 2010-10-24 21:28

Another change that will be included in the next version of Catalina. This one wasn't on the original poll list, but came up in discussions with David Betz, and I think it will prove to be very useful:

Addition of the SDCARD symbol, to complement the existing EEPROM symbol.

Some more explanation of the EEPROM and SDCARD symbols is probably required:

The EEPROM or SDCARD symbols can be defined on the comand line to specify that a two-phase loader should be included in the final executable - even if the program is simply a normal LMM program and could possibly be loaded using the Propeller's built-in one-phase loader.

As the name implies, a two-phase loader loads the program in two completely separate phases - i.e:

The device drivers and other plugins are loaded into Cog RAM and started. This part of the program always occupies the first 32kb of the executable. The first 32kb also contains the second-phase loader, which (when loaded and run) loads the actual application program.
The application program is loaded into Hub RAM and started. The application program always starts at the 32kb boundary in the executable, and expects to be loaded by the second-phase loader that was loaded during the first phase.

The advantage of a two-phase loader is that no Hub RAM is wasted on device drivers or plugins after they are initialized. This Hub RAM can instead be used as more application CODE space. This allows applications to be much closer in size to the full 32kb of Propeller Hub RAM than is possible when using a simple one-phase load.

If a two-phase loader is not used, the memory occupied in the executable by the various device drivers or plugins is automatically recovered and made available by Catalina as DATA space to the application program - but it cannot be used as additional CODE space. This can be a severe limitation for some applications, since the device drivers and plugins that even a very simple program may require can end up consuming most of the available Hub RAM.

As an example, using a one-phase load means even a simple "hello, world" type application in C can result in an executable size of 23kb, leaving only 9kb for additional program code (although Catalina already provides various other mechanisms to reduce this dramatically). But now, simply by using the SDCARD two-phase loader, this program could now require as little as 512 bytes of hub RAM - with the remaining 31kb+ of Hub RAM available for use as additional program CODE space!

The disadvantage of a two-phase loader is that programs that include them will always end up larger than 32kb, and hence cannot be loaded and run on a bare Propeller - e.g. the EEPROM loader requires an external EEPROM at least 64kb in size, and the SDCARD loader requires an external SD Card adapter.

The existing EEPROM loader is used by default when the -x1 (EMM) memory model is used, and can also be specified when using the -x2 (XMM SMALL) or -x5 (XMM LARGE) memory models (note that defining the EEPROM symbol is not related to using the -b or -e command line options, which simply specify whether to use the .binary or .eeprom file format for the final executable).

The -x1 (EMM) option is not really a memory model at all - it specifies a load option. The -x1 option achieves the same result as would have been achieved by specifying -x0 and -D EEPROM if that combination had been supported (in fact it was not previously supported, but now is).

The SDCARD option can bow be used in conjunction with -x0 (LMM TINY), -x2 (XMM SMALL) or -x5 (XMM LARGE) memory models. It is the default for the XMM memory models. Now it can be specified for the LMM memory model programs as well - but the resulting program MUST be loaded from an SD Card even though it is just an ordinary LMM program.

Here is an exhaustive list of the possible combinations of memory models and loader options:

LMM (TINY) and EMM programs:

catalina test.c                 => LMM program, uses built-in one-phase loader
catalina test.c -x0             => same as previous
catalina test.c -x0 -D SDCARD   => LMM program, uses SDCARD two-phase loader
catalina test.c -x0 -D EEPROM   => LMM program, uses EEPROM two-phase loader

catalina test.c -x1             => same as previous
catalina test.c -x1 -D EEPROM   => same as previous
catalina test.c -x1 -D SDCARD   => LMM program, uses SDCARD two-phase loader
 
XMM (SMALL) programs:

catalina test.c -x2             => XMM program, uses SDCARD two-phase loader
catalina test.c -x2 -D SDCARD   => same as previous
catalina test.c -x2 -D EEPROM   => XMM program, uses EEPROM two-phase loader

XMM (LARGE) programs:

catalina test.c -x5             => XMM program, uses SDCARD two-phase loader
catalina test.c -x5 -D SDCARD   => same as previous
catalina test.c -x5 -D EEPROM   => XMM program, uses EEPROM two-phase loader

NOTE: Although -x1 will be is retained for backward compatibility, this new approach clarifies the fact that -x1 (EMM) programs are simply LMM programs that use a two-phase loader (the EEPROM loader by default), and hence that EMM is not really a different memory model to LMM.

This new addition will make Catalina much more useful on platforms (such as the C3) which do not have built-in XMM RAM, but which do have an SD Card.

Ross.

Cluso99 · 2010-10-24 22:27

Nice Ross and David. I am sure the two-phase loader will be a worthwhile addition.

Dr_Acula · 2010-10-24 22:35

As the name implies, a two-phase loader loads the program in two completely separate phases - i.e:
The device drivers and other plugins are loaded into Cog RAM and started. This part of the program always occupies the first 32kb of the executable. The first 32kb also contains the second-phase loader, which (when loaded and run) loads the actual application program.
The application program is loaded into Hub RAM and started. The application program always starts at the 32kb boundary in the executable, and expects to be loaded by the second-phase loader that was loaded during the first phase.

Now this sounds absolutely fascinating. I've tried to think of ways to do this in Spin but I don't think you can. Not without getting inside the compiler/interpreter, anyway.

This sort of thing may make Catalina a *better* language than Spin.

Consider a hybrid C and PASM model using this two phase loader. You could free up a lot more hub ram for video memory for instance, and that is hard to do in Spin as the bits of leftover memory end up scattered all over the hub memory.

Going into the detail here, say you load a program into a cog and then set it running. And say you have a pointer to the cog (C is good with pointers, I'm teaching myself C and this is one big strength of C, but it is really complex to understand).

Then your second phase loader starts running. How does it talk to the cogs? Are there pointers to the cog at certain locations in the hub? Or is the compiler smart enough that this happens in the background and the cog code appears as a function?

Have you got an example of how this might work - eg with something like a keyboard driver you load into a cog on the first phase, then some code in the second phase to see if any bytes are available?

RossH · 2010-10-24 23:11

Hi Dr_Acula, Hi Cluso ...

All Catalina XMM programs can already use the two-phase SDCARD or EEPROM loader - you can look at those for a working example, and the process is exhaustively described in the Catalina reference manual.

All I am doing here is making a variant of the SDCARD loader available for use by LMM programs (LMM programs already have a two-phase loader for loading from EEPROM, but not yet one for loading from an SD Card).

David has not contributed to this, by the way - it's just that it was his comments that helped me realize that I have been focusing Catalina far too heavily at XMM-enabled platforms and not providing enough support for those who might want to use C on a "traditional" propeller platform (such as the forthcoming C3).

I'm attempting to rectify this now with the new loaders, the smaller libraries and the new multithreading capabilities. While all these new features will absolutely fly on the Prop II, they will also go a long way to making Catalina much more usable on the Prop I.

None of this is much work, by the way - all the building blocks are already there, and in fact have been for the past few releases of Catalina - it's just that now Catalina is stable enough that I'm not madly bug fixing, and so instead I'm finally getting a chance to put it all together.

As to your specific questions - essentially the first loader loads up all the plugins and device drivers and allocates them VAR space from upper RAM - which it passes to them as a startup parameter (via a pointer). These plugins and drivers then just sit there doing whatever they do (usually, waiting for a request to arrive via the Catalina registry) while the second phase loader (which has also been loaded into a cog during the first load phase, and started) then does its stuff of loading the actual application code from the second 32kb block of the SD card file. Then as a last step, the second phase loader overwrites itself with the kernel that it has previously stashed away in upper Hub RAM. This is so that the loader itself doesn't waste a cog, and also so that it only uses RAM that will usually be used only as stack space anyway. This technique can be used to load application programs up to approximately 30kb - which is essentially as big as they can get since nearly all programs will need at least 2kb of stack space at run time in any case.

My XMM two-phase loaders are specific to Catalina, since they need to descipher information in the executable that describes the memory model in use and their sizes (they need this information to know where to put the various segments, since some end up in Hub RAM and others end up in XMM RAM). However, if I can I will make the LMM two-phase loaders Catalina-independent - so that they can be used to load non-Catalina programs (e.g. SPIN/PASM programs) as well.

Ross.

Catalina - 500 downloads and counting! Now tell me what you want next!

Comments