Just a couple of notes on using the 2-port and 8-port serial plugins in Propeller 2 XMM programs.
The default pins for the second serial port are 52 & 50. However, these defaults WILL conflict with the pins used for PSRAM on the P2-EC32MB, and MAY conflict with the pins used for the HyperFlash/HyperRAM add-on board on other P2 EVAL or P2 EDGE boards. To use these plugins in an XMM program, change the pins to something else in the relevant Platform configuration file (e.g. P2_EDGE.inc or P2_EVAL.inc). Set both the tx and rx pins to -1 to disable the second port altogether.
Apparently, the 2-port serial plugin works in both XMM SMALL and LARGE programs, but the 8-port serial plugin only works in XMM SMALL programs (and in non-XMM programs, of course). I have confirmed this, but it looks like it needs more than a simple configuration change, so I will need to investigate further and fix it in a future release.
Unlike the 2 port serial plugin, the 8 port auto-initialize process (which automatically opens any ports defined in the platform configuration file) uses static data buffers, not local data buffers. In a SMALL XMM program, all data (static or local) is in Hub RAM so everything works, but in a LARGE XMM program, static data is in XMM RAM not Hub RAM, and this is out of reach of the plugin.
A simple workaround is to manually close any auto-initialized ports (or disable the auto-initialize functionality in the relevant Platform file by setting the tx and rx pins to -1) and open them manually using local data buffers.
I thought it must be something like this, but my d'oh moment was that the program I used to try and test it (demos/test_serial8_2.c) closes the auto-initialized ports, but then re-opens them using more static data buffers
I will fix both the auto-initialize functionality and the test program in the next release.
No major new functionality, just fixes for a few more issues. It is a full release because it requires a rebuild of all the C libraries.
Here is the relevant extract of the README.TXT file:
RELEASE 7.6.2
New Functionality
-----------------
1. Two new catapult multi-model demo programs have been added, which
demonstrate a threaded secondary program being executed from an XMM
primary program (which would not support multi-threading if it was
compiled as a simple XMM program). See demos\catapult\thread_p1.c and
demos\catapult\thread_p2.c - note that the 'build_utilities' script
must be used to build an XMM loader suitable to load them.
2. The default pins used for the second serial port when the 2 port or 8
port serial plugins were in use (pins 50 & 52) conflicted with the pins
used for the PSRAM on the P2-EC32MB, and may conflict with the pins
used for the HyperFlash/HyperRAM add-on board, which made these plugins
unusable in XMM programs. The second port has now been disabled by
default by setting the tx and rx pins to -1 on all P2 platforms.
Other Changes
-------------
1. Fixed a problem with Catapult when a secondary program required threads
- e.g. the secondary pragma included "options(-lthreads)" - but the
primary program did not - the secondary program was being compiled to use
the threads library but was being started using the non-threaded kernel
and thus would not execute correctly. Affected the Propeller 1 and
Propeller 2 under Windows and Linux.
2. Fixed a bug in the XMM dynamic kernel, which broke the start.c program in
the demos\p2_psram folder, which needs to load the XMM kernel dynamically.
This was the only program that uses that particular version of the kernel.
Also, the Makefile referred to the target directory using a relative
reference, which only worked if the programs were built in the installed
source tree. Affected the Propeller 2 only under Windows and Linux.
3. The 8 port serial plugin "autoinitialize" functionality, which opens any
ports defined in the relevant platform file (e.g. P2EDGE.inc, P2EVAL.inc
etc) was allocating buffers using static data, not local data. This meant
that the plugin would not work an XMM LARGE program (it worked in all
other memory models, including XMM SMALL programs). Affected the Propeller
2 only under Windows, Linux and Catalyst.
4. The 8 port serial plugin test program (demos/test_serial8_2.c) was opening
ports using static data buffers instead of local buffers, and so it would
not work in XMM LARGE mode (it worked in all other memory models, including
XMM SMALL programs). Now it uses local buffers and works in all modes. Also,
the default pin numbers for the second port are now 18 (tx) and 20 (rx).
Affected the Propeller 2 only under Windows, Linux and Catalyst.
The current version of catapult can't be used with the parallelizer because it strips out #pragma statements it doesn't understand. This means when the parallelizer runs (which also relies on pragmas), it thinks it has nothing to do. Silly of me.
The next version will still warn about unrecognized pragmas but will leave them in the source, so that catapult can be used to build parallelized programs simply by adding the appropriate options to the the relevant pragmas.
I will include a working demo in the next release.
No major new functionality, mainly just tweaks to allow Catapult to work with the Parallelizer.
The Catapult documentation has also been revised and expanded - see Getting Started with Catapult.
Here is the relevant extract of the README.TXT file:
RELEASE 7.6.3
New Functionality
-----------------
1. The Catalina Catapult utility now ignores pragmas it does not recognize,
but leaves them intact. Only one warning message is now issued (for the
first such pragma) even if there are multiple unrecognised pragmas. This
allows primary and secondary programs to use the Catalina parallelizer.
2 Four new demo programs are now included (in demos\catapult) for the
Propeller 1 and 2 to demonstrate the use of the Catalina parallelizer
with Catapult:
For the Propeller 1:
ll_p_p1.c - Demonstrates a parallelized primary program.
ll_s_p1.c - Demonstrates a parallelized secondary program.
For the Propeller 2:
ll_p_p2.c - Demonstrates a parallelized primary and secondary program.
ll_s_p2.c - Demonstrates two parallelized secondary programs.
3. The Catalina parallelizer utility now unregisters the factory kernel cogs
when they are stopped via the "#pragma propeller stop" pragma. Previously
these cogs were stopped but not unregistered.
4. The Catalina catapult utility now unregisters the secondary kernel cog
when a secondary program terminates by exiting its function. Previously,
these cogs were stopped but not unregistered.
5. On the Propeller 1, the cache cog is now registered when used. This means
the registry is displayed correctly, and that this cog is correctly marked
as being in use if a program wants to search the registry to locate an
unused cog (normally, ANY_COG is used to start a new cog, which does not
depend on whether or not the cog was registered). This was already the
case on the Propeller 2.
Other Changes
-------------
1. The Catalina Geany command to build utilities did not specify that it
should run in the project directory, not the current directory. The
version in this release fixes this for all new installations of Geany,
but it may not fix it for existing versions of Geany or existing projects.
For existing versions of Geany, select the "Build -> Set Build Commands"
menu item, and add %p to the "Working Directory" field of the "Build
Utilities" command.
For existing project files, open the project and select the "Project ->
Properties" menu item, then select the "Build" tab and add %p to the
"Working Directory" field of the "Build Utilities" command.
Affected the Propeller 1 and Propeller 2 on Windows and Linux.
2. Fixed a bug that may have led to some symbols being defined multiple
times in complex parallelized or multi-model programs.
The symbols that may have ended up multiply defined were:
cmmd_array, lmmd_array, nmmd_array (on the Propeller 1 & 2), and
CMM_LUT_LIBRARY_array, LMM_LUT_LIBRARY_array and NMM_LUT_LIBRARY_array
(on the Propeller 2 only).
Affected the Propeller 1 and Propeller 2 under Catalyst, Windows and Linux.
3. Some superseded, unused (and undocumented) functions have been removed
from the threads libraries:
_threadstart_CMM() and _threadstart_CMM_cog()
_threadstart_LMM() and _threadstart_LMM_cog()
_threadstart_NMM() and _threadstart_NMM_cog()
The following functions should be used instead:
_threaded_cogstart_CMM_cog()
_threaded_cogstart_LMM_cog()
_threaded_cogstart_NMM_cog()
Affected the Propeller 1 and Propeller 2 under Catalyst, Windows and Linux.
4. The example of using the Catalina Parallelizer with the Propeller version
of gcc was incorrectly modified to include Catalina's "prop2.h" rather
than gcc's "propeller2.h", which prevented it compiling.
Catalina 7.7 has been released on SourceForge and also now on GitHub.
There are very few functional changes in this release (none to Catalina itself). This release is primarily to facilitate the process of making GitHub one of Catalina's release platforms. To do this, changes had to be made to allow for the removal of various binary utilities that were included in previous releases, as well as updating various Makefiles and scripts to eliminate dependencies on those utilities.
For the next few releases at least, both SourceForge and GitHub will be used to host Catalina releases - SoureForge will continue to host the usual combined binary and source releases, and GitHub will host source only releases. But to avoid the need for Windows users to build the release from source, the Catalina binaries included in the Windows binary releases will also be available on GitHub as binary assets associated with each GitHub release. Other binary utilities (such as GNU make and other GNU utilities) are still included in the Windows release but installing them this way is now deprecated, and this option will be removed altogether in a future release.
Here is an extract of the new README.md file that discusses this in more detail:
Setting up Catalina
Windows using Setup
If you used a Windows Installer package (e.g. Catalina_7.7_Setup.exe) to install Catalina and accepted the recommended settings, the Windows Start Menu should contain the following shortcuts:
Catalina Command Line : Start a Windows command shell (i.e. cmd.exe) in the Catalina directory and set up the Catalina environment variables and paths.
Catalina Geany : Start the Catalina version of the Geany Integrated Development Environment.
Documents : Contains links to Catalina tutorial and reference documents.
Note that the Windows Installer may offers the option of installing Make and additional GNU utilities. While installing make is recommended, this method of doing so is deprecated, and may be removed from future versions of the Windows Installer. See the Catalina and make section of this document (below) for details on the recommended way to install make.
Windows Manual install
If you did not use the Windows Setup package but either cloned the Git repository or downloaded it as a compressed file and then uncompressed it, then the distribution will includes all sources, but not the Windows binaries.
If you do not want to rebuild Catalina from source (which is a complex process under Windows) then a separate package of precompiled binaries will be available suitable for Windows 10 or Windows 11 as an asset associated with the Git release. Download the asset and uncompress it into Catalina's bin directory.
Open a Windows command shell (i.e. cmd.exe), then cd to the Catalina installation folder. Then execute the command use_catalina to set up the Catalina environment variables and paths.
The use_catalina script will also check whether the Catalina binaries have been installed, and also whether there is a version of make installed. While make is not required to use the Catalina compiler, it is required to build Catalina or Catalyst, and also to execute the various build_all scripts in the release. See the section called Catalina and make below.
Then you can either use Catalina directly from the command line, or use the command catalina_geany to start the Catalina Geany Integrated Deveopment Environment. The Catalina documents are available in the Documents folder.
To create Windows Start Menu entries, open a Windows command shell (i.e. cmd.exe) with Administrator privileges, then cd to the Catalina installation folder. Then execute the command catalina_shortcuts optionally specifying the name of the Start Menu entry to create (in quotes). For example: catalina_shortcuts "Catalina 7.7". If you do not specify a name then "Catalina" will be used. Then you can start either a Catalina Command Line or Catalina Geany from the Windows Start Menu. Also, the Start Menu will contain links to the Catalina Documents.
Linux Precompiled Package
If you downloaded a Linux release package (e.g. Catalina_7.7_Linux.tar.gz) then the package will contain executables built for a recent Ubuntu release (currently Ubuntu version 23). Simply open a Terminal window, cd to the directory where you installed Catalina and enter (note the back quotes):
export LCCDIR=`pwd`
source use_catalina
The use_catalina script will also check whether the Catalina binaries have been installed, and also whether there is a version of make installed. While make is not required to use the Catalina compiler, it is required to build Catalina or Catalyst, and also to execute the various build_all scripts in the release. See the section called Catalina and make below.
Then you can either use Catalina directly from the command-line or enter catalina_geany to use the Catalina Geany IDE. However, if the pre-built Catalina executables do not work on your Linux installation, refer to the next section on installing it manually.
Linux Manual Install
If you cloned the Git repository or downloaded it as a compressed file and then uncompressed it then you must always rebuild Catalina from source. Follow the Linux instructions in the BUILD.TXT document in the main Catalina installation folder to build Catalina. This document also contains instructions on setting Catalina up for use.
Catalina and make
While Catalina does not requiremake to just use the C compiler, it is required to rebuild Catalina, Geany and Catalyst from source, and is also used by various Catalina scripts such as the build_all scripts in the demos directory.
The use_catalina script will warn if make is not installed.
Linux will usually have make installed. If it does not, use the appropriate package manager to install it.
Windows does not have a native version of make. The GNU version can be installed either by installing Cygwin, MinGW, MSYS2 or GNuWin32, but the recommended method is to execute the following in a Command Line window (requires an active internet connection):
winget install ezwinports.make
Note that this installation only has to be done once, but that the current Command Line window will have to be closed and a new one opened for the installation to take effect.
Just a quick note for Linux users - I found and have fixed the bugs that were preventing awka from working when compiled as a 64 bit program under Linux (awka is used internally by various Catalina utilities).
Because everything can now be built as 64 bits, this simplifies the Linux build process (see the updated BUILD.TXT), and I had also had complaints from some Linux users because they didn't like having to install 32 bit libraries on 64 bit Linux distributions just to build Catalina. No more!
The changes will be in the next release, but you can get them now from GitHub (tag v7.7.3). This is why I wanted Catalina on GitHub - to make it easier to push out minor changes without the overhead of a release. So far, so good
While messing around with awka (see previous post) I also managed to make it compile under MSYS2. This means Cygwin is no longer required to build Catalina on Windows. This simplifies the Windows build process significantly.
However, unlike Linux users, Windows users don't need to build Catalina from source, so there is no urgency on releasing this - it will just be incorporated in the next release.
There are no functional changes in this release. The main reason for it is that Catalina is moving to being a pure 64 bit application on both Windows and Linux, but in this release the Windows programs are still compiled as 32 bit applications because I had some problems with the Windows 64 bit version of MSYS2. If I have to support Catalina before the 64 bit Windows release is ready (which I hope will be the next release), this will make it easier.
Here is the relevant extract from the README.TXT
RELEASE 7.8
New Functionality
-----------------
1. None. This release updates only the build process and packaging, and
improves GitHub support.
Other Changes
-------------
1. Remove binaries from GitHub repository. They will remain in the Windows
and Linux source/binary distributions but on GitHub they will exist
as binary assets associated with each release.
2. Awka now compiles using 64 bit gcc, on both Windows and Linux.
3. Catalina now compiles using 64 bit gcc on both Windows and Linux.
On Linux, Catalina is now built as a 64 bit application, but on Windows
it is currently still built as a 32 bit application until further testing
is completed. This will probably change in the next release.
4. Catalina no longer needs Cygwin to be installed to build under Windows.
Only MINGW MSYS2 is required. However, the Cygwin DLL is still distributed
with Catalina as it is still required by the Comms VT100 terminal emulator
program.
Catalina now defaults to 64 bit compilation on both Windows and Linux. The changes are available on GitHub (tag v7.8.1). There are no functional changes, so I don't see any need to make it a formal release.
Now that that's done, I can get back to more interesting things than trying to placate an annoyingly overzealous gcc!
Okay! Now that boring 64 bit housekeeping stuff is out of the way, it's time for something a bit more interesting ...
I love Lua, but it is primarily intended to be a scripting language to extend C, not replace C - but C programs that also incorporate Lua can easily end up being too large and too slow to be useful on a micro controller like the Propeller. And this includes Catalina's own multi-processing version of Lua - great in theory, but to provide the multi-processing capabilities, it has to execute from Hub RAM - which limits the sizes of programs it can be used for on the Propeller 2, and prevents it being used on the Propeller 1 at all.
I really wanted an alternative. Specifically, I wanted to be able to write programs where the Lua code in the program executes entirely from XMM RAM, while the C code in the same program executes from Hub RAM and has full access to the Propeller's multi-processing capabilities, and also to Catalina's multi-threading capabilities. Which - not coincidentally - is precisely what catapult was designed to facilitate.
So here is a demo of using Catapult with Lua ...
/******************************************************************************
* Using Catapult with Lua. *
* *
* This program demonstrates executing a Lua primary function from XMM RAM *
* while simultaneously executing a multithreaded C secondary function from *
* Hub RAM, and having the two interact via a shared C data structure. *
* *
* Only one Lua function can be executing, but as many additional secondary *
* C functions can be executing simultaneously as there are free cogs. *
* *
* Compile this program with a command like: *
* *
* catapult lua_p2.c *
* *
* Before loading, execute 'build_utilities' to build the appropriate XMM *
* loader (for a P2_EDGE with PSRAM and an 8K cache), and then load and *
* execute the program with a command like: *
* *
* payload -o2 -i xmm lua_p2 *
* *
* Note that if you modify the program, you will probably have to modify the *
* address specified in the secondary pragma - but the program will tell you *
* what address to use when you compile and execute it. *
* *
******************************************************************************/
#pragma catapult common options(-p2 -C P2_EDGE -C SIMPLE -lmc)
#include <catapult.h>
#include <stdlib.h>
#include <prop.h>
/*
* define a type to be used for exchanging information between
* the Lua primary Lua function and the secondary C function ...
*/
typedef struct shared_data {
int go;
} shared_data_t;
/******************************************************************************
* *
* The secondary function - executes multithreaded C *
* *
******************************************************************************/
#pragma catapult secondary hub_function(shared_data_t) mode(NMM) address(0x76468) stack(8000) options(-lci -lthreads -C NO_FLOAT)
/*
* include Catalina multi-threading:
*/
#include <threads.h>
/*
* define how many threads we want:
*/
#define THREAD_COUNT 10
/*
* define the stack size each thread needs (since this number
* depends on the function executed by the thread, the smallest
* possible stack size has to be established by trial and error):
*/
#define STACK_SIZE (MIN_THREAD_STACK_SIZE + 55)
static int ping;
/*
* function : a function that can be executed as a thread
*/
int function(int me, char *not_used[]) {
while (1) {
if (ping == me) {
// print our id
t_printf("%d ", (unsigned)me);
ping = 0;
}
else {
// nothing to do, so yield
_thread_yield();
}
}
return 0;
}
/*
* hub_function : start THREAD_COUNT threads, then ping each one in turn
*/
void hub_function(shared_data_t *s) {
int i = 0;
void *thread_id;
unsigned long stacks[STACK_SIZE * THREAD_COUNT];
// wait till we are told to go (by Lua!) before proceeding
while (s->go == 0);
t_printf("... C executes multiple threads using cog %d\n\n", _cogid());
// assign a lock to avoid context switch contention
_thread_set_lock(_locknew());
// start instances of function until we have started THREAD_COUNT of them
for (i = 1; i <= THREAD_COUNT; i++) {
thread_id = _thread_start(&function, &stacks[STACK_SIZE*i], i, NULL);
t_printf("thread %d ", i);
if (thread_id == (void *)0) {
t_printf(" failed to start\n");
while (1) { };
}
else {
t_printf(" started, id = %d\n", (unsigned)thread_id);
}
}
// now loop forever, pinging each thread in turn
while (1) {
t_printf("\n\nPress a key to ping all threads\n");
k_wait();
for (i = 1; i <= THREAD_COUNT; i++) {
t_printf("%d:", i);
// ping the thread
ping = i;
// wait till thread responds
while (ping) {
// nothing to do, so yield
_thread_yield();
};
}
}
}
/******************************************************************************
* *
* The primary function - executes Lua *
* *
******************************************************************************/
#pragma catapult primary binary(lua_p2) mode(XMM) options(-lcx -W-w -C CLOCK -llua linit.c)
#include <lua.h>
#include <lualib.h>
#include <lauxlib.h>
/*
* Define the Lua code - here it is defined using a string, but it could
* just as easily be loaded from a file.
*
* The function "flash" prints a friendly message, then initiates the
* multi-threading by setting the "go" flag in the shared data, then
* flashes the specified LED just to indicate it is still executing ...
*/
char *lua_code =
" LED = 38; --[[ suitable for P2 EDGE ]]"
" function flash(shared)"
" print('Hello World (from Lua!)\\n');"
" print('Lua will flash LED ' .. LED .. "
" ' using cog ' .. cogid() .. ' while ...');"
" go(shared, 1);"
" x = 0;"
" while (true) do"
" propeller.setpin(LED, x);"
" x = x ~ 1;"
" propeller.msleep(500);"
" end"
" end";
/*
* define a C function that can be called from Lua to return the current cogid
*/
static int cogid(lua_State *L) {
lua_pushinteger(L, _cogid());
return 1;
}
/*
* define a C function that can be called from Lua to access the go field
* of the shared data - it accepts either one or two parameters - the first
* (required) is the address of the shared data and the second (optional)
* is a new value for go. It returns the old value of go ...
*/
static int go(lua_State *L) {
int old_go = 0;
shared_data_t *s;
int n = lua_gettop(L); // number of arguments
if (n == 2) {
// optional argument specified
register lua_Integer new_go = luaL_checkinteger(L, 2);
s = (shared_data_t *)lua_topointer(L, -2);
old_go = s->go;
s->go = new_go;
lua_pop(L, 2);
lua_pushinteger(L, old_go);
return 1;
}
else if (n == 1) {
// optional argument not specified
s = (shared_data_t *)lua_topointer(L, -1);
old_go = s->go;
lua_pop(L, 1);
lua_pushinteger(L, old_go);
return 1;
}
else {
// incorrect number of arguments
return 0;
}
}
void main(int argc, char *argv[]) {
shared_data_t shared = { 0 };
int cog;
int result;
// creat a new Lua state
lua_State *L = luaL_newstate();
// open the standard Lua libraries
luaL_openlibs(L);
// start the multi-threading program - it will wait
// for the 'go' field of the shared_data argument to
// be set before starting the threads
RESERVE_AND_START(hub_function, shared, ANY_COG, cog);
t_printf("Press a key to start Lua ...\n\n");
k_wait();
// load the Lua code and execute it
result = luaL_loadstring(L, lua_code) || lua_pcall(L, 0, 0 ,0);
if (result == LUA_OK) {
// set up the C 'cogid' function, so it can can be
// called from Lua to return the current cog id
lua_pushcfunction(L, cogid);
lua_setglobal(L, "cogid");
// set up the C 'go' function, so it can can be
// called from Lua to start the multi-threading
lua_pushcfunction(L, go);
lua_setglobal(L, "go");
// call Lua "flash" - i.e. flash(&shared)
lua_getglobal(L, "flash");
lua_pushlightuserdata(L, &shared);
result = lua_pcall(L, 1, 1, 0);
}
if (result != LUA_OK) {
// something went wrong
t_printf("Unexpected result (%d)\n", result);
}
// not necessary here, but it is good practice to tidy up
lua_close(L);
}
In the above program the primary function executes Lua code from XMM RAM while the secondary function executes C code from Hub RAM with full Propeller functionality and at full Propeller speeds. The program could have as many secondary C functions as there are available cogs, and they can all communicate via shared Hub RAM.
Lua can therefore be used for what it was designed for - i.e. configuring and customizing C programs ... plus it can provide additional functionality that is not particularly time critical or which would be too tedious to write in C. And C can do all the rest. The Lua code does not even need to be compiled into the program - typically it would not be, but would instead be loaded from a file at run time so that the application does not need to be re-compiled to be configured, customized or even extended - precisely what Lua was intended for!
The above program is for a Propeller 2 (specifically the P2-EC32MB). But an almost identical program can be executed on a Propeller 1 with sufficient XMM RAM (e.g. the C3). Both programs are included in the attached ZIP file.
These programs will be included as demos in the next release, but they can be compiled and executed with the current Catalina release - just ensure that a suitable version of linit.c is in the same folder when the programs are compiled (a suitable linit.c can be found in demos\lua, and is also included in the attached ZIP file).
See the programs themselves for more details on how to build and execute them.
Ross.
EDIT: The Propeller 1 version of the program said it needed an 8k cache, but in fact it needs a 1k cache - zip file updated.
EDIT: The demo programs were simplified - using a lock pool is not necessary in this case. New versions attached.
The Windows release of Catalina 7.8 was missing some DLLs. This may have prevented payload from executing - it could result in a Windows "The Application Was Unable to Start (0xc000007b)" message, or a message about a missing DLL.
If payload works for you it means you already have the required DLLs installed and you don't need to do anything. If not, you can either download the new installer and re-install Catalina, or just download the missing DLLs from the Github asset and add them to your existing Catalina "bin" directory.
While working on some 'helper' functions to simplify the integration of the various parts of a multi-model catapult program (which allows some parts of the program to execute from XMM RAM while other parts execute from Hub RAM), it occurred to me that the Catalina Registry already offered an ideal mechanism for doing this, since it was specifically designed for use in an analogous situation - i.e. where some parts of a program execute from Hub RAM while other parts execute from Cog RAM - i.e. as plugins).
Silly of me not to realize this earlier
So, here is an example catapult program where the primary program is a Server that executes from XMM RAM, offering services that can be called as C functions from one or more secondary Client programs executing from Hub RAM:
/******************************************************************************
* Using Catapult to build a C server. *
* *
* This program demonstrates executing a primary server from XMM RAM that *
* dispatches service calls made from one or more C secondary clients to *
* C functions implemented in the primary server (i.e. from XMM RAM). *
* *
* The program uses the Catalina Registry as the mechanism, which enables *
* the server to offer multiple services, and also adds lock protection to *
* the services. Using the registry supports only a limited number of *
* parameter profiles - i.e. those that are required to implement Catalina *
* plugins (see func_1 to func_5). However, the basic "short" service can *
* also be used to pass a pointer to the shared data structure as the *
* parameter, which allows for arbitary data to be exchanged between client *
* and server, as usual for catapult programs (see func_6). *
* *
* This version is for a Propeller 2 P2_EDGE with PSRAM (i.e. a P2-EC32MB), *
* but it could be easily modified to suit any Propeller 1 or 2 which has *
* sufficient XMM RAM by modifying the catapult pragmas appropriately. *
* *
* Compile this program with a command like: *
* *
* catapult serve_c.c *
* *
* Before loading, execute 'build_utilities' to build the appropriate XMM *
* loader (for a P2_EDGE with PSRAM and an 8K cache), and then load and *
* execute the program with a command like: *
* *
* payload -o2 -i xmm serve_c *
* *
* Note that if you modify the program, you may have to modify the address *
* specified in the secondary pragma - but the program will tell you what *
* address to use when you compile and execute it. *
* *
******************************************************************************/
#pragma catapult common options(-W-w -p2 -C P2_EDGE -C SIMPLE -lc -lmc service.c)
#include <stdlib.h>
#include "service.h"
#include "plugin.h"
/*
* define global service identifiers for our services
* (can be any unused service ids, but must be unique)
*/
#define MY_SVC_1 (SVC_RESERVED+1)
#define MY_SVC_2 (SVC_RESERVED+2)
#define MY_SVC_3 (SVC_RESERVED+3)
#define MY_SVC_4 (SVC_RESERVED+4)
#define MY_SVC_5 (SVC_RESERVED+5)
#define MY_SVC_6 (SVC_RESERVED+6)
/*
* define a type to be used for exchanging data between
* the primary and secondary functions
*/
typedef struct shared_data {
int ready;
int start;
int data;
} shared_data_t;
/******************************************************************************
* *
* The secondary client - calls services provided by the primary server *
* *
******************************************************************************/
#pragma catapult secondary hub_client(shared_data_t) address(0x770AC) mode(NMM) stack(500)
/*
* define proxy functions, which use Catalina's pre-defined
* service functions to call the real server functions via
* the Registry
*/
int func_1(long l) {
return _short_service(MY_SVC_1, l);
}
int func_2(long l) {
return _long_service(MY_SVC_2, l);
}
int func_3(long l1, long l2) {
return _long_service_2(MY_SVC_3, l1, l2);
}
float func_4(float f1, float f2) {
return _float_service(MY_SVC_4, f1, f2);
}
long func_5(float f1, float f2) {
return _long_float_service(MY_SVC_5, f1, f2);
}
int func_6(shared_data_t *shared) {
return _short_service(MY_SVC_6, (long)shared);
}
/*
* hub_client : call the services offered by the primary function
* by calling the proxy functions defined above
*/
void hub_client(shared_data_t *s) {
int i = 0;
long l = 0;
float f = 0.0;
t_printf("Client ready - press a key to start the server ...\n");
k_wait();
// tell the primary server we are ready
s->ready = 1;
// wait till we are told to start by the primary before proceeding
while (s->start == 0);
t_printf("The secondary client calls from cog %d ...\n", _cogid());
t_printf("... to the primary server on cog %d\n", _locate_plugin(LMM_SVR));
// call the services using the proxy functions
while(1) {
t_printf("\nPress a key to continue ...\n");
k_wait();
i = func_1(1000);
t_printf("result of float_1 = %d\n", i);
i = func_2(2000);
t_printf("result of float_2 = %d\n", i);
i = func_3(3000, 4000);
t_printf("result of float_3 = %d\n", i);
f = func_4(5000.0, 6000.0);
t_printf("result of float_4 = %f\n", f);
l = func_5(7000.0, 8000.0);
t_printf("result of float_5 = %ld\n", l);
i = func_6(s);
t_printf("result of float_6 = %d\n", i);
}
}
/******************************************************************************
* *
* The primary server - executes a C dispatcher, dispatching calls *
* to the C functions specified in my_service_list *
* *
******************************************************************************/
#pragma catapult primary binary(serve_c) mode(XMM) options(dispatch_c.c)
/*
* define our services as C functions
*/
int func_1(long l) { // example SHORT_SVC
t_printf("Hello, World (from C func_1!)\n");
t_printf("Args: l=%ld\n", l);
return l + 1;
}
int func_2(long l) { // example LONG_SVC
t_printf("Hello, World (from C func_2!)\n");
t_printf("Args: l=%ld\n", l);
return l + 2;
}
int func_3(long l1, long l2) { // example LONG_2_SVC
t_printf("Hello, World (from C func_3!)\n");
t_printf("Args: l1=%ld, l2=%ld\n", l1, l2);
return l1 + l2 + 3;
}
float func_4(float f1, float f2) { // example FLOAT_SVC
t_printf("Hello, World (from C func_4!)\n");
t_printf("Args: f1=%f, f2=%f\n", f1, f2);
return f1 + f2 + 4;
}
long func_5(float f1, float f2) { // example LONG_FLOAT_SVC
t_printf("Hello, World (from C func_5!)\n");
t_printf("Args: f1=%f, f2=%f\n", f1, f2);
return (long)(f1 + f2) + 5;
}
long func_6(shared_data_t *shared) { // example SHARED_SVC
t_printf("Hello, World (from C func_6!)\n");
shared->data++;
return shared->data;
}
/*
* define a list of services, parameter profiles and service identifiers
* - note that C services require the address of the functions, the names
* are for information only
*/
svc_list_t my_service_list = {
{"func_1", func_1, SHORT_SVC, MY_SVC_1},
{"func_2", func_2, LONG_SVC, MY_SVC_2},
{"func_3", func_3, LONG_2_SVC, MY_SVC_3},
{"func_4", func_4, FLOAT_SVC, MY_SVC_4},
{"func_5", func_5, LONG_FLOAT_SVC, MY_SVC_5},
{"func_6", func_6, SHARED_SVC, MY_SVC_6},
{"", NULL, 0, 0}
};
void main() {
shared_data_t shared = { 0, 0, 0 };
int cog;
// re-register this cog as a server (it will already
// be registered, but as a kernel, not a server)
_register_plugin(_cogid(), LMM_SVR);
//register the services, so clients can find them
register_services(_locknew(), my_service_list);
// load the secondary client - it will wait for the
// 'start' field of the shared_data argument to be set
RESERVE_AND_START(hub_client, shared, ANY_COG, cog);
// wait for the secondary to be ready (required if the
// secondary print messages, to prevent garbled output)
while (!shared.ready);
// start the secondary client
t_printf("Server ready - press a key to begin ...\n");
k_wait();
shared.start = 1;
// dispatch service calls
while(1) {
dispatch_C(my_service_list);
}
}
"Very nifty" I am sure I can hear you all say ... ... but wait - there's more ... the Server does not even have to be a C program.
Since we are now using the Catalina Registry as our call mechanism, the Server in the above program can simply be replaced - exactly as a plugin can - with the same services implemented using a different Server ... including a Server which offers the same services implemented as Lua functions executing from XMM RAM! Here is the new primary Server component only - the common component and the secondary Client component are exactly the same as above:
/******************************************************************************
* *
* The primary server - executes a Lua dispatcher, dispatching calls *
* to the Lua functions specified in my_service_list *
* *
******************************************************************************/
#pragma catapult primary binary(serve_lua) mode(XMM) options(-llua dispatch_lua.c linit.c)
#include <lua.h>
#include <lualib.h>
#include <lauxlib.h>
/*
* define our services as Lua functions
*/
char *lua_code =
" function func_1(l)" // example SHORT_SVC
" print('Hello World (from Lua func_1!)');"
" print('Args: l=' .. l);"
" return l + 1;"
" end"
""
" function func_2(l)" // example LONG_SVC
" print('Hello World (from Lua func_2!)');"
" print('Args: l=' .. l);"
" return l + 2;"
" end"
""
" function func_3(l1, l2)" // example LONG_2_SVC
" print('Hello World (from Lua func_3!)');"
" print('Args: l1=' .. l1 .. ', l2=' .. l2);"
" return l1 + l2 + 3;"
" end"
""
" function func_4(f1, f2)" // example FLOAT_SVC
" print('Hello World (from Lua func_4!)');"
" print('Args: f1=' .. f1 .. ', f2=' .. f2);"
" return f1 + f2 + 4;"
" end"
""
" function func_5(f1, f2)" // example LONG_FLOAT_SVC
" print('Hello World (from Lua func_5!)');"
" print('Args: f1=' .. f1 .. ', f2=' .. f2);"
" return f1 + f2 + 5;"
" end"
""
" function func_6(shared)" // example SHARED_SVC
" print('Hello World (from Lua func_6!)');"
" return data(shared);"
" end";
/*
* define a list of services, parameter profiles and service identifiers
* - note that Lua services require only the names of the functions, and
* their addresses should be NULL
*/
svc_list_t my_service_list = {
{"func_1", NULL, SHORT_SVC, MY_SVC_1},
{"func_2", NULL, LONG_SVC, MY_SVC_2},
{"func_3", NULL, LONG_2_SVC, MY_SVC_3},
{"func_4", NULL, FLOAT_SVC, MY_SVC_4},
{"func_5", NULL, LONG_FLOAT_SVC, MY_SVC_5},
{"func_6", NULL, SHARED_SVC, MY_SVC_6},
{"", NULL, 0, 0}
};
/*
* define a C function that can be called from Lua to access the shared data
* - it accepts one parameter, which is the address of the shared data, and
* then increments and returns the value of the shared data "data" field,
* just as a demo (it is used in Lua func_6, defined above).
*/
static int data(lua_State *L) {
shared_data_t *s;
if (lua_gettop(L) == 1) {
s = (shared_data_t *)lua_topointer(L, -1);
lua_pop(L, 1);
s->data++;
lua_pushinteger(L, s->data);
return 1;
}
else {
// incorrect number of arguments
return luaL_error(L, "expected 1 argument");
}
}
void main() {
shared_data_t shared = { 0, 0, 0 };
int cog;
int result;
lua_State *L;
// re-register this cog as a server (it will already
// be registered, but as a kernel, not a server)
_register_plugin(_cogid(), LMM_SVR);
//register the services, so clients can find them
register_services(_locknew(), my_service_list);
// load the secondary client - it will wait for the
// 'start' field of the shared_data argument to be set
RESERVE_AND_START(hub_client, shared, ANY_COG, cog);
// create a new Lua state and open the standard Lua libraries
L = luaL_newstate();
luaL_openlibs(L);
// load the Lua code and execute it so Lua knows about the
// functions that can be called
result = luaL_loadstring(L, lua_code) || lua_pcall(L, 0, 0 ,0);
if (result != LUA_OK) {
// something went wrong
t_printf("Unexpected result (%d)\n", result);
}
// tell Lua about the 'data' function, so it can can be called
// from Lua to demonstrate Lua accessing the shared data
lua_pushcfunction(L, data);
lua_setglobal(L, "data");
// wait for the secondary to be ready (required if the
// secondary print messages, to prevent garbled output)
while (!shared.ready);
// start the secondary client
t_printf("Server ready - press a key to begin ...\n");
k_wait();
shared.start = 1;
// dispatch service calls
while(1) {
dispatch_Lua(L, my_service_list);
}
}
Since we are now using the Catalina Registry, the Clients no longer care how the functions offered by the Server are implemented - and catapult makes it trivial to swap a C Server for a Lua Server.
Both files, plus some necessary support files, are attached. They can be compiled with the current release of Catalina using the commands:
catapult serve_c.c
and catapult serve_lua.c
The Catalina Registry also provides a locking mechanism that protects the Server - this was already implemented as it was necessary to protect plugins - so the secondary Client programs can also be multi-threaded.
Getting very close to the best of all possible worlds now!
I have updated the "catapult_server.zip" file attached to the previous post, for two reasons:
There was a bug that may have prevented the server from starting - the program would print a few messages and then the server would hang when it tried to service calls from the client.
The new version has some minor name changes so that it will continue to work when I release the next version of Catalina, which will have some of the same functionality built into the library (so that only the main catapult program file is needed).
There are only minor functional changes in this release. The main reason for it is that I fixed a significant issue with dynamic memory allocation functions (malloc(), free() etc) that meant programs that used them could run out of heap space even though there was still plenty available.
Here is the relevant extract from the README.TXT
RELEASE 7.9
New Functionality
-----------------
1. Catalina, catbind and bcc all have a new command line option. The new
option (-H) accepts an address parameter and can be used to specify the
maximum address that will be used by the heap. In all memory modes except
LARGE mode, the heap and stack share Hub RAM, with the heap growing upward
from the highest used low Hub address, and the stack growing downward from
the lowest used high hub address. This means they can eventually overlap,
with potentially disastrous consequences.
The -H option allows this to be avoided, by limiting the growth of the
heap. The program may run out of heap space, but this can be detected
(e.g. sbrk() or malloc() will return an error if there is no more space).
The required amount of stack space can be determined by printing the
current stack pointer at various suitable points in the program - below
is a macro that uses inline PASM to do this, and a trivial program that
uses it. This program will work in any memory model on any Propeller:
// this handy macro returns the current stack pointer
// in any memory model on the P1 or P2 ...
#define SP PASM( \
"#ifdef COMPACT\n" \
" word I16B_PASM\n" \
"#endif\n" \
" alignl\n" \
" mov r0, SP\n")
void main() {
printf("SP=0x%06X\n", SP);
while(1);
}
Suppose on a Propeller 1 it was known that the stack could grow down to
0x6000 - then it would be appropriate to specify -H 0x6000 to prevent the
heap ever growing large enough to overwrite the stack. The parameter can
be specified as decimal (including an optional 'k' or 'm' suffix) or as
hexadecimal (using the format $XXXXXX or 0xXXXXXX). For example, to ensure
the heap never grows above 24k, leaving the top 8k for buffers and
stack space, use a command like:
catalina prog.c -lc -H 24576
or
catalina prog.c -lc -H 24k
or
catalina prog.c -lc -H 0x6000
The -H option can be used on the Propeller 1 or 2. In all modes except
LARGE mode the address refers to a Hub address. It can also be used in
LARGE mode, where the heap is in XMM RAM, but the address refers to an
XMM RAM address. This could be used (for example) to reserve an upper
area of XMM RAM for other uses, such as for a buffer. However, note that
the start address of the XMM RAM can vary from platform to platform, so
check the XMM_RW_BASE_ADDRESS in the various platform configuration files.
2. New client/server support functions have been added to the libraries, and
several new catapult demos have been added, to demonstrate them - see the
document "Getting Started with Catapult" for details. The new demos in the
demos/catapult folder are:
srv_c_p1.c & svr_c_p2.c -- demonstrate a C client with a C server.
srv_l_p1.c & svr_l_p2.c -- demonstrate a C client with a Lua server.
The above demo programs all use the library functions and a dispatcher
which enables client/server programs to interact using the functionality
that Catalina uses to interact between C programs and plugins. However,
this is limited to service profiles that match the existing ones used for
plugins. If these profiles are not sufficient, new profiles can be created
along with a custom dispatcher - a demonstration of this is given in the
folder demos/catapult/custom folder, using programs very similar to those
above, but extended to include one new custom service profile.
Other Changes
-------------
1. An issue first noted in Catalina 7.6.1 was not fixed in all the affected
kernels. In kernels without sufficient space to implement the basic
floating point operations (+, -, *, / etc), an 8 byte data block is used
to transfer information between the kernel and the floating point plugin.
In some kernels this block was not being set up correctly, leading to the
use of the 8 bytes starting at memory location 0 as the data block, which
would overwrite the clock values on the Propeller 1, leading to issues
with some functions that used timers. The specific kernels affected were
the CMM and XMM kernels on the Propeller 1, and the XMM Kernel on the
Propeller 2.
2. Catalina was not checking if there was sufficient heap space left before
trying to allocate a new block of memory (e.g. via sbrk, malloc, calloc
or realloc). As a result, a program could end up unable to allocate
memory on the heap even though there was plenty of free space once the
heap became too fragmented, because the function intended to defragment
and consolidate the blocks on the free list was not being triggered as
expected. Affected the Propeller 1 and 2.
This issue affected programs that made a large number of dynamic memory
allocation and deallocation calls for small randomly sized blocks of
memory, because existing free blocks are re-used whenever possible,
breaking them up into smaller and smaller pieces, and allocating
a new block when the existing free blocks could not satisfy a request.
The Catalina demo program most affected by this bug was Dumbo Basic,
which allocates and then frees a very large number of small blocks when
performing string operations, which meant the heap very quickly ended up
highly fragmented. For example, the 'eliza.bas' program would stop
responding if a long sentence was entered.
This issue has been fixed by (1) adding automatic detection of the
maximum heap size, so that the defragment function is triggered when
heap space is exhausted - this solves the problem for LARGE mode
programs (e.g. when compiling Dumbo Basic for the P1 in LARGE mode),
and (2) adding a new command line option (-H) which can be used in
any mode to specify the maximum heap address to be used (e.g. when
compiling Dumbo Basic in NATIVE mode for the P2).
Here's a pretty thing ... well, it is if you like this kind of thing, which I do ...
Attached is a preview of what's coming in Catalina 8.0. If (like me) you primarily work on the command line (because it's so much faster and easier than any IDE) the lack of a decent command line in Catalina has been a sore point which has finally led me to spend some time improving it.
Here's the blurb ...
Enhanced Catalyst and Lua command line processing
=================================================
This folder contains a pre-release demo of Catalyst 8.0 and Lua, both with
enhanced command line editing and history capabilities.
This enhanced functionality will be supported in Lua on the Propeller 1 and
2, but in Catalyst only on the Propeller 2 (the Propeller 1 lacks suffiicent
space). The demo versions in this folder is specifically for a Propeller 2,
and should execute on just about any Propeller 2 platform, including the P2
EDGE and P2 EVAL boards.
Copy the contents of this zip file to an SD card, insert it into the
Propeller and set the Propeller to boot from the SD. Then use a VT100
compatible terminal to interact with the Propeller. You can use Catalina's
comms.exe, PuTTY, or any other VT100 terminal emulator. You can also use
the enhanced version of payload (8.0) included in this folder. Note that
prior versions of payload (i.e. 7.9 or earlier) will not work - version 8.0
has some additional VT100 emulation capabilities required to support the
enhanced capabilities.
Note that you may need to reboot the Propeller after starting the terminal
emulator.
To use payload, specify the port the baud rate and options '-i -q2' - for
example, if your P2 is on port 8:
payload -p8 -b230400 -i -q2
To use Catalina's comms program, in a Catalina command line window enter
either:
comms /baud=230400 /com=8 /mode=vt100
or just:
vt100 8
To use PuTTY or another VT100 terminal emulator, the relevant communication
and terminal emulation parameters will need to be set up appropriately. See
the terminal emulator documentation.
The enhanced functionality in both Catalyst and Lua is mainly implemented
by the 'linenoise' line editor, which has now been incorporated into the
Catalina libraries, and compiled into the Catalyst and Lua binaries in the
demo.
The new functionality can be summarized by the following list of enhanced
key functions:
LEFT ARROW (or CTRL B) : move cursor left
RIGHT ARROW (or CTRL F) : move cursor right
UP ARROW (or CTRL P) : previous command in history
DOWN ARROW (or CTRL N) : next command in history
HOME (or CTRL A) : move cursor to start of line
END (or CTRL E) : move cursor to end of line
CTRL U : clear entire line
CTRL K : clear from cursor to end of line
CTRL L : clear screen
CTRL W : clear previous word
CTRL T : swap current and previous characters
CTRL C : exit
CTRL D : at start of line means exit (otherwise delete)
TAB : command completion
Try any of the above at the Catalyst prompt. Note that the Catalyst command
completion functionality is currently implemented as a simple hardcoded list
of all Catalyst commands. This is because interrogating the SD card for the
whole directory structure prior to each command would be too expensive. This
means that Catlyst may suggest commands that are not actually present in the
demo (e.g. entering 'i<TAB>' will suggest both 'if' and 'ilua' even though
only 'ilua' is present in the demo folder).
In Lua, you will need to use the 'ilua' program provided, because the new
functionality is not implemented in the basic 'lua' interactive program.
Instead of just starting 'lua', start the new 'ilua' program either by
entering:
lua ilua.lua
or just:
ilua
You will see a helpful message (implemented in help.lua, which can be edited
to amend or remove it if required). Try any of the above enhanced key
functions at the ilua prompt.
Note that in both cases the command history is stored in a file on the SD
card ('CATALYST.HST' for Catalyst, and '_ILUA.HST' for Lua) and can be
examined using any text editor (the vi editor is included in the demo).
The Catalyst history is saved before each command is executed, but the Lua
history is saved only on exit from Lua. If the Propeller is reset, the
history for the current 'ilua' session will be lost, but the history of
previous 'ilua' sessions will be intact.
For the purposes of this demo, the Catalyst prompt has been changed to "? "
to differentiate it from the Lua prompt "> ". This is done by setting the
value of PROMPT in the file 'CATALYST.ENV' and can be edited or removed if
required.
Ross.
EDIT: How odd. I was just about to delete the files attached to this post since Catalina 8.0 has now been released which superseded them - but there are no longer any binaries attached. I wonder what happened to them?
A small correction to the previous post. I said ...
Note that the Catalyst command completion functionality is currently implemented as a simple hardcoded list of all Catalyst commands. This is because interrogating the SD card for the whole directory structure prior to each command would be too expensive.
This is true in the demo, but it turns out I was worrying needlessly. The self-hosted version of Catalina puts over 8,000 files on the SD card, and I was concerned this this would take too long to process to allow for arbitrary completion of file names - but of course it only has to process one folder at a time, not the whole SD card. I just wrote some code to try it and it turns out to be easily fast enough.
So I won't need to use a hard coded list - release 8.0 will have full command and file name completion, reflecting what is actually on the SD Card.
The main change is the inclusion of the enhanced command line capabilities in both Catalyst and Lua. I had not intended to release 8.0 just yet, but I noticed that the GitHub release of 7.9 had the wrong binary executable files attached (it had the correct sources, but some of the executables were from 7.8). It was about the same amount of work to re-install and rebuild 7.9 as to just release 8.0. The SourceForge release of 7.9 was ok.
Here is the relevant extract from the README.TXT
RELEASE 8.0
New Functionality
-----------------
1. On the Propeller 2, Catalyst now incorporates the 'linenoise' command
line library, which provides enhanced command line editing functionality.
This is implemented as various keyboard shortcuts:
LEFT ARROW (or CTRL B) : move cursor left
RIGHT ARROW (or CTRL F) : move cursor right
UP ARROW (or CTRL P) : previous command in history
DOWN ARROW (or CTRL N) : next command in history
HOME (or CTRL A) : move cursor to start of line
END (or CTRL E) : move cursor to end of line
CTRL U : clear entire line
CTRL K : clear from cursor to end of line
CTRL L : clear screen
CTRL W : clear previous word
CTRL T : swap current and previous characters
CTRL C : exit
CTRL D : at start of line means exit
(otherwise it means delete)
TAB : command or file name completion
2. An enhanced Lua REPL (Read Eval Print Loop) based on 'lua-repl' has been
added to Catalyst, which uses the 'linenoise' command line editing library.
While the 'linenoise' library is now built into Lua, it is not loaded
by the default interactive program (i.e. 'lua'). This is done to save
space, since that program is primarily now used to execute Lua scripts.
Instead, 'lua-repl' is used to load 'linenoise' and a few other plugins
in a new 'ilua.lua' script. So to start the new interactive Lua from the
Catalyst command line, just enter the command 'ilua'. This script provides
the same keyboard shortcuts as those described above for Catalyst.
Other Changes
-------------
1. Payload's interactive mode has had some modifications to make its VT100
terminal emulation more closely match a real VT100. One change is in the
implemetation of what CR and LF do. For payload to correctly display the
output of programs that send both a CR and LF, it is now recommended that
-q2 be used as the appropriate payload option (i.e. ignore the LF) whereas
previously the recommended option was -q1 (i.e. ignore the CR). Various
documents have been updated.
I seem to spend more time building releases than actually coding these days ...
The document Getting Started with Catalina contains some outdated information. The structure of local user-defined libraries changed in release 6.0 to accommodate the limitations of DOS 8.3 file names (to allow Catalina to run under Catalyst on the Propeller 2) but this document was not updated.
Point 3 in the section Compiling Multiple Files covers the option of creating and using a user-defined library. It should now read:
Each library requires its own subdirectory, which must all be in a subdirectory called lib - i.e. lib\name – in this case let’s assume we want to refer to the library using the name x. To do this we must create a subdirectory called lib\x, as follows: mkdir lib\x
Now enter the subdirectory. While not strictly necessary, this makes the next few commands much simpler to type (and also simpler to explain). To do this, enter: cd lib\x
Now compile the file my_func.c from the original directory, adding the –S command-line option. The –S option (note that this is a capital S) tells Catalina to compile the program, but not to try to link it or generate an executable output file – it will just leave the result as an assembly language file (with a .s extension).
The command to use is therefore: catalina –S ..\..\my_func.c
This produces a file called my_func.s. If you had other functions to add to the library, you would use a similar command for each function.
The last step in creating the library is to create a library index. The compiler expects each library to have an index called catalina.idx – this allows symbols within the library to be located and resolved by the Catalina Binder without having to laboriously re-process each file in the library every time.
The Catalina Binder (which is normally invoked silently for you by Catalina) is also the tool you use to do library management. To create an index of all symbols imported or exported by all the assembly language files in the current directory, enter the following command: catbind –i –e *.s –o catalina.idx
Now your library is complete. Go back to the original directory: cd ..\..
We are now ready to compile the main program, specifying that Catalina should look in the library we have just created to find a definition for any unresolved symbols – which we do by adding the option –lx to the Catalina command. So enter the command: catalina my_prog.c –lc –lx
The main enhancement is the inclusion of eLua - a new enhanced version of Lua that supports client/server programming, either on multiple cogs on the same propeller, or on up to nine 'clustered' propellers.
Here is the relevant extract from the README.TXT
Release 8.1
New Functionality
-----------------
1. A new Lua variant has been added - eLua supports client/server programs
where the client and server execute on the same Propeller, and it can
be extended by the inclusion of a new serial protocol (ALOHA) to support
clients and servers executing on different propellers. Client/server
programs are supported on the Propeller 2 only. See the document
'Aloha from Lua' for full details.
2. The "exec" command (i.e. exec.lua) has been extended to allow parameter
substitution anywhere in the executed script. Previously, parameter
substitution was performed only when "_n" was specified as an argument to
the exec command itself - this was replaced with the value of the nth
parameter to the current script. This is still supported, but in addition
"%n" can now be specified anywhere in the script and it will be substituted
with the current value of the nth parameter to the current script. In most
cases, "%n" can be used wherever "_n" was used, except where "_n" represents
the name of an environment variable and not a parameter, such as if it was
the name (but not the parameters) of an "if" command. For an example of
this, see the "loop2" script in the folder demos/catalyst/core, which is a
minor variant of the "loop" script.
The "exec" command is available on the Propeller 2 only.
3. A new "call" command has been added (i.e. call.lua) that allows a Catalyst
script to be called from within another Catalyst script, which resumes the
original script when the called script terminates.
The "call" command can also be used on the command line, where it behaves
effectively the same as the "exec" command - but when used WITHIN a script,
"call" and "exec" behave differently - when "exec" is used, the executed
script REPLACES the current script, whereas when "call" is used, the
currently executing script will be resumed once the called script
terminates. The "exec" command is still the mechanism to be used for
looping scripts, as demonstrated in the "loop" and "loop2" scripts.
For example, to execute the Catalyst script "script_2" from within the
script "script_1", passing it the original parameter 1 and "xxx" as
parameters, and then once that completes, call it again with the original
parameter 2 and "yyy" as parameters, the script "script_1" might contain
lines like:
call script_2 _1 xxx
call script_2 _2 yyy
If "exec" were used instead of "call", the second invocation of script_2
would never be executed because the first invocation would not return.
Note that calls cannot be nested - i.e. a script can be executed from the
command line that calls another script, but that script cannot then call
any further scripts.
To support the "call" capability, two additional lua scripts have been
provided (_save.lua and _restore.lua) which can save and restore the
current values of up to 23 parameters in a single environment variable:
_save : save parameters _0 .. _22 in a named environment variable
(_SAVED_PARAMS is the default name)
_restore : restore parameters _0 .. _22 from a named environment
variable (_SAVED_PARAMS is the default name)
The "call" command is available on the Propeller 2 only.
4. The self-hosted catalina command (i.e. catalina.lua) can now be executed
within a script. Previously, doing so would terminate the script.
5. A simple "echo" command has been added (i.e. echo.lua) which just echoes
its arguments. For example:
echo hello there!
or
echo %1 %2 %3 etc
6. New functions have been added to the Lua "propeller" module, which are
analogous to the corresponding C library functions:
cogid, clockfreq, clockmode, getcnt, muldiv64,
locknew, lockclr, lockset, lockret, locktry, lockrel
See the document "Lua on the Propeller 2 with Catalina" for details.
Note that locktry, lockrel and muldiv64 functions are implemented on the
Propeller 2 only.
7. All the HMI functions (e.g. k_get, k_ready ... t_char, t_geometry ...,
m_buttton, m_reset ...) have been removed from the Lua "propeller"
module and put in a separate "hmi" module. Also, t_string was missing
and has now been added. If these functions are now undefined and cause
runtime errors when called because they are nil, then they should be
renamed - e.g. from "propeller.k_new()" to "hmi.k_new()" etc.
All the Lua examples in the release have been updated accordingly.
8. A new example program (ex13.lua) has been added to demonstrate the use of
Propeller locks as another possible synchronization mechanism in the
Multi-Processing version of Lua (i.e. mlua or mluax). See the document
"Lua on the Propeller 2 with Catalina" for details.
9. The multi-processing example 10 (ex10.lua) has been updated to the LED
numbers appropriate for a P2 EDGE board rather than a P2 Evaluation
board. Also, the example that tries to execute the maximum number of Lua
threads (ex8.lua) has had the default reduced to 10 from 12 (which was
not achievable in some cases).
10. A new example program (args.lua) has been added to demonstrate how a Lua
script can process arguments whether executed from the command line, such
as:
args one 2.0 3
or
lua args.lua one 2.0 3
Or from within another Lua script, such as:
f = loadfile("args.lua");
f("one", 2.0, 3);
or
loadfile("args.lua")("one", 2.0, 3);
11. It is now possible to specify the pins and baud rates to be used for the
default serial port and the blackcat/blackbox port to be specified on
the Catalina command line using the -C command line option - e.g:
-C _RX_PIN=1
-C _TX_PIN=2
-C _BAUDRATE=115200
-C _BLACKCAT_RX_PIN=1
-C _BLACKCAT_TX_PIN=2
-C _BLACKCAT_BAUD=115200
12. There is a new library function that can be used to align the value of
the system break (aka sbrk). If aligning the system break is needed, this
function should be called before any memory allocation function (i.e.
malloc() etc). The function is:
/*
* _align_sbrk - align sbrk to (1<<align), add offset, and report
* the final value (using t_printf) if requested.
* For example:
*
* _align_sbrk(10,0,0); // align to next 1k boundary
*/
_align_sbrk(int align, int offset, report);
This is currently used on startup by Lua programs that can experience a
lockup when loading libraries - this appears to be a Lua bug to do with
the alignment of the system break. The following code resolve the bug in
all currently known instances:
/* align sbrk to 2k boundary - Lua needs this! */
_align_sbrk(11,0,0);
Other Changes
-------------
1. The Serial 2 plugin was not being initialized correctly when the first
function called was s2_rxcount(). In such a case the function would never
return. The s2_rxcount() function worked correctly as long as it was NOT
the first function called. Affected the Propeller 2 only.
2. The definition of the LOCKNEW macro in prop.h (aka propeller.h) may have
led to symbol "lock" being declared undefined because the macro used it
as a parameter to the _locknew() function which is neither required nor
used. Affected the Propeller 1 and Propeller 2.
3. The definition of the _lockrel() function in prop2.h (aka propeller2.h)
was incorrect - it defined the function as returning an int when in fact
it does not return any value. Affected the Propeller 2 only.
4. The filename completion function in Catalyst would complete commands
in the BIN directory without including the "/BIN/" prefix. If the intent
was just to execute the command this worked ok, but if the intent was to
process the file (e.g. if the command was a Lua script to be edited) then
the completed filename was not correct and the file would not be found.
Affected the Propeller 2 only.
5. The information in the 'Catalina Reference Manual (Propeller 2)' document
regarding the configuration of the 2 port and 8 port serial plugins and how
this may interact with the TTY and SIMPLE HMI options was out of date and
has been updated. Affected the Propeller 2 only.
6. The size of the Openspin symbol table has been doubled (from 16384 symbols
to 32768 symbols) as it was possible that some programs exceeded the
original capacity. Affected the Propeller 1 only.
7. Some of the dynamic kernels were not correctly setting up the first thread
block, which led to lockups when these kernels were used by programs that
used pthreads, including some multi-processing Lua programs. Affected
the Propeller 1 and Propeller 2.
8. The Catalina_FAQ.txt document was out of date. It has been removed.
A quick note about using Catapult to compile multi-model programs on Linux vs Windows ...
When you first compile and run a multi-model demo program (such as the recently released eLua) using Linux and then execute it, you may get a message like:
Error: secondary client must specify address(0x00023AD8)
This is Catapult telling you to adjust the address used for the secondary program. If you see such a message, find the secondary pragma in the appropriate C source file and amend it. It will look something like this:
In this case (demos/elua/aloha/eluax.c) I changed the address to 0x23AD8 and the program then compiled and executed correctly.
For some reason the Catalina Optimizer gives slightly different results on Linux than on Windows, and all the addresses in the demo files in the release were generated for Windows. The programs works correctly compiled under Linux once the address adjustment is made.
I think the difference is due to awka working differently on Linux vs Windows, but I only just noticed it and I need to investigate further.
UPDATE: No, its not the Optimizer. Nor is it awka In fact, it's not Catalina at all. The difference is way deeper than that. The most likely candidate is the different versions of bison used to process the LCC grammar on Windows and Linux. The difference is enough to occasionally cause the LCC code generator to allocate different registers to use. Not wrong, just different. But this can then result in different code sizes when registers have to be spilled and restored. Which means the Optimizer can sometimes perform optimizations on Windows that it cannot do on Linux. Or vice-versa. Which (eventually) means secondary programs may have to be compiled for a different address on Windows than Linux. Both programs work properly - they are just different. And since the version of bison used is outside my control, there is probably not much I can do about it.
The main enhancement is better defaults for the Lua garbage collector, which improves Lua itself as well as all the Lua derivatives (mLua, iLua and eLua).
Here is the relevant extract from the README.TXT
RELEASE 8.1.1
New Functionality
-----------------
1. The default Lua garbage collector has been made more aggressive, to better
suit the Hub RAM constraints of the Propeller. The default is now to use
the "incremental" collector with a "pause" of 110, which means it will
start a new cycle when the use of memory exceeds 110% of the use after
the previous collection.. Once a program is functioning correctly, in
most instances the settings can be relaxed to improve performance.
Other Changes
-------------
1. The Lua "hmi" module was not being initialized correctly in some versions
of Lua, and it would result in a run-time error complaining that "hmi" was
a nil value. Affected the Propeller 2 only on Windows, Linux and Catalyst.
2. The p2_asm script was not generating a listing when requested using the -l
option. Affected Linux only.
3. Add a note about the need to recompile eLua programs if the address of the
secondary needs to be modified to be compiled under Linux. Affected the
Propeller 2 on Linux only.
This is a full release, but it has only one significant enhancement - the ability for eLua to execute both clients and servers in NATIVE mode from Hub RAM.
Here is the relevant extract from the README.TXT
RELEASE 8.1.2
New Functionality
-----------------
1. The default garbage collection for all Lua programs (including Lua itself,
iLua, mLua and eLua) now has a "pause" value of 105, which means it will
start a new cycle when the use of memory exceeds 105% of the use after
the previous collection. This makes the garbage collector even more
aggressive, which reduces the allocation of dynamic memory at the cost
of some performance.
2. eLua now has two additional variants that include ALOHA support:
eluafx - implements an XMM server and a NATIVE client
sluafx - implements a NATIVE server but no client (and so must
be used in conjunction with a remote ALOHA client, which
could be elua, eluax, or eluafx)
The 'Aloha from Lua' document has been updated to describe these variants,
which (when used together) allow both the client and the server to execute
entirely in NATIVE mode from Hub RAM, using two different propellers. This
offers the fastest possible client/server execution option, achieving
significantly better performance than any single propeller option.
Other Changes
-------------
1. Fix issues in the build_all scripts in the eLua/ALOHA demo folder (i.e.
/demos/elua/aloha/build_all and /demos/elua/aloha/build_all.bat).
Affected both Windows and Linux.
Just pushed a minor update to GitHub to add some new eLua/ALOHA variants.
Here is an extract from the updated README.TXT in demos/elua/aloha:
The ALOHA extensions to eLua are described in the document 'Aloha from Lua'.
See that document for more details.
Mentioned in that document is a variant of a slave ALOHA server that omits
the slave client altogether, called sluafx (in the file sluafx.c). This allows
the server to execute entirely from Hub RAM.
When sluafx is used on the slave propeller in conjunction with masterx or
masterfx on the master propeller, it means both the client and server can be
executed entirely from Hub RAM, for a significant speed improvement over the
same program executed from XMM RAM.
Not mentioned in that document are the other possible variants of an ALOHA
enabled Lua server without a client that execute entirely from Hub RAM.
The main purpose of these variants is to maximize the Hub RAM available to
the Lua server.
In addition to the sluafx.c server already mentioned, the following variants
are also provided - they differ only in Catapult pragma options:
sluafix.c - Similar to sluafx.c, but makes more Hub RAM available by
using an integer-only version of the Lua initialization
module (i.e. iinit.c instead of xinit.c) which does NOT
load the Lua maths module. It also uses an updated version
of the binser module that does not require the math module.
The binary is also linked with the integer version of the
extended Catalina library (i.e. -lcix instead of -lcx).
Lua programs executed using this server can declare and use
floating point types and do basic floating point operations,
but they cannot use the functions provided by the maths
module, nor can they do I/O on any floating point types
(e.g. using print).
sluacx.c - As sluafx.c but using COMPACT mode. This is slower than
NATIVE mode but it allows larger Lua programs to be executed
from Hub RAM, so it is still faster than an XMM sever.
sluacix.c - As sluafix.c but using COMPACT mode.
These variants will be built by the 'build_all' script for the P2_SLAVE
propeller (they are only built for execution on 'slave' propellers), or
they can be built individually using commands such as:
set CATALINA_DEFINE=P2_SLAVE
catapult sluafix.c
As an example of the use of these variants, when the client/server version of
the Game of Life (in the folder demos/elua/life) is executed with masterfx and
sluafx, it can support a universe of up to 80 columns by 30 rows (edit the
file common.lua and recompile it to change the size of the universe):
On the P2_MASTER:
masterfx client.lux remote.lux
On the P2_SLAVE:
sluafx
When executed using masterfx and sluafix, it can support a universe of up to
80 columns by 40 rows:
On the P2_MASTER:
masterfx client.lux remote.lux
On the P2_SLAVE:
sluafix
When executed using masterfx and sluacix, it can support a universe of 80
columns by 60 rows - but will execute more slowly:
On the P2_MASTER:
masterfx client.lux remote.lux
On the P2_SLAVE:
sluacix
I have a question for SourceForge and GitHub users ... Are there any Linux users who use SourceForge, or any Windows users who use GitHub?
When I only released Catalina on SourceForge I used to see about the same number of Linux and Windows downloads. Since I began releasing on both SourceForge and GitHub I get about the same numbers of Windows downloads on SourceForge, but close to zero Linux downloads. I now get about the same number of GitHub downloads as I used to get for Linux on SourceForge. But GitHub doesn't tell me whether these are Linux or Windows users.
I could save work on future releases by not bothering with Linux binaries on SourceForge, or Windows binaries on GitHub.
As a Linux user, I'd never go through the hassle of downloading and unzipping a zip file to update something when I could just do git pull. If I see that a project is hosted on both GitHub and Sourceforge, I'll always check GitHub first because people too often don't have a git repo on Sourceforge, and because GitHub's website is so much less clunky (and I can ignore it and just use the Git client).
Why bother manually uploading releases anywhere when GitHub Actions makes building a release a simple matter of tagging a release commit and pushing the tag, and letting GitHub do the rest? You could probably even get them automatically uploaded to SourceForge - then having all builds on both sites wouldn't be any more work.
Or you could just ditch SourceForge entirely, since there's nothing it can do that GitHub can't do better.
@Electrodude said:
Why bother manually uploading releases anywhere when GitHub Actions makes building a release a simple matter of tagging a release commit and pushing the tag, and letting GitHub do the rest? You could probably even get them automatically uploaded to SourceForge - then having all builds on both sites wouldn't be any more work.
Or you could just ditch SourceForge entirely, since there's nothing it can do that GitHub can't do better.
Thanks for your response.
For Linux I could probably automate the build using GitHub Actions myself fairly easily, but only because it is almost completely trivial (one bash script does the whole job, and it takes only a few minutes). I may have a go at this, but to be honest I can't see any real advantage in doing so.
But Windows is a whole 'nother beast. First of all, it takes hours - this is the main reason I have to provide Windows binaries**. Also, maybe it's changed now (I will check again) but last time I looked using MinGW in GitHub Actions was known to be problematic. Since I am not already familiar with GitHub Actions, the thought of going through all the effort to learn about it only to find it still has issues is a little off-putting. I'd rather spend the time actually developing Catalina.
Still, if anyone wanted to have a go at this, I'd be happy to assist - there is a document in the release (BUILD.TXT) that describes in detail how to build Catalina under Windows. Perhaps someone with GitHub Actions experience could try turning that into a workflow?
Ross.
** I have never really understood this. Compiling Catalina and all its demo programs under Windows takes hours, but compiling it using Linux takes only minutes. Using a virtual machine running on the same hardware. Under the same version of Windows.
On Windows NT, especially if you have any sort of security software enabled, spawning a new process and attaching it to the console subsystem is comically slow. Reading directory information is also slow. This can combine into a make+gcc style build taking orders of magnitude longer than reasonable.
Using MinGW in GH Actions is perfectly fine (assuming use as a cross-compiler on a Linux action runner), I set it up for flexspin/spin2cpp.
The real big thing with actions IMO is that you can use it to auto-run any test scripts you have. For flexspin I made a modified version of spinsim, so P1 code can be tested end-to-end.
Comments
Just a couple of notes on using the 2-port and 8-port serial plugins in Propeller 2 XMM programs.
Ross.
Another d'oh! moment ...
Unlike the 2 port serial plugin, the 8 port auto-initialize process (which automatically opens any ports defined in the platform configuration file) uses static data buffers, not local data buffers. In a SMALL XMM program, all data (static or local) is in Hub RAM so everything works, but in a LARGE XMM program, static data is in XMM RAM not Hub RAM, and this is out of reach of the plugin.
A simple workaround is to manually close any auto-initialized ports (or disable the auto-initialize functionality in the relevant Platform file by setting the tx and rx pins to -1) and open them manually using local data buffers.
I thought it must be something like this, but my d'oh moment was that the program I used to try and test it (demos/test_serial8_2.c) closes the auto-initialized ports, but then re-opens them using more static data buffers
I will fix both the auto-initialize functionality and the test program in the next release.
Ross.
Catalina 7.6.2 has been released here.
No major new functionality, just fixes for a few more issues. It is a full release because it requires a rebuild of all the C libraries.
Here is the relevant extract of the README.TXT file:
Another d'oh moment
...
The current version of catapult can't be used with the parallelizer because it strips out #pragma statements it doesn't understand. This means when the parallelizer runs (which also relies on pragmas), it thinks it has nothing to do. Silly of me.
The next version will still warn about unrecognized pragmas but will leave them in the source, so that catapult can be used to build parallelized programs simply by adding the appropriate options to the the relevant pragmas.
I will include a working demo in the next release.
Ross.
Catalina 7.6.3 has been released here.
No major new functionality, mainly just tweaks to allow Catapult to work with the Parallelizer.
The Catapult documentation has also been revised and expanded - see Getting Started with Catapult.
Here is the relevant extract of the README.TXT file:
RELEASE 7.6.3 New Functionality ----------------- 1. The Catalina Catapult utility now ignores pragmas it does not recognize, but leaves them intact. Only one warning message is now issued (for the first such pragma) even if there are multiple unrecognised pragmas. This allows primary and secondary programs to use the Catalina parallelizer. 2 Four new demo programs are now included (in demos\catapult) for the Propeller 1 and 2 to demonstrate the use of the Catalina parallelizer with Catapult: For the Propeller 1: ll_p_p1.c - Demonstrates a parallelized primary program. ll_s_p1.c - Demonstrates a parallelized secondary program. For the Propeller 2: ll_p_p2.c - Demonstrates a parallelized primary and secondary program. ll_s_p2.c - Demonstrates two parallelized secondary programs. 3. The Catalina parallelizer utility now unregisters the factory kernel cogs when they are stopped via the "#pragma propeller stop" pragma. Previously these cogs were stopped but not unregistered. 4. The Catalina catapult utility now unregisters the secondary kernel cog when a secondary program terminates by exiting its function. Previously, these cogs were stopped but not unregistered. 5. On the Propeller 1, the cache cog is now registered when used. This means the registry is displayed correctly, and that this cog is correctly marked as being in use if a program wants to search the registry to locate an unused cog (normally, ANY_COG is used to start a new cog, which does not depend on whether or not the cog was registered). This was already the case on the Propeller 2. Other Changes ------------- 1. The Catalina Geany command to build utilities did not specify that it should run in the project directory, not the current directory. The version in this release fixes this for all new installations of Geany, but it may not fix it for existing versions of Geany or existing projects. For existing versions of Geany, select the "Build -> Set Build Commands" menu item, and add %p to the "Working Directory" field of the "Build Utilities" command. For existing project files, open the project and select the "Project -> Properties" menu item, then select the "Build" tab and add %p to the "Working Directory" field of the "Build Utilities" command. Affected the Propeller 1 and Propeller 2 on Windows and Linux. 2. Fixed a bug that may have led to some symbols being defined multiple times in complex parallelized or multi-model programs. The symbols that may have ended up multiply defined were: cmmd_array, lmmd_array, nmmd_array (on the Propeller 1 & 2), and CMM_LUT_LIBRARY_array, LMM_LUT_LIBRARY_array and NMM_LUT_LIBRARY_array (on the Propeller 2 only). Affected the Propeller 1 and Propeller 2 under Catalyst, Windows and Linux. 3. Some superseded, unused (and undocumented) functions have been removed from the threads libraries: _threadstart_CMM() and _threadstart_CMM_cog() _threadstart_LMM() and _threadstart_LMM_cog() _threadstart_NMM() and _threadstart_NMM_cog() The following functions should be used instead: _threaded_cogstart_CMM_cog() _threaded_cogstart_LMM_cog() _threaded_cogstart_NMM_cog() Affected the Propeller 1 and Propeller 2 under Catalyst, Windows and Linux. 4. The example of using the Catalina Parallelizer with the Propeller version of gcc was incorrectly modified to include Catalina's "prop2.h" rather than gcc's "propeller2.h", which prevented it compiling.Catalina 7.7 has been released on SourceForge and also now on GitHub.
There are very few functional changes in this release (none to Catalina itself). This release is primarily to facilitate the process of making GitHub one of Catalina's release platforms. To do this, changes had to be made to allow for the removal of various binary utilities that were included in previous releases, as well as updating various Makefiles and scripts to eliminate dependencies on those utilities.
For the next few releases at least, both SourceForge and GitHub will be used to host Catalina releases - SoureForge will continue to host the usual combined binary and source releases, and GitHub will host source only releases. But to avoid the need for Windows users to build the release from source, the Catalina binaries included in the Windows binary releases will also be available on GitHub as binary assets associated with each GitHub release. Other binary utilities (such as GNU make and other GNU utilities) are still included in the Windows release but installing them this way is now deprecated, and this option will be removed altogether in a future release.
Here is an extract of the new README.md file that discusses this in more detail:
Ross.
Just a quick note for Linux users - I found and have fixed the bugs that were preventing awka from working when compiled as a 64 bit program under Linux (awka is used internally by various Catalina utilities).
Because everything can now be built as 64 bits, this simplifies the Linux build process (see the updated BUILD.TXT), and I had also had complaints from some Linux users because they didn't like having to install 32 bit libraries on 64 bit Linux distributions just to build Catalina. No more!
The changes will be in the next release, but you can get them now from GitHub (tag v7.7.3). This is why I wanted Catalina on GitHub - to make it easier to push out minor changes without the overhead of a release. So far, so good
Ross.
More good news!
While messing around with awka (see previous post) I also managed to make it compile under MSYS2. This means Cygwin is no longer required to build Catalina on Windows. This simplifies the Windows build process significantly.
However, unlike Linux users, Windows users don't need to build Catalina from source, so there is no urgency on releasing this - it will just be incorporated in the next release.
Ross.
Catalina 7.8 has been released on SourceForge and also on GitHub.
There are no functional changes in this release. The main reason for it is that Catalina is moving to being a pure 64 bit application on both Windows and Linux, but in this release the Windows programs are still compiled as 32 bit applications because I had some problems with the Windows 64 bit version of MSYS2. If I have to support Catalina before the 64 bit Windows release is ready (which I hope will be the next release), this will make it easier.
Here is the relevant extract from the README.TXT
Ross.
Catalina now defaults to 64 bit compilation on both Windows and Linux. The changes are available on GitHub (tag v7.8.1). There are no functional changes, so I don't see any need to make it a formal release.
Now that that's done, I can get back to more interesting things than trying to placate an annoyingly overzealous gcc!
Ross.
Okay! Now that boring 64 bit housekeeping stuff is out of the way, it's time for something a bit more interesting ...
I love Lua, but it is primarily intended to be a scripting language to extend C, not replace C - but C programs that also incorporate Lua can easily end up being too large and too slow to be useful on a micro controller like the Propeller. And this includes Catalina's own multi-processing version of Lua - great in theory, but to provide the multi-processing capabilities, it has to execute from Hub RAM - which limits the sizes of programs it can be used for on the Propeller 2, and prevents it being used on the Propeller 1 at all.
I really wanted an alternative. Specifically, I wanted to be able to write programs where the Lua code in the program executes entirely from XMM RAM, while the C code in the same program executes from Hub RAM and has full access to the Propeller's multi-processing capabilities, and also to Catalina's multi-threading capabilities. Which - not coincidentally - is precisely what catapult was designed to facilitate.
So here is a demo of using Catapult with Lua ...
/****************************************************************************** * Using Catapult with Lua. * * * * This program demonstrates executing a Lua primary function from XMM RAM * * while simultaneously executing a multithreaded C secondary function from * * Hub RAM, and having the two interact via a shared C data structure. * * * * Only one Lua function can be executing, but as many additional secondary * * C functions can be executing simultaneously as there are free cogs. * * * * Compile this program with a command like: * * * * catapult lua_p2.c * * * * Before loading, execute 'build_utilities' to build the appropriate XMM * * loader (for a P2_EDGE with PSRAM and an 8K cache), and then load and * * execute the program with a command like: * * * * payload -o2 -i xmm lua_p2 * * * * Note that if you modify the program, you will probably have to modify the * * address specified in the secondary pragma - but the program will tell you * * what address to use when you compile and execute it. * * * ******************************************************************************/ #pragma catapult common options(-p2 -C P2_EDGE -C SIMPLE -lmc) #include <catapult.h> #include <stdlib.h> #include <prop.h> /* * define a type to be used for exchanging information between * the Lua primary Lua function and the secondary C function ... */ typedef struct shared_data { int go; } shared_data_t; /****************************************************************************** * * * The secondary function - executes multithreaded C * * * ******************************************************************************/ #pragma catapult secondary hub_function(shared_data_t) mode(NMM) address(0x76468) stack(8000) options(-lci -lthreads -C NO_FLOAT) /* * include Catalina multi-threading: */ #include <threads.h> /* * define how many threads we want: */ #define THREAD_COUNT 10 /* * define the stack size each thread needs (since this number * depends on the function executed by the thread, the smallest * possible stack size has to be established by trial and error): */ #define STACK_SIZE (MIN_THREAD_STACK_SIZE + 55) static int ping; /* * function : a function that can be executed as a thread */ int function(int me, char *not_used[]) { while (1) { if (ping == me) { // print our id t_printf("%d ", (unsigned)me); ping = 0; } else { // nothing to do, so yield _thread_yield(); } } return 0; } /* * hub_function : start THREAD_COUNT threads, then ping each one in turn */ void hub_function(shared_data_t *s) { int i = 0; void *thread_id; unsigned long stacks[STACK_SIZE * THREAD_COUNT]; // wait till we are told to go (by Lua!) before proceeding while (s->go == 0); t_printf("... C executes multiple threads using cog %d\n\n", _cogid()); // assign a lock to avoid context switch contention _thread_set_lock(_locknew()); // start instances of function until we have started THREAD_COUNT of them for (i = 1; i <= THREAD_COUNT; i++) { thread_id = _thread_start(&function, &stacks[STACK_SIZE*i], i, NULL); t_printf("thread %d ", i); if (thread_id == (void *)0) { t_printf(" failed to start\n"); while (1) { }; } else { t_printf(" started, id = %d\n", (unsigned)thread_id); } } // now loop forever, pinging each thread in turn while (1) { t_printf("\n\nPress a key to ping all threads\n"); k_wait(); for (i = 1; i <= THREAD_COUNT; i++) { t_printf("%d:", i); // ping the thread ping = i; // wait till thread responds while (ping) { // nothing to do, so yield _thread_yield(); }; } } } /****************************************************************************** * * * The primary function - executes Lua * * * ******************************************************************************/ #pragma catapult primary binary(lua_p2) mode(XMM) options(-lcx -W-w -C CLOCK -llua linit.c) #include <lua.h> #include <lualib.h> #include <lauxlib.h> /* * Define the Lua code - here it is defined using a string, but it could * just as easily be loaded from a file. * * The function "flash" prints a friendly message, then initiates the * multi-threading by setting the "go" flag in the shared data, then * flashes the specified LED just to indicate it is still executing ... */ char *lua_code = " LED = 38; --[[ suitable for P2 EDGE ]]" " function flash(shared)" " print('Hello World (from Lua!)\\n');" " print('Lua will flash LED ' .. LED .. " " ' using cog ' .. cogid() .. ' while ...');" " go(shared, 1);" " x = 0;" " while (true) do" " propeller.setpin(LED, x);" " x = x ~ 1;" " propeller.msleep(500);" " end" " end"; /* * define a C function that can be called from Lua to return the current cogid */ static int cogid(lua_State *L) { lua_pushinteger(L, _cogid()); return 1; } /* * define a C function that can be called from Lua to access the go field * of the shared data - it accepts either one or two parameters - the first * (required) is the address of the shared data and the second (optional) * is a new value for go. It returns the old value of go ... */ static int go(lua_State *L) { int old_go = 0; shared_data_t *s; int n = lua_gettop(L); // number of arguments if (n == 2) { // optional argument specified register lua_Integer new_go = luaL_checkinteger(L, 2); s = (shared_data_t *)lua_topointer(L, -2); old_go = s->go; s->go = new_go; lua_pop(L, 2); lua_pushinteger(L, old_go); return 1; } else if (n == 1) { // optional argument not specified s = (shared_data_t *)lua_topointer(L, -1); old_go = s->go; lua_pop(L, 1); lua_pushinteger(L, old_go); return 1; } else { // incorrect number of arguments return 0; } } void main(int argc, char *argv[]) { shared_data_t shared = { 0 }; int cog; int result; // creat a new Lua state lua_State *L = luaL_newstate(); // open the standard Lua libraries luaL_openlibs(L); // start the multi-threading program - it will wait // for the 'go' field of the shared_data argument to // be set before starting the threads RESERVE_AND_START(hub_function, shared, ANY_COG, cog); t_printf("Press a key to start Lua ...\n\n"); k_wait(); // load the Lua code and execute it result = luaL_loadstring(L, lua_code) || lua_pcall(L, 0, 0 ,0); if (result == LUA_OK) { // set up the C 'cogid' function, so it can can be // called from Lua to return the current cog id lua_pushcfunction(L, cogid); lua_setglobal(L, "cogid"); // set up the C 'go' function, so it can can be // called from Lua to start the multi-threading lua_pushcfunction(L, go); lua_setglobal(L, "go"); // call Lua "flash" - i.e. flash(&shared) lua_getglobal(L, "flash"); lua_pushlightuserdata(L, &shared); result = lua_pcall(L, 1, 1, 0); } if (result != LUA_OK) { // something went wrong t_printf("Unexpected result (%d)\n", result); } // not necessary here, but it is good practice to tidy up lua_close(L); }In the above program the primary function executes Lua code from XMM RAM while the secondary function executes C code from Hub RAM with full Propeller functionality and at full Propeller speeds. The program could have as many secondary C functions as there are available cogs, and they can all communicate via shared Hub RAM.
Lua can therefore be used for what it was designed for - i.e. configuring and customizing C programs ... plus it can provide additional functionality that is not particularly time critical or which would be too tedious to write in C. And C can do all the rest. The Lua code does not even need to be compiled into the program - typically it would not be, but would instead be loaded from a file at run time so that the application does not need to be re-compiled to be configured, customized or even extended - precisely what Lua was intended for!
The above program is for a Propeller 2 (specifically the P2-EC32MB). But an almost identical program can be executed on a Propeller 1 with sufficient XMM RAM (e.g. the C3). Both programs are included in the attached ZIP file.
These programs will be included as demos in the next release, but they can be compiled and executed with the current Catalina release - just ensure that a suitable version of linit.c is in the same folder when the programs are compiled (a suitable linit.c can be found in demos\lua, and is also included in the attached ZIP file).
See the programs themselves for more details on how to build and execute them.
Ross.
EDIT: The Propeller 1 version of the program said it needed an 8k cache, but in fact it needs a 1k cache - zip file updated.
EDIT: The demo programs were simplified - using a lock pool is not necessary in this case. New versions attached.
The Windows release of Catalina 7.8 was missing some DLLs. This may have prevented payload from executing - it could result in a Windows "The Application Was Unable to Start (0xc000007b)" message, or a message about a missing DLL.
On SourceForge I have updated the installer (i.e. Catalina_7.8_Setup.exe), and on GitHub I have added the missing DLLs as an additional binary asset - https://github.com/rosshigson/Catalina/releases/download/v7.8/Catalina_7.8_Windows_binaries_2.ZIP
If payload works for you it means you already have the required DLLs installed and you don't need to do anything. If not, you can either download the new installer and re-install Catalina, or just download the missing DLLs from the Github asset and add them to your existing Catalina "bin" directory.
Ross.
While working on some 'helper' functions to simplify the integration of the various parts of a multi-model catapult program (which allows some parts of the program to execute from XMM RAM while other parts execute from Hub RAM), it occurred to me that the Catalina Registry already offered an ideal mechanism for doing this, since it was specifically designed for use in an analogous situation - i.e. where some parts of a program execute from Hub RAM while other parts execute from Cog RAM - i.e. as plugins).
Silly of me not to realize this earlier
So, here is an example catapult program where the primary program is a Server that executes from XMM RAM, offering services that can be called as C functions from one or more secondary Client programs executing from Hub RAM:
/****************************************************************************** * Using Catapult to build a C server. * * * * This program demonstrates executing a primary server from XMM RAM that * * dispatches service calls made from one or more C secondary clients to * * C functions implemented in the primary server (i.e. from XMM RAM). * * * * The program uses the Catalina Registry as the mechanism, which enables * * the server to offer multiple services, and also adds lock protection to * * the services. Using the registry supports only a limited number of * * parameter profiles - i.e. those that are required to implement Catalina * * plugins (see func_1 to func_5). However, the basic "short" service can * * also be used to pass a pointer to the shared data structure as the * * parameter, which allows for arbitary data to be exchanged between client * * and server, as usual for catapult programs (see func_6). * * * * This version is for a Propeller 2 P2_EDGE with PSRAM (i.e. a P2-EC32MB), * * but it could be easily modified to suit any Propeller 1 or 2 which has * * sufficient XMM RAM by modifying the catapult pragmas appropriately. * * * * Compile this program with a command like: * * * * catapult serve_c.c * * * * Before loading, execute 'build_utilities' to build the appropriate XMM * * loader (for a P2_EDGE with PSRAM and an 8K cache), and then load and * * execute the program with a command like: * * * * payload -o2 -i xmm serve_c * * * * Note that if you modify the program, you may have to modify the address * * specified in the secondary pragma - but the program will tell you what * * address to use when you compile and execute it. * * * ******************************************************************************/ #pragma catapult common options(-W-w -p2 -C P2_EDGE -C SIMPLE -lc -lmc service.c) #include <stdlib.h> #include "service.h" #include "plugin.h" /* * define global service identifiers for our services * (can be any unused service ids, but must be unique) */ #define MY_SVC_1 (SVC_RESERVED+1) #define MY_SVC_2 (SVC_RESERVED+2) #define MY_SVC_3 (SVC_RESERVED+3) #define MY_SVC_4 (SVC_RESERVED+4) #define MY_SVC_5 (SVC_RESERVED+5) #define MY_SVC_6 (SVC_RESERVED+6) /* * define a type to be used for exchanging data between * the primary and secondary functions */ typedef struct shared_data { int ready; int start; int data; } shared_data_t; /****************************************************************************** * * * The secondary client - calls services provided by the primary server * * * ******************************************************************************/ #pragma catapult secondary hub_client(shared_data_t) address(0x770AC) mode(NMM) stack(500) /* * define proxy functions, which use Catalina's pre-defined * service functions to call the real server functions via * the Registry */ int func_1(long l) { return _short_service(MY_SVC_1, l); } int func_2(long l) { return _long_service(MY_SVC_2, l); } int func_3(long l1, long l2) { return _long_service_2(MY_SVC_3, l1, l2); } float func_4(float f1, float f2) { return _float_service(MY_SVC_4, f1, f2); } long func_5(float f1, float f2) { return _long_float_service(MY_SVC_5, f1, f2); } int func_6(shared_data_t *shared) { return _short_service(MY_SVC_6, (long)shared); } /* * hub_client : call the services offered by the primary function * by calling the proxy functions defined above */ void hub_client(shared_data_t *s) { int i = 0; long l = 0; float f = 0.0; t_printf("Client ready - press a key to start the server ...\n"); k_wait(); // tell the primary server we are ready s->ready = 1; // wait till we are told to start by the primary before proceeding while (s->start == 0); t_printf("The secondary client calls from cog %d ...\n", _cogid()); t_printf("... to the primary server on cog %d\n", _locate_plugin(LMM_SVR)); // call the services using the proxy functions while(1) { t_printf("\nPress a key to continue ...\n"); k_wait(); i = func_1(1000); t_printf("result of float_1 = %d\n", i); i = func_2(2000); t_printf("result of float_2 = %d\n", i); i = func_3(3000, 4000); t_printf("result of float_3 = %d\n", i); f = func_4(5000.0, 6000.0); t_printf("result of float_4 = %f\n", f); l = func_5(7000.0, 8000.0); t_printf("result of float_5 = %ld\n", l); i = func_6(s); t_printf("result of float_6 = %d\n", i); } } /****************************************************************************** * * * The primary server - executes a C dispatcher, dispatching calls * * to the C functions specified in my_service_list * * * ******************************************************************************/ #pragma catapult primary binary(serve_c) mode(XMM) options(dispatch_c.c) /* * define our services as C functions */ int func_1(long l) { // example SHORT_SVC t_printf("Hello, World (from C func_1!)\n"); t_printf("Args: l=%ld\n", l); return l + 1; } int func_2(long l) { // example LONG_SVC t_printf("Hello, World (from C func_2!)\n"); t_printf("Args: l=%ld\n", l); return l + 2; } int func_3(long l1, long l2) { // example LONG_2_SVC t_printf("Hello, World (from C func_3!)\n"); t_printf("Args: l1=%ld, l2=%ld\n", l1, l2); return l1 + l2 + 3; } float func_4(float f1, float f2) { // example FLOAT_SVC t_printf("Hello, World (from C func_4!)\n"); t_printf("Args: f1=%f, f2=%f\n", f1, f2); return f1 + f2 + 4; } long func_5(float f1, float f2) { // example LONG_FLOAT_SVC t_printf("Hello, World (from C func_5!)\n"); t_printf("Args: f1=%f, f2=%f\n", f1, f2); return (long)(f1 + f2) + 5; } long func_6(shared_data_t *shared) { // example SHARED_SVC t_printf("Hello, World (from C func_6!)\n"); shared->data++; return shared->data; } /* * define a list of services, parameter profiles and service identifiers * - note that C services require the address of the functions, the names * are for information only */ svc_list_t my_service_list = { {"func_1", func_1, SHORT_SVC, MY_SVC_1}, {"func_2", func_2, LONG_SVC, MY_SVC_2}, {"func_3", func_3, LONG_2_SVC, MY_SVC_3}, {"func_4", func_4, FLOAT_SVC, MY_SVC_4}, {"func_5", func_5, LONG_FLOAT_SVC, MY_SVC_5}, {"func_6", func_6, SHARED_SVC, MY_SVC_6}, {"", NULL, 0, 0} }; void main() { shared_data_t shared = { 0, 0, 0 }; int cog; // re-register this cog as a server (it will already // be registered, but as a kernel, not a server) _register_plugin(_cogid(), LMM_SVR); //register the services, so clients can find them register_services(_locknew(), my_service_list); // load the secondary client - it will wait for the // 'start' field of the shared_data argument to be set RESERVE_AND_START(hub_client, shared, ANY_COG, cog); // wait for the secondary to be ready (required if the // secondary print messages, to prevent garbled output) while (!shared.ready); // start the secondary client t_printf("Server ready - press a key to begin ...\n"); k_wait(); shared.start = 1; // dispatch service calls while(1) { dispatch_C(my_service_list); } }"Very nifty" I am sure I can hear you all say ...
... but wait - there's more ... the Server does not even have to be a C program.
Since we are now using the Catalina Registry as our call mechanism, the Server in the above program can simply be replaced - exactly as a plugin can - with the same services implemented using a different Server ... including a Server which offers the same services implemented as Lua functions executing from XMM RAM! Here is the new primary Server component only - the common component and the secondary Client component are exactly the same as above:
/****************************************************************************** * * * The primary server - executes a Lua dispatcher, dispatching calls * * to the Lua functions specified in my_service_list * * * ******************************************************************************/ #pragma catapult primary binary(serve_lua) mode(XMM) options(-llua dispatch_lua.c linit.c) #include <lua.h> #include <lualib.h> #include <lauxlib.h> /* * define our services as Lua functions */ char *lua_code = " function func_1(l)" // example SHORT_SVC " print('Hello World (from Lua func_1!)');" " print('Args: l=' .. l);" " return l + 1;" " end" "" " function func_2(l)" // example LONG_SVC " print('Hello World (from Lua func_2!)');" " print('Args: l=' .. l);" " return l + 2;" " end" "" " function func_3(l1, l2)" // example LONG_2_SVC " print('Hello World (from Lua func_3!)');" " print('Args: l1=' .. l1 .. ', l2=' .. l2);" " return l1 + l2 + 3;" " end" "" " function func_4(f1, f2)" // example FLOAT_SVC " print('Hello World (from Lua func_4!)');" " print('Args: f1=' .. f1 .. ', f2=' .. f2);" " return f1 + f2 + 4;" " end" "" " function func_5(f1, f2)" // example LONG_FLOAT_SVC " print('Hello World (from Lua func_5!)');" " print('Args: f1=' .. f1 .. ', f2=' .. f2);" " return f1 + f2 + 5;" " end" "" " function func_6(shared)" // example SHARED_SVC " print('Hello World (from Lua func_6!)');" " return data(shared);" " end"; /* * define a list of services, parameter profiles and service identifiers * - note that Lua services require only the names of the functions, and * their addresses should be NULL */ svc_list_t my_service_list = { {"func_1", NULL, SHORT_SVC, MY_SVC_1}, {"func_2", NULL, LONG_SVC, MY_SVC_2}, {"func_3", NULL, LONG_2_SVC, MY_SVC_3}, {"func_4", NULL, FLOAT_SVC, MY_SVC_4}, {"func_5", NULL, LONG_FLOAT_SVC, MY_SVC_5}, {"func_6", NULL, SHARED_SVC, MY_SVC_6}, {"", NULL, 0, 0} }; /* * define a C function that can be called from Lua to access the shared data * - it accepts one parameter, which is the address of the shared data, and * then increments and returns the value of the shared data "data" field, * just as a demo (it is used in Lua func_6, defined above). */ static int data(lua_State *L) { shared_data_t *s; if (lua_gettop(L) == 1) { s = (shared_data_t *)lua_topointer(L, -1); lua_pop(L, 1); s->data++; lua_pushinteger(L, s->data); return 1; } else { // incorrect number of arguments return luaL_error(L, "expected 1 argument"); } } void main() { shared_data_t shared = { 0, 0, 0 }; int cog; int result; lua_State *L; // re-register this cog as a server (it will already // be registered, but as a kernel, not a server) _register_plugin(_cogid(), LMM_SVR); //register the services, so clients can find them register_services(_locknew(), my_service_list); // load the secondary client - it will wait for the // 'start' field of the shared_data argument to be set RESERVE_AND_START(hub_client, shared, ANY_COG, cog); // create a new Lua state and open the standard Lua libraries L = luaL_newstate(); luaL_openlibs(L); // load the Lua code and execute it so Lua knows about the // functions that can be called result = luaL_loadstring(L, lua_code) || lua_pcall(L, 0, 0 ,0); if (result != LUA_OK) { // something went wrong t_printf("Unexpected result (%d)\n", result); } // tell Lua about the 'data' function, so it can can be called // from Lua to demonstrate Lua accessing the shared data lua_pushcfunction(L, data); lua_setglobal(L, "data"); // wait for the secondary to be ready (required if the // secondary print messages, to prevent garbled output) while (!shared.ready); // start the secondary client t_printf("Server ready - press a key to begin ...\n"); k_wait(); shared.start = 1; // dispatch service calls while(1) { dispatch_Lua(L, my_service_list); } }Since we are now using the Catalina Registry, the Clients no longer care how the functions offered by the Server are implemented - and catapult makes it trivial to swap a C Server for a Lua Server.
Both files, plus some necessary support files, are attached. They can be compiled with the current release of Catalina using the commands:
catapult serve_c.cand
catapult serve_lua.cThe Catalina Registry also provides a locking mechanism that protects the Server - this was already implemented as it was necessary to protect plugins - so the secondary Client programs can also be multi-threaded.
Getting very close to the best of all possible worlds now!
Ross.
EDIT: Attached zip updated. See next post.
I have updated the "catapult_server.zip" file attached to the previous post, for two reasons:
Ross.
Catalina 7.9 has been released on SourceForge and also on GitHub.
There are only minor functional changes in this release. The main reason for it is that I fixed a significant issue with dynamic memory allocation functions (malloc(), free() etc) that meant programs that used them could run out of heap space even though there was still plenty available.
Here is the relevant extract from the README.TXT
RELEASE 7.9 New Functionality ----------------- 1. Catalina, catbind and bcc all have a new command line option. The new option (-H) accepts an address parameter and can be used to specify the maximum address that will be used by the heap. In all memory modes except LARGE mode, the heap and stack share Hub RAM, with the heap growing upward from the highest used low Hub address, and the stack growing downward from the lowest used high hub address. This means they can eventually overlap, with potentially disastrous consequences. The -H option allows this to be avoided, by limiting the growth of the heap. The program may run out of heap space, but this can be detected (e.g. sbrk() or malloc() will return an error if there is no more space). The required amount of stack space can be determined by printing the current stack pointer at various suitable points in the program - below is a macro that uses inline PASM to do this, and a trivial program that uses it. This program will work in any memory model on any Propeller: // this handy macro returns the current stack pointer // in any memory model on the P1 or P2 ... #define SP PASM( \ "#ifdef COMPACT\n" \ " word I16B_PASM\n" \ "#endif\n" \ " alignl\n" \ " mov r0, SP\n") void main() { printf("SP=0x%06X\n", SP); while(1); } Suppose on a Propeller 1 it was known that the stack could grow down to 0x6000 - then it would be appropriate to specify -H 0x6000 to prevent the heap ever growing large enough to overwrite the stack. The parameter can be specified as decimal (including an optional 'k' or 'm' suffix) or as hexadecimal (using the format $XXXXXX or 0xXXXXXX). For example, to ensure the heap never grows above 24k, leaving the top 8k for buffers and stack space, use a command like: catalina prog.c -lc -H 24576 or catalina prog.c -lc -H 24k or catalina prog.c -lc -H 0x6000 The -H option can be used on the Propeller 1 or 2. In all modes except LARGE mode the address refers to a Hub address. It can also be used in LARGE mode, where the heap is in XMM RAM, but the address refers to an XMM RAM address. This could be used (for example) to reserve an upper area of XMM RAM for other uses, such as for a buffer. However, note that the start address of the XMM RAM can vary from platform to platform, so check the XMM_RW_BASE_ADDRESS in the various platform configuration files. 2. New client/server support functions have been added to the libraries, and several new catapult demos have been added, to demonstrate them - see the document "Getting Started with Catapult" for details. The new demos in the demos/catapult folder are: srv_c_p1.c & svr_c_p2.c -- demonstrate a C client with a C server. srv_l_p1.c & svr_l_p2.c -- demonstrate a C client with a Lua server. The above demo programs all use the library functions and a dispatcher which enables client/server programs to interact using the functionality that Catalina uses to interact between C programs and plugins. However, this is limited to service profiles that match the existing ones used for plugins. If these profiles are not sufficient, new profiles can be created along with a custom dispatcher - a demonstration of this is given in the folder demos/catapult/custom folder, using programs very similar to those above, but extended to include one new custom service profile. Other Changes ------------- 1. An issue first noted in Catalina 7.6.1 was not fixed in all the affected kernels. In kernels without sufficient space to implement the basic floating point operations (+, -, *, / etc), an 8 byte data block is used to transfer information between the kernel and the floating point plugin. In some kernels this block was not being set up correctly, leading to the use of the 8 bytes starting at memory location 0 as the data block, which would overwrite the clock values on the Propeller 1, leading to issues with some functions that used timers. The specific kernels affected were the CMM and XMM kernels on the Propeller 1, and the XMM Kernel on the Propeller 2. 2. Catalina was not checking if there was sufficient heap space left before trying to allocate a new block of memory (e.g. via sbrk, malloc, calloc or realloc). As a result, a program could end up unable to allocate memory on the heap even though there was plenty of free space once the heap became too fragmented, because the function intended to defragment and consolidate the blocks on the free list was not being triggered as expected. Affected the Propeller 1 and 2. This issue affected programs that made a large number of dynamic memory allocation and deallocation calls for small randomly sized blocks of memory, because existing free blocks are re-used whenever possible, breaking them up into smaller and smaller pieces, and allocating a new block when the existing free blocks could not satisfy a request. The Catalina demo program most affected by this bug was Dumbo Basic, which allocates and then frees a very large number of small blocks when performing string operations, which meant the heap very quickly ended up highly fragmented. For example, the 'eliza.bas' program would stop responding if a long sentence was entered. This issue has been fixed by (1) adding automatic detection of the maximum heap size, so that the defragment function is triggered when heap space is exhausted - this solves the problem for LARGE mode programs (e.g. when compiling Dumbo Basic for the P1 in LARGE mode), and (2) adding a new command line option (-H) which can be used in any mode to specify the maximum heap address to be used (e.g. when compiling Dumbo Basic in NATIVE mode for the P2).Ross.
Here's a pretty thing ... well, it is if you like this kind of thing, which I do ...
Attached is a preview of what's coming in Catalina 8.0. If (like me) you primarily work on the command line (because it's so much faster and easier than any IDE) the lack of a decent command line in Catalina has been a sore point which has finally led me to spend some time improving it.
Here's the blurb ...
Enhanced Catalyst and Lua command line processing ================================================= This folder contains a pre-release demo of Catalyst 8.0 and Lua, both with enhanced command line editing and history capabilities. This enhanced functionality will be supported in Lua on the Propeller 1 and 2, but in Catalyst only on the Propeller 2 (the Propeller 1 lacks suffiicent space). The demo versions in this folder is specifically for a Propeller 2, and should execute on just about any Propeller 2 platform, including the P2 EDGE and P2 EVAL boards. Copy the contents of this zip file to an SD card, insert it into the Propeller and set the Propeller to boot from the SD. Then use a VT100 compatible terminal to interact with the Propeller. You can use Catalina's comms.exe, PuTTY, or any other VT100 terminal emulator. You can also use the enhanced version of payload (8.0) included in this folder. Note that prior versions of payload (i.e. 7.9 or earlier) will not work - version 8.0 has some additional VT100 emulation capabilities required to support the enhanced capabilities. Note that you may need to reboot the Propeller after starting the terminal emulator. To use payload, specify the port the baud rate and options '-i -q2' - for example, if your P2 is on port 8: payload -p8 -b230400 -i -q2 To use Catalina's comms program, in a Catalina command line window enter either: comms /baud=230400 /com=8 /mode=vt100 or just: vt100 8 To use PuTTY or another VT100 terminal emulator, the relevant communication and terminal emulation parameters will need to be set up appropriately. See the terminal emulator documentation. The enhanced functionality in both Catalyst and Lua is mainly implemented by the 'linenoise' line editor, which has now been incorporated into the Catalina libraries, and compiled into the Catalyst and Lua binaries in the demo. The new functionality can be summarized by the following list of enhanced key functions: LEFT ARROW (or CTRL B) : move cursor left RIGHT ARROW (or CTRL F) : move cursor right UP ARROW (or CTRL P) : previous command in history DOWN ARROW (or CTRL N) : next command in history HOME (or CTRL A) : move cursor to start of line END (or CTRL E) : move cursor to end of line CTRL U : clear entire line CTRL K : clear from cursor to end of line CTRL L : clear screen CTRL W : clear previous word CTRL T : swap current and previous characters CTRL C : exit CTRL D : at start of line means exit (otherwise delete) TAB : command completion Try any of the above at the Catalyst prompt. Note that the Catalyst command completion functionality is currently implemented as a simple hardcoded list of all Catalyst commands. This is because interrogating the SD card for the whole directory structure prior to each command would be too expensive. This means that Catlyst may suggest commands that are not actually present in the demo (e.g. entering 'i<TAB>' will suggest both 'if' and 'ilua' even though only 'ilua' is present in the demo folder). In Lua, you will need to use the 'ilua' program provided, because the new functionality is not implemented in the basic 'lua' interactive program. Instead of just starting 'lua', start the new 'ilua' program either by entering: lua ilua.lua or just: ilua You will see a helpful message (implemented in help.lua, which can be edited to amend or remove it if required). Try any of the above enhanced key functions at the ilua prompt. Note that in both cases the command history is stored in a file on the SD card ('CATALYST.HST' for Catalyst, and '_ILUA.HST' for Lua) and can be examined using any text editor (the vi editor is included in the demo). The Catalyst history is saved before each command is executed, but the Lua history is saved only on exit from Lua. If the Propeller is reset, the history for the current 'ilua' session will be lost, but the history of previous 'ilua' sessions will be intact. For the purposes of this demo, the Catalyst prompt has been changed to "? " to differentiate it from the Lua prompt "> ". This is done by setting the value of PROMPT in the file 'CATALYST.ENV' and can be edited or removed if required.Ross.
EDIT: How odd. I was just about to delete the files attached to this post since Catalina 8.0 has now been released which superseded them - but there are no longer any binaries attached. I wonder what happened to them?
A small correction to the previous post. I said ...
This is true in the demo, but it turns out I was worrying needlessly. The self-hosted version of Catalina puts over 8,000 files on the SD card, and I was concerned this this would take too long to process to allow for arbitrary completion of file names - but of course it only has to process one folder at a time, not the whole SD card. I just wrote some code to try it and it turns out to be easily fast enough.
So I won't need to use a hard coded list - release 8.0 will have full command and file name completion, reflecting what is actually on the SD Card.
Catalina 8.0 has been released on SourceForge and also on GitHub.
The main change is the inclusion of the enhanced command line capabilities in both Catalyst and Lua. I had not intended to release 8.0 just yet, but I noticed that the GitHub release of 7.9 had the wrong binary executable files attached (it had the correct sources, but some of the executables were from 7.8). It was about the same amount of work to re-install and rebuild 7.9 as to just release 8.0. The SourceForge release of 7.9 was ok.
Here is the relevant extract from the README.TXT
RELEASE 8.0 New Functionality ----------------- 1. On the Propeller 2, Catalyst now incorporates the 'linenoise' command line library, which provides enhanced command line editing functionality. This is implemented as various keyboard shortcuts: LEFT ARROW (or CTRL B) : move cursor left RIGHT ARROW (or CTRL F) : move cursor right UP ARROW (or CTRL P) : previous command in history DOWN ARROW (or CTRL N) : next command in history HOME (or CTRL A) : move cursor to start of line END (or CTRL E) : move cursor to end of line CTRL U : clear entire line CTRL K : clear from cursor to end of line CTRL L : clear screen CTRL W : clear previous word CTRL T : swap current and previous characters CTRL C : exit CTRL D : at start of line means exit (otherwise it means delete) TAB : command or file name completion 2. An enhanced Lua REPL (Read Eval Print Loop) based on 'lua-repl' has been added to Catalyst, which uses the 'linenoise' command line editing library. While the 'linenoise' library is now built into Lua, it is not loaded by the default interactive program (i.e. 'lua'). This is done to save space, since that program is primarily now used to execute Lua scripts. Instead, 'lua-repl' is used to load 'linenoise' and a few other plugins in a new 'ilua.lua' script. So to start the new interactive Lua from the Catalyst command line, just enter the command 'ilua'. This script provides the same keyboard shortcuts as those described above for Catalyst. Other Changes ------------- 1. Payload's interactive mode has had some modifications to make its VT100 terminal emulation more closely match a real VT100. One change is in the implemetation of what CR and LF do. For payload to correctly display the output of programs that send both a CR and LF, it is now recommended that -q2 be used as the appropriate payload option (i.e. ignore the LF) whereas previously the recommended option was -q1 (i.e. ignore the CR). Various documents have been updated.I seem to spend more time building releases than actually coding these days ...
The document Getting Started with Catalina contains some outdated information. The structure of local user-defined libraries changed in release 6.0 to accommodate the limitations of DOS 8.3 file names (to allow Catalina to run under Catalyst on the Propeller 2) but this document was not updated.
Point 3 in the section Compiling Multiple Files covers the option of creating and using a user-defined library. It should now read:
The document has been updated on GitHub.
Catalina 8.1 has been released on GitHub and SourceForge
The main enhancement is the inclusion of eLua - a new enhanced version of Lua that supports client/server programming, either on multiple cogs on the same propeller, or on up to nine 'clustered' propellers.
Here is the relevant extract from the README.TXT
Release 8.1 New Functionality ----------------- 1. A new Lua variant has been added - eLua supports client/server programs where the client and server execute on the same Propeller, and it can be extended by the inclusion of a new serial protocol (ALOHA) to support clients and servers executing on different propellers. Client/server programs are supported on the Propeller 2 only. See the document 'Aloha from Lua' for full details. 2. The "exec" command (i.e. exec.lua) has been extended to allow parameter substitution anywhere in the executed script. Previously, parameter substitution was performed only when "_n" was specified as an argument to the exec command itself - this was replaced with the value of the nth parameter to the current script. This is still supported, but in addition "%n" can now be specified anywhere in the script and it will be substituted with the current value of the nth parameter to the current script. In most cases, "%n" can be used wherever "_n" was used, except where "_n" represents the name of an environment variable and not a parameter, such as if it was the name (but not the parameters) of an "if" command. For an example of this, see the "loop2" script in the folder demos/catalyst/core, which is a minor variant of the "loop" script. The "exec" command is available on the Propeller 2 only. 3. A new "call" command has been added (i.e. call.lua) that allows a Catalyst script to be called from within another Catalyst script, which resumes the original script when the called script terminates. The "call" command can also be used on the command line, where it behaves effectively the same as the "exec" command - but when used WITHIN a script, "call" and "exec" behave differently - when "exec" is used, the executed script REPLACES the current script, whereas when "call" is used, the currently executing script will be resumed once the called script terminates. The "exec" command is still the mechanism to be used for looping scripts, as demonstrated in the "loop" and "loop2" scripts. For example, to execute the Catalyst script "script_2" from within the script "script_1", passing it the original parameter 1 and "xxx" as parameters, and then once that completes, call it again with the original parameter 2 and "yyy" as parameters, the script "script_1" might contain lines like: call script_2 _1 xxx call script_2 _2 yyy If "exec" were used instead of "call", the second invocation of script_2 would never be executed because the first invocation would not return. Note that calls cannot be nested - i.e. a script can be executed from the command line that calls another script, but that script cannot then call any further scripts. To support the "call" capability, two additional lua scripts have been provided (_save.lua and _restore.lua) which can save and restore the current values of up to 23 parameters in a single environment variable: _save : save parameters _0 .. _22 in a named environment variable (_SAVED_PARAMS is the default name) _restore : restore parameters _0 .. _22 from a named environment variable (_SAVED_PARAMS is the default name) The "call" command is available on the Propeller 2 only. 4. The self-hosted catalina command (i.e. catalina.lua) can now be executed within a script. Previously, doing so would terminate the script. 5. A simple "echo" command has been added (i.e. echo.lua) which just echoes its arguments. For example: echo hello there! or echo %1 %2 %3 etc 6. New functions have been added to the Lua "propeller" module, which are analogous to the corresponding C library functions: cogid, clockfreq, clockmode, getcnt, muldiv64, locknew, lockclr, lockset, lockret, locktry, lockrel See the document "Lua on the Propeller 2 with Catalina" for details. Note that locktry, lockrel and muldiv64 functions are implemented on the Propeller 2 only. 7. All the HMI functions (e.g. k_get, k_ready ... t_char, t_geometry ..., m_buttton, m_reset ...) have been removed from the Lua "propeller" module and put in a separate "hmi" module. Also, t_string was missing and has now been added. If these functions are now undefined and cause runtime errors when called because they are nil, then they should be renamed - e.g. from "propeller.k_new()" to "hmi.k_new()" etc. All the Lua examples in the release have been updated accordingly. 8. A new example program (ex13.lua) has been added to demonstrate the use of Propeller locks as another possible synchronization mechanism in the Multi-Processing version of Lua (i.e. mlua or mluax). See the document "Lua on the Propeller 2 with Catalina" for details. 9. The multi-processing example 10 (ex10.lua) has been updated to the LED numbers appropriate for a P2 EDGE board rather than a P2 Evaluation board. Also, the example that tries to execute the maximum number of Lua threads (ex8.lua) has had the default reduced to 10 from 12 (which was not achievable in some cases). 10. A new example program (args.lua) has been added to demonstrate how a Lua script can process arguments whether executed from the command line, such as: args one 2.0 3 or lua args.lua one 2.0 3 Or from within another Lua script, such as: f = loadfile("args.lua"); f("one", 2.0, 3); or loadfile("args.lua")("one", 2.0, 3); 11. It is now possible to specify the pins and baud rates to be used for the default serial port and the blackcat/blackbox port to be specified on the Catalina command line using the -C command line option - e.g: -C _RX_PIN=1 -C _TX_PIN=2 -C _BAUDRATE=115200 -C _BLACKCAT_RX_PIN=1 -C _BLACKCAT_TX_PIN=2 -C _BLACKCAT_BAUD=115200 12. There is a new library function that can be used to align the value of the system break (aka sbrk). If aligning the system break is needed, this function should be called before any memory allocation function (i.e. malloc() etc). The function is: /* * _align_sbrk - align sbrk to (1<<align), add offset, and report * the final value (using t_printf) if requested. * For example: * * _align_sbrk(10,0,0); // align to next 1k boundary */ _align_sbrk(int align, int offset, report); This is currently used on startup by Lua programs that can experience a lockup when loading libraries - this appears to be a Lua bug to do with the alignment of the system break. The following code resolve the bug in all currently known instances: /* align sbrk to 2k boundary - Lua needs this! */ _align_sbrk(11,0,0); Other Changes ------------- 1. The Serial 2 plugin was not being initialized correctly when the first function called was s2_rxcount(). In such a case the function would never return. The s2_rxcount() function worked correctly as long as it was NOT the first function called. Affected the Propeller 2 only. 2. The definition of the LOCKNEW macro in prop.h (aka propeller.h) may have led to symbol "lock" being declared undefined because the macro used it as a parameter to the _locknew() function which is neither required nor used. Affected the Propeller 1 and Propeller 2. 3. The definition of the _lockrel() function in prop2.h (aka propeller2.h) was incorrect - it defined the function as returning an int when in fact it does not return any value. Affected the Propeller 2 only. 4. The filename completion function in Catalyst would complete commands in the BIN directory without including the "/BIN/" prefix. If the intent was just to execute the command this worked ok, but if the intent was to process the file (e.g. if the command was a Lua script to be edited) then the completed filename was not correct and the file would not be found. Affected the Propeller 2 only. 5. The information in the 'Catalina Reference Manual (Propeller 2)' document regarding the configuration of the 2 port and 8 port serial plugins and how this may interact with the TTY and SIMPLE HMI options was out of date and has been updated. Affected the Propeller 2 only. 6. The size of the Openspin symbol table has been doubled (from 16384 symbols to 32768 symbols) as it was possible that some programs exceeded the original capacity. Affected the Propeller 1 only. 7. Some of the dynamic kernels were not correctly setting up the first thread block, which led to lockups when these kernels were used by programs that used pthreads, including some multi-processing Lua programs. Affected the Propeller 1 and Propeller 2. 8. The Catalina_FAQ.txt document was out of date. It has been removed.Ross.
The SourceForge release of Catalina 8.1 is now available.
Ross.
A quick note about using Catapult to compile multi-model programs on Linux vs Windows ...
When you first compile and run a multi-model demo program (such as the recently released eLua) using Linux and then execute it, you may get a message like:
Error: secondary client must specify address(0x00023AD8)This is Catapult telling you to adjust the address used for the secondary program. If you see such a message, find the secondary pragma in the appropriate C source file and amend it. It will look something like this:
#pragma catapult secondary client(shared_data_t) address(0x23B3C) mode(CMM) stack(100000) options(-lluax -lthreads xinit.c)In this case (demos/elua/aloha/eluax.c) I changed the address to
0x23AD8and the program then compiled and executed correctly.For some reason the Catalina Optimizer gives slightly different results on Linux than on Windows, and all the addresses in the demo files in the release were generated for Windows. The programs works correctly compiled under Linux once the address adjustment is made.
I think the difference is due to awka working differently on Linux vs Windows, but I only just noticed it and I need to investigate further.
UPDATE: No, its not the Optimizer. Nor is it awka In fact, it's not Catalina at all. The difference is way deeper than that. The most likely candidate is the different versions of bison used to process the LCC grammar on Windows and Linux. The difference is enough to occasionally cause the LCC code generator to allocate different registers to use. Not wrong, just different. But this can then result in different code sizes when registers have to be spilled and restored. Which means the Optimizer can sometimes perform optimizations on Windows that it cannot do on Linux. Or vice-versa. Which (eventually) means secondary programs may have to be compiled for a different address on Windows than Linux. Both programs work properly - they are just different. And since the version of bison used is outside my control, there is probably not much I can do about it.
I'll add a note in the next release.
Ross.
Catalina 8.1.1 has been released on GitHub and SourceForge
The main enhancement is better defaults for the Lua garbage collector, which improves Lua itself as well as all the Lua derivatives (mLua, iLua and eLua).
Here is the relevant extract from the README.TXT
There is a bug in the Linux build_all script for elua/ALOHA (i.e. demos\elua\aloha\build_all).
Line 28 currently says:
mv eluax.bin slave.binIt should say:
mv eluax.bin slavex.binThere is also a bug in the Windows script (i.e. demos\elua\aloha\build_all,bat) - it says mv when it should say move /Y on lines 14, 16, 19 & 21.
I've fixed these on GitHub, but not yet on SourceForge.
Catalina 8.1.2 has been released on GitHub and SourceForge
This is a full release, but it has only one significant enhancement - the ability for eLua to execute both clients and servers in NATIVE mode from Hub RAM.
Here is the relevant extract from the README.TXT
RELEASE 8.1.2 New Functionality ----------------- 1. The default garbage collection for all Lua programs (including Lua itself, iLua, mLua and eLua) now has a "pause" value of 105, which means it will start a new cycle when the use of memory exceeds 105% of the use after the previous collection. This makes the garbage collector even more aggressive, which reduces the allocation of dynamic memory at the cost of some performance. 2. eLua now has two additional variants that include ALOHA support: eluafx - implements an XMM server and a NATIVE client sluafx - implements a NATIVE server but no client (and so must be used in conjunction with a remote ALOHA client, which could be elua, eluax, or eluafx) The 'Aloha from Lua' document has been updated to describe these variants, which (when used together) allow both the client and the server to execute entirely in NATIVE mode from Hub RAM, using two different propellers. This offers the fastest possible client/server execution option, achieving significantly better performance than any single propeller option. Other Changes ------------- 1. Fix issues in the build_all scripts in the eLua/ALOHA demo folder (i.e. /demos/elua/aloha/build_all and /demos/elua/aloha/build_all.bat). Affected both Windows and Linux.Just pushed a minor update to GitHub to add some new eLua/ALOHA variants.
Here is an extract from the updated README.TXT in demos/elua/aloha:
The ALOHA extensions to eLua are described in the document 'Aloha from Lua'. See that document for more details. Mentioned in that document is a variant of a slave ALOHA server that omits the slave client altogether, called sluafx (in the file sluafx.c). This allows the server to execute entirely from Hub RAM. When sluafx is used on the slave propeller in conjunction with masterx or masterfx on the master propeller, it means both the client and server can be executed entirely from Hub RAM, for a significant speed improvement over the same program executed from XMM RAM. Not mentioned in that document are the other possible variants of an ALOHA enabled Lua server without a client that execute entirely from Hub RAM. The main purpose of these variants is to maximize the Hub RAM available to the Lua server. In addition to the sluafx.c server already mentioned, the following variants are also provided - they differ only in Catapult pragma options: sluafix.c - Similar to sluafx.c, but makes more Hub RAM available by using an integer-only version of the Lua initialization module (i.e. iinit.c instead of xinit.c) which does NOT load the Lua maths module. It also uses an updated version of the binser module that does not require the math module. The binary is also linked with the integer version of the extended Catalina library (i.e. -lcix instead of -lcx). Lua programs executed using this server can declare and use floating point types and do basic floating point operations, but they cannot use the functions provided by the maths module, nor can they do I/O on any floating point types (e.g. using print). sluacx.c - As sluafx.c but using COMPACT mode. This is slower than NATIVE mode but it allows larger Lua programs to be executed from Hub RAM, so it is still faster than an XMM sever. sluacix.c - As sluafix.c but using COMPACT mode. These variants will be built by the 'build_all' script for the P2_SLAVE propeller (they are only built for execution on 'slave' propellers), or they can be built individually using commands such as: set CATALINA_DEFINE=P2_SLAVE catapult sluafix.c As an example of the use of these variants, when the client/server version of the Game of Life (in the folder demos/elua/life) is executed with masterfx and sluafx, it can support a universe of up to 80 columns by 30 rows (edit the file common.lua and recompile it to change the size of the universe): On the P2_MASTER: masterfx client.lux remote.lux On the P2_SLAVE: sluafx When executed using masterfx and sluafix, it can support a universe of up to 80 columns by 40 rows: On the P2_MASTER: masterfx client.lux remote.lux On the P2_SLAVE: sluafix When executed using masterfx and sluacix, it can support a universe of 80 columns by 60 rows - but will execute more slowly: On the P2_MASTER: masterfx client.lux remote.lux On the P2_SLAVE: sluacixI have a question for SourceForge and GitHub users ... Are there any Linux users who use SourceForge, or any Windows users who use GitHub?
When I only released Catalina on SourceForge I used to see about the same number of Linux and Windows downloads. Since I began releasing on both SourceForge and GitHub I get about the same numbers of Windows downloads on SourceForge, but close to zero Linux downloads. I now get about the same number of GitHub downloads as I used to get for Linux on SourceForge. But GitHub doesn't tell me whether these are Linux or Windows users.
I could save work on future releases by not bothering with Linux binaries on SourceForge, or Windows binaries on GitHub.
Thanks in advance!
Ross.
As a Linux user, I'd never go through the hassle of downloading and unzipping a zip file to update something when I could just do
git pull. If I see that a project is hosted on both GitHub and Sourceforge, I'll always check GitHub first because people too often don't have a git repo on Sourceforge, and because GitHub's website is so much less clunky (and I can ignore it and just use the Git client).Why bother manually uploading releases anywhere when GitHub Actions makes building a release a simple matter of tagging a release commit and pushing the tag, and letting GitHub do the rest? You could probably even get them automatically uploaded to SourceForge - then having all builds on both sites wouldn't be any more work.
Or you could just ditch SourceForge entirely, since there's nothing it can do that GitHub can't do better.
Thanks for your response.
For Linux I could probably automate the build using GitHub Actions myself fairly easily, but only because it is almost completely trivial (one bash script does the whole job, and it takes only a few minutes). I may have a go at this, but to be honest I can't see any real advantage in doing so.
But Windows is a whole 'nother beast. First of all, it takes hours - this is the main reason I have to provide Windows binaries**. Also, maybe it's changed now (I will check again) but last time I looked using MinGW in GitHub Actions was known to be problematic. Since I am not already familiar with GitHub Actions, the thought of going through all the effort to learn about it only to find it still has issues is a little off-putting. I'd rather spend the time actually developing Catalina.
Still, if anyone wanted to have a go at this, I'd be happy to assist - there is a document in the release (BUILD.TXT) that describes in detail how to build Catalina under Windows. Perhaps someone with GitHub Actions experience could try turning that into a workflow?
Ross.
** I have never really understood this. Compiling Catalina and all its demo programs under Windows takes hours, but compiling it using Linux takes only minutes. Using a virtual machine running on the same hardware. Under the same version of Windows.
On Windows NT, especially if you have any sort of security software enabled, spawning a new process and attaching it to the console subsystem is comically slow. Reading directory information is also slow. This can combine into a make+gcc style build taking orders of magnitude longer than reasonable.
Using MinGW in GH Actions is perfectly fine (assuming use as a cross-compiler on a Linux action runner), I set it up for flexspin/spin2cpp.
The real big thing with actions IMO is that you can use it to auto-run any test scripts you have. For flexspin I made a modified version of spinsim, so P1 code can be tested end-to-end.