I could have predicted this would happen as soon as I finished the release. It always seems to!
I wondered why one of my Lua example programs (ex9.lua) seemed to need much more stack space than the other examples to execute reliably, even though it didn't really do much more. I never could figure out why, so I also used it as an example of how Lua's garbage collector can help when memory gets tight. But it turns out this example does not need the larger stack at all - it was apparently the result of an obscure bug in Lua itself, which I seem to have now fixed quite by chance while working on something else.
There are no functional changes to either Lua or the new multi-threading module, and in any case I will probably leave the example intact (except for the stack size) as an example of managing Lua's garbage collector, but I'll wait to see if anything else crops up before I re-package release 5.0 (or release 5.1 if I find anything more serious).
This release adds no significant new functionality, but it fixes one major Lua issue and a couple of other minor issues.
Here is the relevant part of the README.TXT:
RELEASE 5.0.1
New Functionality
-----------------
1. Modified various Catalyst commands to allow combined command-line
options - e.g. -dh now means the same as -d -h
Other Changes
-------------
1. Fixed a bug in the DOSFS file system which allowed a file to be opened
as a directory using the DFS_OPENDIR option to DFS_OpenFile instead of
returning that a directory of that name did not exist. This confused
the Catalyst cp command when copying files into a directory.
2. Fixed a bug in the Catalyst rm command that prevented recursive file
deletions.
3. Fixed a bug in the Lua threads module that prevented the number of
factories being reduced. Also, the Lua threads module documentation
has been updated to reflect the fact that reducing the number of
factories now necessarily terminates all workers first (they are
restarted after the number of factories has been reduced) and that
this should therefore only be done from the main thread and when that
thread is executing on factory 1.
4. Modified Lua to zero all newly allocated blocks of memory. Lua seems to
assume all new blocks of memory will be initialized to zero, which is
not actually guaranteed by C, so this led to some Lua programs behaving
unexpectedly. While this fix resolves the issue, it does so at a small
run-time cost, so further investigation is required. Multi-threaded Lua
example 9 was badly affected by this issue and has been updated. The
Lua threads module documentation has also been updated.
I have also updated the version of Multi-threaded Lua pre-compiled for the P2 Evaluation board which is attached to an earlier post.
As usual, I wish I had thought to do this before the release!
Now that I have fixed Lua's memory issue, I thought I'd try compiling Multi-threaded Lua in NATIVE mode rather than COMPACT mode. Doing so means there is less memory available for Lua programs, but the programs will execute much faster. The command I used was:
build_all P2_EVAL SIMPLE VT100 OPTIMIZE
Here are the results for the 10 tutorial examples:
ex1.lua, ex2.lua, ex3.lua, ex5.lua, ex7.lua, ex10.lua : All good. Can be executed from source (using mlua), or compiled (using lauc and mluax).
ex4.lua : The increased execution speed highlights that the original program has a race condition. Adding a small delay to the ping and pong functions fixes the problem:
function ping()
t = threads
for i = 1, t.shared("count") do
ball = t.get("ping")
t.output("ping ")
t.msleep(10) -- DELAY ADDED
t.aput("pong", ball)
end
end
function pong()
t = threads
for i = 1, t.shared("count") do
ball = t.get("pong")
t.output("pong\n")
t.msleep(10) -- DELAY ADDED
t.aput("ping", ball)
end
end
ex6.lua : Must now be compiled. Otherwise, all good.
ex8.lua : There is only enough memory available to execute 6 workers concurrently (not 12) and the increased execution speed is such that a small delay needs to be added to the Thread function to illustrate that all the workers are indeed executing in parallel:
count = 6
function Thread(me, iter)
return function()
t = threads
-- wait till we are told to go
repeat until t.shared("go")
repeat
t.output(me .. " ")
t.msleep(10) -- DELAY ADDED
iter = iter-1
until iter == 0
end
end
ex9.lua : There is only enough memory available to recycle up to 2 workers, not all 4:
t.recycle(2) -- WAS 4
Other than example 9, these programs are too small and simple to show much improvement in execution speed ... except (in strict accordance with Murphey's Law) where it highlights a problem in my example programs!
There is an interesting synergy between Lua co-routines and threads. I will add this to the threads documentation in the next release, but the example program below can be executed with the current release, so I thought I would post a preview here because it shows what a "rich" language Lua is ...
Even without the threads module, Lua has a simple type of multi-tasking. It has non-preemptive multi-tasking in the form of co-routines. The threads module adds preemptive multi-tasking. However, this has interesting implications when co-routines are executed using the threads module.
Here is a simple example:
-- Generate a function which can be executed
-- as either a coroutine or as a thread
function Generate(me)
return function ()
for i = 1, 10 do
threads.print(me ..":" .. i)
threads.msleep(100)
coroutine.yield()
end
end
end
-- Execute the functions as coroutines
threads.print("\nFUNCTIONS AS COROUTINES")
-- create 4 functions as coroutines and also
-- wrap the coroutines inside another function
fee = coroutine.wrap(Generate("fee"))
fi = coroutine.wrap(Generate("fi"))
fo = coroutine.wrap(Generate("fo"))
fum = coroutine.wrap(Generate("fum"))
-- execute the coroutines
fee()
fi()
fo()
fum()
-- and again
fee()
fi()
fo()
fum()
-- Execute the functions as threads
threads.print("\nFUNCTIONS AS THREADS")
-- we only need one worker to execute any number of
-- threads if those threads explicitly "yield"
threads.workers(1)
-- create (and start) 4 functions as threads
threads.new(Generate("fee"))
threads.new(Generate("fi"))
threads.new(Generate("fo"))
threads.new(Generate("fum"))
-- wait for all threads to complete
threads.wait()
When the functions are executed as co-routines, then each time the function is called, it executes one iteration and then "yields". This "yield" call is the key to co-routine behavior. It returns control to the caller, but "suspends" the co-routine rather than exiting it. When the function is called again, it does not start from the beginning as a normal function would, it instead continues execution from where it was last suspended - i.e. just after the last "yield". This is normal co-routine behavior (for more details on co-routines, see the Lua documentation).
The interesting thing is the consequence of calling "yield" when the functions are executed as threads instead of co-routines. When a thread yields then the thread is suspended, but the worker executing that thread is not - it moves on to resume execution of the next suspended thread. If there is no other suspended thread it will resume execution of the same thread. Since we started all our functions as threads, this means that one worker (which is one Posix thread) will execute all the functions concurrently (using non preemptive multitasking) provided those functions are written as co-routines. If the functions are not written as co-routines, then you need multiple workers available to execute them concurrently (using preemptive multitasking). Which of course is very easily done in this example - just change the number of workers from 1 to 4. You can try it, but in this simple example it makes little or no obvious difference to the output.
In summary, with 1 worker the program uses non preemptive multitasking. With 4 workers it uses fully preemptive multitasking. With 2 or 3 workers, it would use a mixture of both.
This release is a patch release for both Windows and Linux. It must be installed over Catalina 5.0.1. It adds no new functionality, but it fixes one Lua issue and adds a couple of new Lua examples. If you are not interested in Lua you don't need it - the changes will also be included in the next full release.
Here is the relevant part of the README.TXT:
RELEASE 5.0.2
New Functionality
-----------------
This release contains no new functionality.
Other Changes
-------------
1. Fixed a bug in the Lua threads module when sending and receiving messages.
The program could lock up under certain circumstances. Since the Lua
threads module is only supported on the Propeller 2, this affected the
Propeller 2 only.
2. Multi-threaded Lua examples 4 has been updated to eliminate a race
condition.
3. Multi-threaded Lua example 8 has been updated to make it more evident the
example is using preemptive multi-tasking.
4. Multi-threaded Lua examples 11 and 12 have been added to demonstrate
interactions between co-routines and threads.
5. The document Lua on the Propeller 2 with Catalina has been updated to
reflect the above changes. Also, some terminology issues have been
clarified.
I have also updated the version of Lua pre-compiled for the P2 Evaluation board which is attached to an earlier post.
Interesting. Now that Lua 5.1 seems rock solid (yes, I realize I'm tempting fate here ) I have been re-looking at Lua 5.4, which is the latest and greatest version.
When I looked at it a while ago, the first thing I noticed was that the initial code/data footprint of Lua 5.4 is much larger than that of Lua 5.1 - by about 40kb even in COMPACT mode. This was enough for me to stay with 5.1 as the default version even though 5.4 seemed to have some useful improvements.
But it turns out I didn't look deeply enough - yes, the initial footprint of Lua itself is larger, but the footprint of each concurrent thread is smaller - by about 5kb. This means that if you have 8 or more workers (as you might if you have 8 cogs!), then Lua 5.4 actually ends up using less memory than Lua 5.1.
Only one of the tutorial examples (ex8.lua) actually creates this many workers - this particular example is specifically designed to see how many workers it can execute concurrently. It turns out that you can execute more workers with Lua 5.4 than you can with Lua 5.1 - something I had not thought to check!
As well as better memory management, Lua 5.4 also includes integer numeric types (like some versions of Basic, Lua originally only had floats) and bitwise operators, which makes it much more suitable for the type of applications common on the Propeller.
Catalina will continue to support all versions of Lua (from 5.1 onwards) but unless I find a showstopper, I am likely to switch the default to Lua 5.4 in the next release.
I can't believe I missed this! I just found a version of the classic startrek game ported to Lua! I already had C, Pascal and Basic versions included with Catalina, but I had never thought to even look for a Lua version. It seems I am not the only one that remembers this game very fondly!
In the attached zip file are two Lua versions that I have modified very slightly, because the originals were written before Lua 5.4 was released, and the math library has changed slightly. They also needed a few tweaks to the garbage collector to stop them running out of memory on the Propeller. One version is a faithful port of the original 1978 game (sst.lua) and the other is an "enhanced" version (sst-tos.lua) that incorporates dialog taken from the original scripts.
I have not done extensive testing, but I have played them both for a few minutes each and they both seem to run ok when compiled with luac and executed using mluax.
For instance, in Catalyst, you would use commands like:
I have been testing Catalina on the new P2 Edge module, and can confirm that Catalina works ok on it.
However, while testing various things I found a bug in Catalina's SD Card plugin that may have prevented programs from being able to write to some types of SD Cards (reading was ok). I will include the updated version of the plugin with the next release of Catalina, but I have also attached it here - it can simply be copied over the current version in the target_p2 directory. Of course, you will have to recompile any programs that use it.
I have just posted Catalina 5.1 to SourceForge here.
I am not getting much time for Catalina at the moment so there are not many changes, but the new SD Card plugin is included so I thought I had better do a release.
Here is the relevant extract from the README.TXT:
RELEASE 5.1
New Functionality
-----------------
1. Decoding the CATALINA_DEFINE environment variable was taking place AFTER
the symbols to specify the clock frequency were defined, so the Catalina
symbols MHZ_220, MHZ_260 & MHZ_300 only worked when specified on the
command line, and had no effect if specified using CATALINA_DEFINE.
2. Added MHZ_200 as a Catalina symbol, to make it easier to build Catalyst
using 200Mhz on the Propeller 2. Note that the Propeller 2 reference
manual, the tutorial Getting Started with Catalina and the documentation
in the Propeller 2 VGA plugin source file all mistakenly referred to
MHZ_200 when they should have referred to the existing MHZ_220 symbol.
All these documents have now been updated.
3. The Lua "propeller" module has had the functions sleep, msleep and sbrk
added. These functions are the same as the ones in the "threads" module,
but are handy when the propeller module is used in a non-threaded Lua
program.
4. The Lua "threads" and "propeller" modules now accepts a string parameter.
If "lua" is specified, these functions return the version of Lua.
Otherwise they return the version of the module
5. Lua versions of the classic Star Trek game (sst.lua and sst-tos.lua)
are now included as Lua test programs.
6. A change was made to the gettoken procedure in Dumbo Basic 1.0, which was
introduced in Catalina release 4.7. This change led to a syntax error when
executing TREK15.BAS. This change has been reverted in Dumbo Basic 1.1, but
if this breaks any existing basic code, it can be re-implemented by
modifying the file basic.c and changing the #define of NEW_GETTOKEN to 1.
7. When compiled for the Propeller, Dumbo Basic now has a delay on exit of
100ms to allow time for any pending output to be printed. This prevents
error messages or final program output from being truncated.
Other Changes
-------------
1. The command-line options to set the clock frequency (-f, -F and -E) were
fully implemented, but not described in the documentation. These options
apply to the Propeller 2 only.
2. On the Propeller 2, it is now recommended that Catalyst be compiled using
a 200Mhz clock speed. This is a work-around for an obscure bug which only
seems to affect some programs loaded from the SD card and which use a clock
speed of 180Mhz.
3. A bug in the SD Card plugin was preventing programs being able to write to
some SD cards (reading was ok). This affected the Propeller 2 only.
I have just posted Catalina 5.1.1 to SourceForge here.
This is a very small patch release for the Propeller 1 only, mostly to fix a silly bug that was introduced in Catalina 4.9.3.
This patch release must be installed over an existing installation of Catalina 5.1. Here is an extract of the README.TXT file:
RELEASE 5.1.1
New Functionality
-----------------
1. Added a new demo folder ("xeprom") containing programs that demonstrate how
to read from the EEPROM from XEPROM XMM programs. In such programs, the
EEPROM cannot be read independently (e.g. using an I2C driver) because it
is constantly in use by the cache. This applies only to the Propeller 1,
and requires a platform with an EEPROM larger than 32k, such as a FLIP, a
C3 or a HYDRA.
Other Changes
-------------
1. Fixed a bug that meant the -e command line switch to Catalina, which is
used to specify that a ".eeprom" file be generated instead of a ".binary"
file did not work correctly in some instances. In particular, it could not
be used in conjunction with the -C XEPROM option. This affected the
Propeller 1 only, and only Catalina version 4.9.3 or later.
2. The catalina_clean utility script had not been updated recently, and
would miss cleaning some directories when the optional "all" parameter
was specified.
I have just posted Catalina 5.1.2 to SourceForge here.
This is another small patch release, mostly to fix another silly Propeller 1 bug that was introduced in Catalina 5.0, but it also incorporates a few changes to Lua that affect the Propeller 2.
This patch release can be installed over an existing installation of Catalina 5.1 or Catalina 5.1.1. Here is an extract of the README.TXT file:
RELEASE 5.1.2
New Functionality
-----------------
1. The path that Lua uses to search for modules has changed. The old path
was the one used on Windows, which was not appropriate for the Propeller.
The path is now:
"?.lua;?/init.lua;lua/?.lua;lua/?/init.lua"
This means that (for example) if you say "require 'm'" then Lua will
search for "m.lua", "m/init.lua", "lua/m.lua" and "lua/m/init.lua"
in that order.
2. The Lua demos sst.lua and sst-tos.lua now use "propeller.sbrk()" instead of
"threads.sbrk()", which means the mem command (which displays the top of
the C heap) now works when the program is run under both lua and mlua.
Other Changes
-------------
1. Fixed a bug in the threads library, which prevented some multithreaded
programs from compiling. Affected only Propeller 1 programs which were
compiled in LMM TINY mode.
2. Eliminated the file minit.c from Lua. This was a temporary replacement
for the Lua file linit.c which included the threads module. Now linit.c
will include the "threads" module if LUA_THREADS is defined in the file
luaconf.h. For completeness, there is also a corresponding LUA_PROPELLER
that is used to specify that the "propeller" module be included. This is
more in keeping with the way Lua normally defines compile-time options,
eliminates the need to have the extra file, and it also means that when
compiling the executables (e.g. lua and mlua or luax and mluax) under
Catalina, Catalina can automatically detect the compile options and
include the correct modules. But if you compile mlua or mluax using gcc
and posix threads, you will now need to explicitly define LUA_THREADS in
luaconf.h to include the "threads" module.
3. Fixed a bug that meant the Lua "propeller" module would not compile using
any compiler other than Catalina. Although it is only functional when
compiled using Catalina, it should still have compiled using other
compilers (such as gcc).
4. The number of threads the "test_maximum_threads.c" demo program creates
has been reduced to 145, since at 150 there are some plugin combinations
that do not have enough free Hub RAM to support 150 threads. Affected
the Propeller 1 only.
5. The build_all scripts and Makefiles now always build Lua in COMPACT mode
on the Propeller 2, because NATIVE mode does not leave enough Hub RAM
for the various demo programs.
I have to apologize for the recent patch releases. I have been working almost exclusively on P2 and Lua support recently, and I had not bothered to run the validation suite before the releases - if I had done so the validation suite would have picked the problems up straight away. I will do so before future releases
I have just posted Catalina 5.2 to SourceForge here.
This is a full release, primarily to update Catalyst to add some outstanding features that were always intended to be there, but I had never bothered to finish - like wildcard support for various Catalyst commands (i.e. ls, cp mv, rm, cat).
I have become a real fan of Lua, and I tend to mainly use my P2 as a stand-alone Lua development platform these days. These additions make developing Lua software on the P2 for the P2 very much easier ... as long as you are also a fan of the vi text editor. Let the editor flame wars commence!
Here is an extract of the README.TXT file:
RELEASE 5.2
New Functionality
-----------------
1. Added a new inline pasm demo ("test_inline_pasm_5.c") to demonstrate how
to CALL an inline PASM function from within an inline PASM function in a
COMPACT program.
2. Updated the validation suite. Now logs more details to make it easier to
identify what failed, and also added "short" scripts to do faster
validation.
3. Updated the file 'globbing' demo (in folder "demos\globbing"). Tidied it
up, fixed a few bugs and it now uses the correct DOS filename syntax,
prints the filenames properly, and is now correctly case insensitive in
all cases. The listDir() function has been re-written to be an instance
of a new generic doDir() function, which calls a specified function on
each matching filename.
4. Wildcard matching ('globbing') has been added to many of the Catalyst
utilities. You can now specify wildcard expressions like *.bin or
[a-g]*.b?? - see "glob.c" for details. The options and syntax of the
utilities have not changed, except for 'ls' (see 5 below).
The commands affected are:
ls - list files and/or directories
rm - delete files or directories
cp - copy files
mv - move files
cat - concatenate and print files
For example, you can now say things like:
ls *.bin *.dat
mv [a-f]*.bin bin
rm ???.dat
cat *.txt
Note that wildcards can only be used in the file name part of a path, not
in the directory part, so you cannot specify an argument like "/b??/*.*".
For arguments that may be interpreted as files or directories, adding a
trailing "/" ensures they will be treated as directories. For example:
ls bin -- list just the entry "bin" (if it exists)
ls bin/ -- list the contents of directory "bin" (if it exists)
5. The Catalyst "ls" command now uses a short listing format (similar to
that used by Unix or the built-in "dir" command) by default. The previous
default listing format can be specified using the "-l" option, and the
previous long listing format (which used to be specified by "-l") must
now be specified by "-l -l" or "-ll". Also fixed interactive mode, which
was not working in some cases.
6. The Catalyst "cp", "mv" and "rm" commands now accept 'a' or 'A' (for "all")
at the interactive prompt, to indicate that the command should assume "yes"
for all subsequent files.
7. The catalyst "build_all" script now does "clean_all" before building, to
ensure there are no binaries left from previous builds.
Other Changes
-------------
1. Updated the file "catalina_compact.inc" in the target and target_p2
directories to add more notes about writing inline PASM in COMPACT
programs.
2. The tiny library had a version of the C stdio function gets(). This
prevented programs that used gets() from linking with the tiny library,
because it ended up being multiply defined. The version in the tiny
library has been renamed tiny_gets() in this library, but like the other
tiny functions (i.e. tiny_printf() etc) it will be used automatically if
a program is linked with the tiny library. Affected the Propeller 1 and
Propeller 2.
3. Fixed a bug in the Propeller 1 SD Card plugin - multiple block writes
must not be used on platforms that must disable the SD Card to use XMM
RAM (e.g. because they share pins). Affected the Propeller 1 only.
4. Fixed a bug in Catalina's "unmanaged" file close function which meant that
only a limited number of files could ever be opened. Did not affect the
normal stdio open/close functions, but it affected the Catalyst "cat"
command, which uses the unmanaged versions of open/close to save space.
5. Catalyst's example basic files are now all Unix format, not DOS format.
This makes no difference to their functionality, but makes them easier
to edit usng "vi" or list using "cat".
6. Fixed a bug in DOSFS which affected the ability to delete files with
zero length (e.g. using the Catalyst "rm" command) - doing so may
have corrupted the file system.
While it can be used with any Propeller programs, Catalyst is specifically intended to
facilitate loading and using programs compiled with the Catalina C compiler.
It is not currently possible to compile Catalina C programs on the Propeller, but
Catalyst comes with various other applications that can be used for self-hosted
Propeller development, including the vi text editor, and such tools as a BASIC
interpreter, a Pascal compiler and interpreter, and the Lua scripting language. It
could also be use to edit, compile and then run SPIN programs using with the Sphinx
SPIN compiler (not included – see http://www.sphinxcompiler.com/).
Looking up Spinx I see it's been around since 2009. So self-hosted Spin programming has been an option for a while then. I've not seen anyone mention this before when the topic of self-hosting gets its regular airing.
No, it's not self-hosting because you cannot compile the Lua compiler itself on the Propeller - you have to compile that using Catalina. There is a version of Lua written entirely in Lua, but I think it would be too large to compile on the P2, and too slow to compile on the P1 even if you had megabytes of XMM RAM available. I have not tried it, and I don't intend to do so - I've never understood why people are so obsessed with self-hosted development on a microcontroller, when you can buy a full-blown computer that will do the job better and cheaper.
Still, Catalyst is fine for developing and executing Lua programs on the Propeller, and it's a fun way to learn Lua!
I've not been a big follower myself but Chip encourages it and there have been a number of requests for self-hosting a Spin development environment. It occured me, reading you release notes, that it's probably not that hard to be achieved using Catalyst + Spinx.
Why is it that as soon as you do a release, you notice a stupid little problem ... Murphey's Law, I guess ...
One of the Lua demos - fact.lua - was written when all Lua numbers were floating point. Recent versions of Lua have both floats and integers, but it is still an untyped language - so an expression like n = 1 makes n an integer, whereas n = 1.0 makes n a floating point.
So fact.lua should now look like this, with one minor change on line 18:
-- function closures are powerful
-- traditional fixed-point operator from functional programming
Y = function (g)
local a = function (f) return f(f) end
return a(function (f)
return g(function (x)
local c=f(f)
return c(x)
end)
end)
end
-- factorial without recursion
F = function (f)
return function (n)
if n == 0 then return 1.0 -- <<< changed 1 to 1.0 to return a float instead of an int
else return n*f(n-1) end
end
end
factorial = Y(F) -- factorial is the fixed point of F
-- now test it
function test(x)
io.write(x,"! = ",factorial(x),"\n")
end
for n=0,16 do
test(n)
end
-- added for Catalina/Catalyst, which may clear the screen on program exit
print("\nPress any key to continue")
any = io.stdin:read()
I'll leave it to you to figure out how this program actually works, and how it can claim to be non-recursive.
Lua looks a lot like any other procedural language (Pascal, Modula, C, C++ etc) - but it is very much not. It is so expressive and powerful that it can seriously do your head in. That's probably why I like it
@RossH said:
Lua looks a lot like any other procedural language (Pascal, Modula, C, C++ etc) - but it is very much not.
Well, without reading anything from the Web, first thing of note is the apparent function call to factorial(x) and that that is the one thing that isn't defined as a function.
@RossH said:
Lua looks a lot like any other procedural language (Pascal, Modula, C, C++ etc) - but it is very much not.
Well, without reading anything from the Web, first thing of note is the apparent function call to factorial(x) and that that is the one thing that isn't defined as a function.
Ok. here goes ... **spoiler alert! **...
It is best to start from the bottom ...
The line "factorial = Y(F)" tells you that factorial is what is returned from calling function Y with F as a parameter. But since Lua is an untyped language, we don't know what that is - at least not from just this line.
The lines "F = function(f) return function (n) ..." tells you that F is a function that accepts an argument f - and we learn a few lines down that f has to be a function (remember Lua is typeless - we only know f has to be a function because F uses it as such). But note that F itself does not call f - instead F uses f to generate another function, which accepts a parameter n - and that function would calculate factorials --- provided F was passed a function f that does exactly what the generated function does (big hint there!).
The lines "Y = function (g) ..." is where the magic happens. This tells us that Y is a function that accepts a parameter g, and it produces the "fixed point of g" - i.e. it calculates "fix g" = g(fix g) = g(g(fix g)) = ... etc ...
So, when "fix g" is F, Y returns a function that takes one numeric argument, and that function calculates the factorial of its numeric argument.
I warned you that Lua can do your head in! But it is a surprisingly small, clean and highly orthogonal language that can be used to do procedural programming, object-oriented programming, or (as in this case) functional programming.
I have seen 'uncle' Murphy pay a visit too many times when either there was an insane schedule to meet or there was a piece of equipment on the verge of death.
Another thing that I have found is that it's best to wait a day or two before doing your final checks, because when your mind is fresh mistakes and/or problems are easier to spot.
Another thing that I have found is that it's best to wait a day or two before doing your final checks, because when your mind is fresh mistakes and/or problems are easier to spot.
Good advice. However, I generally only have limited time to devote to this at the moment, so I have to get it out there while I can!
Good advice. However, I generally only have limited time to devote to this at the moment, so I have to get it out there while I can!
I myself prefer to take my time and focus on quality, because I have found that if I just try to get it done, then sooner or later I will have to fix it.
Attached is a preview of what will be coming in the next release of Catalina. If I had realized how easy this was going to be, I would have held up the last release for it, but no matter - I am sure I will find a few more annoying bugs that also need fixing.
I have decided to make Lua the scripting language for Catalyst itself (an obvious thing to do really, but I had just not bothered, because I had not realized both how easy and how useful it would be until I had learned more about Lua).
You could of course always invoke Lua manually, and you still can - but now Catalyst will look for Lua programs to execute the same way it executes any normal Propeller binaries, so it is much easier to extend Catalyst's basic functionality using Lua and then invoke these programs seamlessly from the command line as if they plain old Propeller binaries.
The attached file includes a working prototype of the new versions of both Catalyst and Lua, compiled for the P2 EDGE or P2 EVAL boards (you need an SD Card and a VT100 terminal emulator such as putty or payload). A version of payload is included in the attachment. If you use putty you may need to tweak its line termination options.
It includes three demos designed to highlight the new Lua functionality, and illustrate how easily Lua now integrates with Catalyst:
list.lua - a simple directory listing program (similar to Unix "ls").
find.lua - a simple file searching program (similar to Unix "find").
freq.lua - a simple word frequency counting program.
Load the programs onto an SD Card, set up your P2 to boot from it, and then use the following payload command (where X is the USB port connected to your P2):
payload -i -b230400 -pX
If you reboot your P2, you should see the Catalyst prompt:
Catalyst 5.2.1
>
As well as all the usual Catalyst commands, you can try the following new commands:
list <-- list file details of all files
list *.lua <-- list file details of Lua files
find *.bas PRINT <-- find PRINT statements in BASIC files
freq *.txt <-- count word frequency in all text files
To see how easy stuff like this is to do, here is a complete (but quite mimimalist) version of a directory listing program, implemented in Lua:
-- print the file name only
function print_entry(name, attr, size)
print(" " .. name)
end
-- mount the DOSFS volume
propeller.mount()
-- scan the DOSFS directory specified in arg[1] and call print_entry
-- on any files matching the wildcard pattern specified in arg[2]
propeller.scan(print_entry, arg[1], arg[2])
The real version (in list.lua) has a few more bells and whistles, but is basically the same program.
You may notice that this program uses two new functions that have been added to the Lua propeller module:
mount - mount a DOSFS file system
scan - scan the mounted DOSFS file system and execute a Lua function on each matching file
This is using essentially the same wildcard 'globbing' functionality now used in the new Catalyst utilities (ls, mv, cp, rm etc), and it makes it especially easy to write new file processing utilities in Lua.
So, is it worth rewriting all the existing Catalyst utilities (e.g. ls) as Lua scripts?
Well, in short "no". The C versions are faster and can run on any Propeller - 1 or 2, whereas the Lua versions could only run on a Propeller 2, or on a Propeller 1 with sufficient XMM RAM. And while loading Lua each time you need it does work ok on a Propeller 1, on some platforms - especially those that use FLASH memory as XMM RAM - it is painfully slow!
Lua is fun!
EDIT: Updated freq.lua in the attached zip file (to add garbage collection).
EDIT: I will leave this zip file here, because it is a stand-alone demo - but a slightly more advanced version of the same functionality has now been released as part of Catalina 5.3, so it is generally recommended to download that instead.
I have just posted Catalina 5.3 to SourceForge here.
This is a full release, primarily to update Catalyst to add Lua scripting capabilities.
Lua can now be used as a simple method of extending Catalyst functionality, as well as being a powerful high-level language for developing programs that can exploit the Propeller's unique capabilities.
Lua provides the kind of capabilities in a high-level language that I wanted the Propeller to have from day 1, but which were only previously available from low-level languages like Spin, Forth, C and C++.
Here is an extract of the README.TXT file:
RELEASE 5.3
New Functionality
-----------------
1. The Catalyst utilities cp, mv, rm, ls and cat now ignore volume id entries,
except that the ls utility will still include them when long or very long
listing output is specified.
2. Catalyst now has the ability to execute a once-only command on startup.
This is similar to the existing AUTOEXEC.TXT file processing when the
AUTODELETE option was also enabled. The AUTOEXEC.TXT and AUTODELETE
functionality has been retained, but AUTODELETE is not enabled by default,
which means that normally the AUTOEXEC.TXT is executed on every Propeller
reboot. The new once-only execution functionality overrides this if the
file EXECONCE.TXT exists. If it does then it will be executed and also
deleted. This new functionality is used by the Lua propeller module
"execute" function, described in point 3 below. It requires that the
SD Card be writable.
3. The Lua propeller module now includes new propeller-specific functions.
Lua analogues of the C DOSFS wildcard/globbing functions:
mount()
scan(function, directory, filename)
The filename can include wildcards. See the Catalyst Reference Manual
for details, and see list.lua for an example of using these functions.
A Lua function to execute any Catalyst command (reboots the Propeller):
execute(command, filename)
For example:
propeller.execute("list *.bas")
The first parameter is the command (which may include parameters), and the
second (optional) parameter is the file name to write this command to.
This defaults to "EXECONCE.TXT", which means the command will only be
executed once, but it can be used to write the command to any file. For
instance:
propeller.execute("lua", "AUTOEXEC.TXT")
will cause the Propeller to execute "lua" on each reboot. To disable this
from within Lua, just delete the file by executing a Lua command like:
os.remove("AUTOEXEC.TXT")
Lua analogues of the keyboard HMI functions:
k_get, k_wait, k_new, k_ready, k_clear
Lua analogues of the screen HMI functions:
t_geometry, t_mode, t_setpos, t_getpos, t_scroll,
t_color, t_color_fg, t_color_bg
Lua analogues of the mouse HMI functions:
m_button, m_abs_x, m_abs_y, m_delta_x, m_delta_y, m_reset,
m_bound_limits, m_bound_scales, m_bound_x, m_bound_y
Each of these HMI functions accepts the same parameters and returns the
same values as their C counterparts. See the Catalina Reference Manual
for details.
Note that to save space, the mouse functions will not be included if the
Catalina symbol NO_MOUSE is defined when Lua is compiled (which it will be
if you use the "build_all" scripts to build Lua - to change this you can
edit the file Makefile.Catalina to remove the "-C NO_MOUSE" option).
You can detect whether mouse functions have been included from within
Lua itself by testing if any of them are nil, such as:
if (propeller.m_button) then
-- mouse functions have been included
else
-- no mouse functions
end
4. Catalyst will now try to execute commands as Lua scripts, in addition to
just executing binary files. The order of priority for Catalyst command
execution is now:
1. As a built-in command (e.g. "dir")
2. As a Lua script, adding a ".lua" extension if none is specified
3. As a binary file, using the command as the file name
4. As a binary file, adding the following extensions (in this order
if the command does not specify an extension):
.BIN
.BIX (propeller 2 only)
.XMM
.SMM
.LMM
Note that all command types accept command line arguments, including
Lua scripts. Several example Lua scripts are included:
list.lua - a simple directory listing program (similar to Unix "ls").
find.lua - a simple file searching program (similar to Unix "find").
freq.lua - a simple word frequency counting program.
You invoke Lua scripts from the Catalyst command line just like any other
command. For example:
list <-- list file details of all files
list *.lua <-- list file details of Lua files
find *.bas PRINT <-- find PRINT statements in all BASIC files
freq *.txt <-- count word frequency in all text files
Note that Lua scripts can be compiled to improve load and execution
times, but they should still have the extension ".lua".
As a consequence of this, the Lua versions of the Super Star Trek demo
programs have changed name, so that executing the command "sst" at the
Catalyst prompt will still invoke the C version ("sst.bin") rather than
now executing the Lua version (which used to be called "sst.lua"):
sst.lua --> now called star.lua
sst-tos.lua --> now called star-tos.lua
5. The Propeller 2 Catalina command line argument processing has been modified
to match that of the Propeller 1. On startup, all Catalina programs check
if CogStore is running. If it is, the program fetches any arguments stored
in it. Previously on the Propeller 2, if CogStore was not running then
Catalina would set argc to zero and argv[0] to NULL - but some C programs
expect argc to always be at least 1, so now if CogStore is not running
argc will be set to 1 and argv[0] will point to a string with the value
"null" (since without CogStore the real program name is not available).
This functionality was already implemented on the Propeller 1.
6. The wildcard/globbing doDir() function in storage.h and storage.c has been
extended to make it more useful - it now calls the file processing
function with the file size and the DOSFS file attributes of each file
in addition to the file name. One advantage of this is that programs
that use doDir() do not need to use DOSFS functions to retrieve the file
attributes themselves, and therefore only need to use standard C stdio
file functions.
7. The build_catalyst script now detects whether it is being run in the
current directory. If so, it builds catalyst in this directory,
otherwise it builds it in the demos\catalyst folder in the Catalina
installation tree (as it did in previous releases). The purpose of this
is that if you copy the Catalyst folder to your own local user directory,
you can build Catalyst locally and do not need to have write permission
for the Catalina installation tree. Note that the build_all script already
supported local builds, it was only the build_catalyst script that did
not.
8. The build_utilities script now detects whether it is being run in the
current directory. If so, it builds the utilities in this directory,
otherwise it builds them in the utilities folder in the Catalina
installation tree (as it did in previous releases). The purpose of this
is that if you copy the utilities folder to your own local user directory,
you can build the utilities locally and do not need to have write
permission for the Catalina installation tree. Also, the build_utilities
script has been modified to prompt whether the utilities binaries should
also be copied to Catalina's bin directory, or simply left in the current
directory. Copying the utilities to Catalina's bin directory is convenient
since payload looks for them there if it does not find them in the current
directory, but it requires write permission to the Catalina installation
tree. If you do not have this permission, you can now copy the utilities
folder to your own user directory, build the utilities there, then copy
the binaries to each directory from which programs will be loaded - which
may actually be a better solution if you have multiple Propeller platforms
or configurations which need different versions of the utilities. Note
that the utilities are required only for Propeller 1 platforms - there
are currently none needed for any of the supported Propeller 2 platforms.
9. The build_all script in the utilities folder has been removed (it was
deprecated quite a few releases ago). Use the build_utilities script
instead. The main difference is that build_all used to build binaries
for all CPUs of multi-CPU platforms (such as the TRIBLADEPROP) whereas
the build_utilities script has to be re-run for each CPU.
Other Changes
-------------
1. Implemented a workaround for an OpenSpin/Spinnaker bug that meant compiling
Spin programs with too many short symbol names could fail unexpectedly. The
current workaround is to simply use longer symbol names. Affected only
Propeller 1 programs that used the Catalina Optimizer (which may generate
many additional symbol names).
2. Fixed a typo in the DOSFS demo program that prevented it from compiling.
3. Updated the version of Lua used in payload and blackbox to Lua 5.4.4,
which is the current version. Lua 5.1.5 is still included as a Catalyst
demo program, but is no longer compiled by default (Lua 5.4.4 is now used
everywhere) and Lua 5.1.5 is now deprecated and may be removed from a
future release.
4. Since Lua is now compiled in COMPACT mode by default, the pre-compiled P2
demo versions of Catalyst (in P2_EVAL.ZIP and P2_VGA.ZIP) are now compiled
in NATIVE mode, which improves execution speed, but at the expense of
larger executables for some of the demo programs. If this is a problem,
simply recompile Catalyst, specifying COMPACT mode as one of the
parameters to the "build_all" script.
5. Updated the notes about compiling Catalyst - a memory model should only be
specified as an argument to the "build_all" script when compiling for the
Propeller 2, not for the Propeller 1.
I have just updated the Catalyst spin demo (located in the "demos/catalyst/demo" folder of the Catalina installation tree) to add a version for the Propeller 2. Unzip the attached zip file over the contents of that folder (it is a complete replacement) or to a separate folder.
This demo shows how Spin programs can be executed from Catalyst and passed command-line arguments, just like C and Lua programs.
Demo.spin only supports the Propeller 1, but I have added Demo.spin2 for the Propeller 2. You can use the Propeller Tool to compile them both. I used version 2.7.0.
Ross.
EDIT: This is also included in the Catalina 5.3.1 patch, which is now available.
I have just posted Catalina 5.3.1 to SourceForge here.
This is a patch release that must be installed over Catalina 5.3 (Windows or Linux). The purpose of this patch is to fix a bug I found (finally!) in the Propeller 2 SD Card plugin. There are a few other minor additions and fixes. Here is the relevant portion of the README.TXT:
RELEASE 5.3.1
New Functionality
-----------------
1. Catalyst now understands ".lux" as a filename extension, as well as ".lua".
Catalyst assumes ".lux" represents a compiled Lua script, whereas ".lua"
can represent either a compiled or a non-compiled Lua script. By default,
files with a ".lux" extension will be executed with LUAX.BIN and files
with a ".lua" extension will be executed with LUA.BIN. If no extension
is specified on the command line and files with both extensions exist,
".lux" will be used.
2. There is now a Propeller 2 version of the Catalyst Spin demo that shows how
to use Catalyst command-line arguments in Spin2 files. It is in the folder
demos/catalyst/demo and called Demo.spin2.
Other Changes
-------------
1. Fixed a race condition in the SD Card plugin, which meant that it could
fail when a sequence of SD Card operations was performed at a specific
rate. Affected the Propeller 2 only.
2. The demo program ex_time.c was not clearing the daylight savings flag when
setting the time, which meant that when the time was retrieved, the hour
would sometimes be one hour different than what was set. Affected both
the Propeller 1 and Propeller 2.
3. Dumbo Basic was not flushing its output buffers, so prompts and input was
not appearing in some cases when executing input statements until an
end-of-line character was entered.
I have just posted Catalina 5.4 to SourceForge here.
This is a full release, primarily to update Catalina to add PSRAM support on the Propeller 2, and Catalyst to add HIRES_VGA support on the Propeller 1, and also add enhanced Catalyst scripting capabilities on both the Propeller 1 and 2. Plus a few significant bug fixes.
Here is an extract of the README.TXT file concerning the new functionality - the full file is too long to include here, but is attached:
RELEASE 5.4
New Functionality
-----------------
1. Roger Loh's 16 Bit PSRAM driver has been added as a Catalina plugin.
It is supported on the Propeller 2 only. It is enabled by linking with
the new psram library (i.e. adding -lpsram to the Catalina command).
An example of its use has been added in "demos\examples\ex_psram.c".
The configuration parameters for the driver must be specified in the
platform files in the target_p2 directory, such as P2_EDGE.inc.
You would compile the PSRAM example program with a command like:
catalina -p2 -lc -lpsram -C P2_EDGE ex_psram.c
The only tested platform is the P2 EDGE, and like the driver itself,
Catalina's support for it should be considered a beta version until
further notice.
2. As a demonstration of the use of the PSRAM plugin, Lua now has the option
to store code in PSRAM on those platforms that support it, such as the
P2 EDGE. This allows larger Lua programs to be executed at the cost of a
slight speed reduction. Supported on the Propeller 2 only.
To enable the use of PSRAM in Lua, specify ENABLE_PSRAM to the Catalyst or
Lua build_all scripts. For example:
build_all P2_EDGE SIMPLE VT100 ENABLE_PSRAM
Note that PSRAM is supported only by the Lua execution engine (luax) which
executes compiled Lua programs, and not for the interactive version (lua)
that executes text programs or the Multiprocessing version (mlua or mluax).
So if ENABLE_PSRAM is specified, only luax will be built. This means you
may need to build Lua twice - once to build the lua programs that do not
use PSRAM, and then again to build luax (only) to use PSRAM.
For example, to compile Lua in directory "demos\catalyst\lua-5.4.4" and put
the executables in "demos\catalyst\bin" and call the PSRAM version luaxp
rather than overwrite the standard luax binary, you might use commands
such as:
cd demos\catalyst\lua-5.4.4
build_all P2_EDGE SIMPLE VT100
copy src\*.bin ..\bin\
build_all P2_EDGE SIMPLE VT100 ENABLE_PSRAM
copy src\luax.bin ..\bin\luaxp.bin
Note that specifying ENABLE_PSRAM is applicable only when using the
Catalyst and Lua build_all scripts and Makefiles - it is not a general
Catalina symbol that can be used on the Catalina command-line to enable
PSRAM in other cases (which is done via the usual mechanism of linking
the program with the psram library - i.e. adding -lpsram to the catalina
command).
Finally, note that for small Lua programs, the Hub RAM usage of the PSRAM
version may not be much smaller than that of the non-PSRAM version - it
may even be larger. This is not only because of the additional PSRAM
support code required, it is also because the PSRAM version allocates a
fixed amount of Hub RAM on startup to use as a PSRAM cache, and for small
Lua programs the cache may be larger than the program being loaded.
However, the amount of Hub RAM used for Lua code will never increase
beyond the cache size no matter how big the program code gets.
3. Catalyst on the Propeller 1 can now use the HIRES_VGA option. However,
Hub RAM limitations mean that Catalyst itself needs to be built as an
EEPROM program, and some of the Catalyst utilities may only work if
they are built as LARGE programs.
This means that while Catalyst itself will work in HIRES_VGA mode on all
platforms, some utilities (such as cp & mv) may only work in HIRES_VGA
mode on platforms with XMM RAM. Alternatively, you might build Catalyst
itself in HIRES_VGA mode, but the utilities in LORES_VGA mode.
To facilitate this, two new options have been added that can be used with
the Catalyst build_all scripts:
EEPROM_CATALYST specifies that the Catalyst binary should be
built as an EEPROM program.
LARGE_UTILITIES specifies that the Catalyst utilities should be
built as LARGE programs.
Whether you need to specify one or both of these options can depend on the
other options used. For instance, if you need to use the cache to access
XMM RAM, then you will generally need to use both of these options to
build Catalyst in HIRES_VGA mode. For example, here is how you might
build Catalyst to use HIRES_VGA on the C3:
build_all C3 FLASH CACHED_1K HIRES_VGA EEPROM_CATALYST LARGE_UTILITIES
When you build Catalyst to use LORES_VGA or HIRES_TV HMI option, you may
also find that catalyst.binary exceeds the size of Hub RAM and in that
case you can specify the EEPROM_CATALYST option, but you may not need the
LARGE_UTILITIES option - e.g:
build_all C3 FLASH CACHED_1K HIRES_TV EEPROM_CATALYST
Note that to program Catalyst into EEPROM when the CATALYST_EEPROM option
is used, you will need to run the build_utilities script to build the
EEPROM loader, and then use that with payload. For example, to build and
load Catalyst to use HIRES_VGA on the C3, you might use commands like:
cd demos\catalyst
build_all C3 FLASH CACHED_1K HIRES_VGA EEPROM_CATALYST LARGE_UTILITIES
build_utilities
payload -o1 EEPROM ..\bin\catalyst.bin
Note that these new options are applicable only when using the Catalyst
build_all scripts and Makefiles - they are not Catalina symbol that can
be used in other circumstances (i.e. specifying -C EEPROM_CATALYST when
compiling catalyst.c manually will not have any effect. It is the Makefile
that intercepts this symbol and instead uses -C EEPROM, but only when
building catalyst.binary).
4. The Catalyst ONCE capability (i.e. to execute a command once on reboot)
has been extended to execute MORE than one command. If enabled when
building Catalyst (by setting both ENABLE_ONCE and ENABLE_MORE to 1) then
the file (EXECONCE.TXT by default) can contain more than one command.
The commands in the file will be executed in sequence on successive
reboots. The Lua "execute" function can be used to easily add multiple
commands to the file, one per line. For example, if the MORE capability
is enabled, the Lua statement:
propeller.execute("vi abc\n vi def")
will cause the propeller to first reboot and execute the command "vi abc",
and then when vi exits, the propeller will reboot and execute the command
"vi def".
This also allows a very basic scripting capability to be implemented.
For example, if you have a file called commands.txt which contains all
the commands you want executed, then executing the command:
cp command.txt execonce.txt
at the Catalyst prompt, or executing the Lua statement:
propeller.execute("cp command.txt execonce.txt")
will cause all the commands in the file to be executed in sequence.
Note that this capability is enabled by default on the Propeller 2, and
on the Propeller 1 unless the HIRES_VGA HMI option is used, because there
is not enough Hub RAM available. If you want to enable it, you may need to
disable something else, such as the capability to allow Lua commands to be
executed directly from the command line (note that you can still execute
them by specifying them as parameters to Lua - e.g. by entering a command
like "lua list.lua" instead of just "list"). This can be disabled by
editing "demos\catalyst\core\catalyst.h" and rebuilding Catalyst.
5. The internal cat, dir and help commands have been removed from Catalyst.
The help and cat commands are now always external. The dir command has
simply been dropped since the ls command is far more capable. However, if
you prefer typing "dir" to "ls", then just make a copy of "ls.bin" called
"dir.bin" - i.e:
cp ls.bin dir.bin
6. Added a new Lua demo called "wild.lua" that can be used to add wildcard
capability to a Catalyst command that accepts multiple file names.
For example, if the command specified in wild.lua is "vi" (which it is
by default) then
luac -o xvi.lux wild.lua
will create a command "xvi" which can then be used on the command line
to invoke vi on multiple files - e.g:
xvi ex*.lua
will execute vi on all the Lua example files.
7. The length of a Catalyst command line has been increased to 300
characters on the Propeller 1, and 1200 characters on the Propeller 2.
Hub RAM is limited on the Propeller 1, but 300 is enough to accommodate
24 MSDOS 8.3 filenames, which could be generated when using wildcards,
and 1200 is the maximum that CogStore can store. This maximises the
potential usefulness of the wildcard functionality added in the last
few releases.
8. A new "getrand()" function has been added to the C library. It is defined
in "propeller.h" and is implemented on both the Propeller 1 and the
Propeller 2 (but differently - see below). A program that demonstrates
the use of the function has been added in "demos\examples\ex_random.c"
On the Propeller 1, the first time this function is called it calls
srand() with the current CNT value and is therefore best called after
some user input or other random source of delay. It then returns the
result of 3 combined calls to rand() to make up 32 random bits (rand
itself only returns 15 bits).
On the Propeller 2, the first time this function is called it calls
srand() with the result of the GETRND opcode, and also returns that
value. Thereafter it just returns the result of the GETRND opcode.
This means you can either use just this function, or use this function
once to generate a seed for srand() and thereafter use rand(), which is
what most traditional C programs would typically do.
Note that rand() only returns a value between 0 and RAND_MAX (inclusive)
(i.e. 0 .. 32727 on the Propeller) whereas getrand() returns 32 bits.
To simulate rand() using getrand(), use an expression like:
(getrand() % (RAND_MAX + 1))
9. On the P2_EDGE, the base pin for VGA has been changed to 0 (was 32),
and the base pin for USB has been changed to 8 (was 40). This makes it
possible to use the P2-ES VGA and USB accessory boards with Catalina
on the P2_EDGE. The base pins can be edited if required in the file
P2_EDGE.inc in the target_p2 folder.
Oops! A silly mistake in the Propeller 1 XMM startup code meant an XMM SMALL program loaded using Catalyst could not access command-line arguments. Does not affect other program types on the Propeller 1, or the Propeller 2.
To fix it, just drop the attached file over the existing one in Catalina's target directory and recompile any affected programs (you don't need to recompile either Catalina or Catalyst).
This will also be included in the next release, which is likely to be a very small patch release unless I find any other problems.
UPDATE: This file is now also included in Catalina patch release 5.4.1
I have just posted Catalina 5.4.1 to SourceForge here.
This is a patch release that must be installed over Catalina 5.4 (Windows or Linux). The purpose of this patch is to fix the bug that was preventing XMM SMALL programs using command line arguments when loaded using Catalyst, plus a few other minor changes. Here is the relevant portion of the README.TXT:
RELEASE 5.4.1
New Functionality
-----------------
1. p2asm now supports the "IF_NN" and "IF_NOT_NN" (where N=0,1 or X) PASM
instruction prefixes, as well as "IF_SAME" & "IF_DIFF". For example,
the following two statements will generate the same PASM instruction:
if_c_or_z mov r0, #1
if_not_00 mov r0, #1
Other Changes
-------------
1. Programs compiled in SMALL mode and loaded using Catalyst were not setting
up the argc and argv parameters to the C main function correctly. Affected
the Propeller 1 only.
2. The arg.c argument diagnostic program in the utilities folder was not
working correctly on the Propeller 1.
3. It was not clear in the Catalyst documentation that when Catalyst is
used to load XMM programs, then both Catalyst and the XMM programs
must use the same caching and flash options. For example, if Catalyst
is compiled using FLASH CACHED_1K then so must all XMM programs (but
note that they can be SMALL or LARGE programs). This only applies to
XMM programs, and not to LMM, CMM, SMM, Spin or Lua programs. Applies
to the Propeller 1 only.
4. The Catalina Optimizer was failing to perform some optimizations when the
optimization level was 3 or greater, so that -O3 could result in a larger
code size than -O2. Affected all memory models except COMPACT on both the
Propeller 1 and the Propeller 2.
Just a quick - but important - update on Lua on the Propeller 2 ...
I have just realized that Lua can very easily take advantage of the Propeller 2's ability to read and write data to Hub RAM without requiring the address to be aligned on a particular boundary (e.g. you can read or write a long to any address, even if that address is not a multiple of 4).
This gets around a significant problem using Lua in memory constrained environments, which occurs because Lua is a dynamically typed language. That means that every value in Lua (even a simple integer) must be tagged to tell Lua what type it is (e.g. that the value should be interpreted as an integer and not as a float or a pointer to a string). This requirement means that on most processors, Lua requires 8 bytes to store each 4 byte value - 4 for the value and another 4 for the tag, because even though the tag only needs 1 byte, the alignment requirements of most processors force anything that follows the tag to the next 4 byte boundary, so effectively the tag also takes 4 bytes. For things like arrays this can be a real problem because it can double the memory required!
But the Propeller 2 can do this in 5 bytes rather than 8 bytes - 4 bytes for the value and 1 byte for the tag, at the trivial cost of one additional clock cycle per access. Chalk one up for the Propeller 2!
By itself, this can reduce a typical Lua program's memory requirement by 5-10% - more in some cases. When combined with the PSRAM support, it can reduce a typical Lua program's memory requirement by 20-30% - more in some cases.
This will be in the next release of Catalina. But it turns out to be so easy that if anyone is interested in a preview, just drop the attached files over the existing ones in demos\catalyst\lua-5.4.4\src and then rebuild Lua. Note that the alignment fix will work on all Propeller 2 platforms, but PSRAM is currently only supported on the P2 EDGE, and to enable it you must add ENABLE_PSRAM on the command line - e.g:
cd demos\catalyst\lua-5.4.4
build_all P2_EDGE SIMPLE ENABLE_PSRAM
Comments
A linux release of Catalina 5.0 has been added to SourceForge - see https://sourceforge.net/projects/catalina-c/files/releases/5.0/
I could have predicted this would happen as soon as I finished the release. It always seems to!
I wondered why one of my Lua example programs (ex9.lua) seemed to need much more stack space than the other examples to execute reliably, even though it didn't really do much more. I never could figure out why, so I also used it as an example of how Lua's garbage collector can help when memory gets tight. But it turns out this example does not need the larger stack at all - it was apparently the result of an obscure bug in Lua itself, which I seem to have now fixed quite by chance while working on something else.
There are no functional changes to either Lua or the new multi-threading module, and in any case I will probably leave the example intact (except for the stack size) as an example of managing Lua's garbage collector, but I'll wait to see if anything else crops up before I re-package release 5.0 (or release 5.1 if I find anything more serious).
Ross.
Catalina 5.0.1 has been released here.
This release adds no significant new functionality, but it fixes one major Lua issue and a couple of other minor issues.
Here is the relevant part of the README.TXT:
I have also updated the version of Multi-threaded Lua pre-compiled for the P2 Evaluation board which is attached to an earlier post.
Ross.
As usual, I wish I had thought to do this before the release!
Now that I have fixed Lua's memory issue, I thought I'd try compiling Multi-threaded Lua in NATIVE mode rather than COMPACT mode. Doing so means there is less memory available for Lua programs, but the programs will execute much faster. The command I used was:
build_all P2_EVAL SIMPLE VT100 OPTIMIZEHere are the results for the 10 tutorial examples:
ex1.lua, ex2.lua, ex3.lua, ex5.lua, ex7.lua, ex10.lua : All good. Can be executed from source (using mlua), or compiled (using lauc and mluax).
ex4.lua : The increased execution speed highlights that the original program has a race condition. Adding a small delay to the ping and pong functions fixes the problem:
function ping() t = threads for i = 1, t.shared("count") do ball = t.get("ping") t.output("ping ") t.msleep(10) -- DELAY ADDED t.aput("pong", ball) end end function pong() t = threads for i = 1, t.shared("count") do ball = t.get("pong") t.output("pong\n") t.msleep(10) -- DELAY ADDED t.aput("ping", ball) end endex6.lua : Must now be compiled. Otherwise, all good.
ex8.lua : There is only enough memory available to execute 6 workers concurrently (not 12) and the increased execution speed is such that a small delay needs to be added to the Thread function to illustrate that all the workers are indeed executing in parallel:
count = 6 function Thread(me, iter) return function() t = threads -- wait till we are told to go repeat until t.shared("go") repeat t.output(me .. " ") t.msleep(10) -- DELAY ADDED iter = iter-1 until iter == 0 end endex9.lua : There is only enough memory available to recycle up to 2 workers, not all 4:
Other than example 9, these programs are too small and simple to show much improvement in execution speed ... except (in strict accordance with Murphey's Law) where it highlights a problem in my example programs!
Ross.
There is an interesting synergy between Lua co-routines and threads. I will add this to the threads documentation in the next release, but the example program below can be executed with the current release, so I thought I would post a preview here because it shows what a "rich" language Lua is ...
Even without the threads module, Lua has a simple type of multi-tasking. It has non-preemptive multi-tasking in the form of co-routines. The threads module adds preemptive multi-tasking. However, this has interesting implications when co-routines are executed using the threads module.
Here is a simple example:
-- Generate a function which can be executed -- as either a coroutine or as a thread function Generate(me) return function () for i = 1, 10 do threads.print(me ..":" .. i) threads.msleep(100) coroutine.yield() end end end -- Execute the functions as coroutines threads.print("\nFUNCTIONS AS COROUTINES") -- create 4 functions as coroutines and also -- wrap the coroutines inside another function fee = coroutine.wrap(Generate("fee")) fi = coroutine.wrap(Generate("fi")) fo = coroutine.wrap(Generate("fo")) fum = coroutine.wrap(Generate("fum")) -- execute the coroutines fee() fi() fo() fum() -- and again fee() fi() fo() fum() -- Execute the functions as threads threads.print("\nFUNCTIONS AS THREADS") -- we only need one worker to execute any number of -- threads if those threads explicitly "yield" threads.workers(1) -- create (and start) 4 functions as threads threads.new(Generate("fee")) threads.new(Generate("fi")) threads.new(Generate("fo")) threads.new(Generate("fum")) -- wait for all threads to complete threads.wait()Here is the output this program produces:
When the functions are executed as co-routines, then each time the function is called, it executes one iteration and then "yields". This "yield" call is the key to co-routine behavior. It returns control to the caller, but "suspends" the co-routine rather than exiting it. When the function is called again, it does not start from the beginning as a normal function would, it instead continues execution from where it was last suspended - i.e. just after the last "yield". This is normal co-routine behavior (for more details on co-routines, see the Lua documentation).
The interesting thing is the consequence of calling "yield" when the functions are executed as threads instead of co-routines. When a thread yields then the thread is suspended, but the worker executing that thread is not - it moves on to resume execution of the next suspended thread. If there is no other suspended thread it will resume execution of the same thread. Since we started all our functions as threads, this means that one worker (which is one Posix thread) will execute all the functions concurrently (using non preemptive multitasking) provided those functions are written as co-routines. If the functions are not written as co-routines, then you need multiple workers available to execute them concurrently (using preemptive multitasking). Which of course is very easily done in this example - just change the number of workers from 1 to 4. You can try it, but in this simple example it makes little or no obvious difference to the output.
In summary, with 1 worker the program uses non preemptive multitasking. With 4 workers it uses fully preemptive multitasking. With 2 or 3 workers, it would use a mixture of both.
Isn't Lua an interesting language?
Ross.
Catalina 5.0.2 has been released here.
This release is a patch release for both Windows and Linux. It must be installed over Catalina 5.0.1. It adds no new functionality, but it fixes one Lua issue and adds a couple of new Lua examples. If you are not interested in Lua you don't need it - the changes will also be included in the next full release.
Here is the relevant part of the README.TXT:
I have also updated the version of Lua pre-compiled for the P2 Evaluation board which is attached to an earlier post.
Ross.
Interesting. Now that Lua 5.1 seems rock solid (yes, I realize I'm tempting fate here
) I have been re-looking at Lua 5.4, which is the latest and greatest version.
When I looked at it a while ago, the first thing I noticed was that the initial code/data footprint of Lua 5.4 is much larger than that of Lua 5.1 - by about 40kb even in COMPACT mode. This was enough for me to stay with 5.1 as the default version even though 5.4 seemed to have some useful improvements.
But it turns out I didn't look deeply enough - yes, the initial footprint of Lua itself is larger, but the footprint of each concurrent thread is smaller - by about 5kb. This means that if you have 8 or more workers (as you might if you have 8 cogs!), then Lua 5.4 actually ends up using less memory than Lua 5.1.
Only one of the tutorial examples (ex8.lua) actually creates this many workers - this particular example is specifically designed to see how many workers it can execute concurrently. It turns out that you can execute more workers with Lua 5.4 than you can with Lua 5.1 - something I had not thought to check!
As well as better memory management, Lua 5.4 also includes integer numeric types (like some versions of Basic, Lua originally only had floats) and bitwise operators, which makes it much more suitable for the type of applications common on the Propeller.
Catalina will continue to support all versions of Lua (from 5.1 onwards) but unless I find a showstopper, I am likely to switch the default to Lua 5.4 in the next release.
Ross.
I can't believe I missed this! I just found a version of the classic startrek game ported to Lua! I already had C, Pascal and Basic versions included with Catalina, but I had never thought to even look for a Lua version. It seems I am not the only one that remembers this game very fondly!
The sources of the original Lua port are available here: https://github.com/emabolo/superstartrek
In the attached zip file are two Lua versions that I have modified very slightly, because the originals were written before Lua 5.4 was released, and the math library has changed slightly. They also needed a few tweaks to the garbage collector to stop them running out of memory on the Propeller. One version is a faithful port of the original 1978 game (sst.lua) and the other is an "enhanced" version (sst-tos.lua) that incorporates dialog taken from the original scripts.
I have not done extensive testing, but I have played them both for a few minutes each and they both seem to run ok when compiled with luac and executed using mluax.
For instance, in Catalyst, you would use commands like:
or
Enjoy!
I have been testing Catalina on the new P2 Edge module, and can confirm that Catalina works ok on it.
However, while testing various things I found a bug in Catalina's SD Card plugin that may have prevented programs from being able to write to some types of SD Cards (reading was ok). I will include the updated version of the plugin with the next release of Catalina, but I have also attached it here - it can simply be copied over the current version in the target_p2 directory. Of course, you will have to recompile any programs that use it.
Note that this bug affects the Propeller 2 only.
I have just posted Catalina 5.1 to SourceForge here.
I am not getting much time for Catalina at the moment so there are not many changes, but the new SD Card plugin is included so I thought I had better do a release.
Here is the relevant extract from the README.TXT:
I have just posted Catalina 5.1.1 to SourceForge here.
This is a very small patch release for the Propeller 1 only, mostly to fix a silly bug that was introduced in Catalina 4.9.3.
This patch release must be installed over an existing installation of Catalina 5.1. Here is an extract of the README.TXT file:
RELEASE 5.1.1 New Functionality ----------------- 1. Added a new demo folder ("xeprom") containing programs that demonstrate how to read from the EEPROM from XEPROM XMM programs. In such programs, the EEPROM cannot be read independently (e.g. using an I2C driver) because it is constantly in use by the cache. This applies only to the Propeller 1, and requires a platform with an EEPROM larger than 32k, such as a FLIP, a C3 or a HYDRA. Other Changes ------------- 1. Fixed a bug that meant the -e command line switch to Catalina, which is used to specify that a ".eeprom" file be generated instead of a ".binary" file did not work correctly in some instances. In particular, it could not be used in conjunction with the -C XEPROM option. This affected the Propeller 1 only, and only Catalina version 4.9.3 or later. 2. The catalina_clean utility script had not been updated recently, and would miss cleaning some directories when the optional "all" parameter was specified.I have just posted Catalina 5.1.2 to SourceForge here.
This is another small patch release, mostly to fix another silly Propeller 1 bug that was introduced in Catalina 5.0, but it also incorporates a few changes to Lua that affect the Propeller 2.
This patch release can be installed over an existing installation of Catalina 5.1 or Catalina 5.1.1. Here is an extract of the README.TXT file:
RELEASE 5.1.2 New Functionality ----------------- 1. The path that Lua uses to search for modules has changed. The old path was the one used on Windows, which was not appropriate for the Propeller. The path is now: "?.lua;?/init.lua;lua/?.lua;lua/?/init.lua" This means that (for example) if you say "require 'm'" then Lua will search for "m.lua", "m/init.lua", "lua/m.lua" and "lua/m/init.lua" in that order. 2. The Lua demos sst.lua and sst-tos.lua now use "propeller.sbrk()" instead of "threads.sbrk()", which means the mem command (which displays the top of the C heap) now works when the program is run under both lua and mlua. Other Changes ------------- 1. Fixed a bug in the threads library, which prevented some multithreaded programs from compiling. Affected only Propeller 1 programs which were compiled in LMM TINY mode. 2. Eliminated the file minit.c from Lua. This was a temporary replacement for the Lua file linit.c which included the threads module. Now linit.c will include the "threads" module if LUA_THREADS is defined in the file luaconf.h. For completeness, there is also a corresponding LUA_PROPELLER that is used to specify that the "propeller" module be included. This is more in keeping with the way Lua normally defines compile-time options, eliminates the need to have the extra file, and it also means that when compiling the executables (e.g. lua and mlua or luax and mluax) under Catalina, Catalina can automatically detect the compile options and include the correct modules. But if you compile mlua or mluax using gcc and posix threads, you will now need to explicitly define LUA_THREADS in luaconf.h to include the "threads" module. 3. Fixed a bug that meant the Lua "propeller" module would not compile using any compiler other than Catalina. Although it is only functional when compiled using Catalina, it should still have compiled using other compilers (such as gcc). 4. The number of threads the "test_maximum_threads.c" demo program creates has been reduced to 145, since at 150 there are some plugin combinations that do not have enough free Hub RAM to support 150 threads. Affected the Propeller 1 only. 5. The build_all scripts and Makefiles now always build Lua in COMPACT mode on the Propeller 2, because NATIVE mode does not leave enough Hub RAM for the various demo programs.I have to apologize for the recent patch releases. I have been working almost exclusively on P2 and Lua support recently, and I had not bothered to run the validation suite before the releases - if I had done so the validation suite would have picked the problems up straight away. I will do so before future releases
I have just posted Catalina 5.2 to SourceForge here.
This is a full release, primarily to update Catalyst to add some outstanding features that were always intended to be there, but I had never bothered to finish - like wildcard support for various Catalyst commands (i.e. ls, cp mv, rm, cat).
I have become a real fan of Lua, and I tend to mainly use my P2 as a stand-alone Lua development platform these days. These additions make developing Lua software on the P2 for the P2 very much easier ... as long as you are also a fan of the vi text editor. Let the editor flame wars commence!
Here is an extract of the README.TXT file:
RELEASE 5.2 New Functionality ----------------- 1. Added a new inline pasm demo ("test_inline_pasm_5.c") to demonstrate how to CALL an inline PASM function from within an inline PASM function in a COMPACT program. 2. Updated the validation suite. Now logs more details to make it easier to identify what failed, and also added "short" scripts to do faster validation. 3. Updated the file 'globbing' demo (in folder "demos\globbing"). Tidied it up, fixed a few bugs and it now uses the correct DOS filename syntax, prints the filenames properly, and is now correctly case insensitive in all cases. The listDir() function has been re-written to be an instance of a new generic doDir() function, which calls a specified function on each matching filename. 4. Wildcard matching ('globbing') has been added to many of the Catalyst utilities. You can now specify wildcard expressions like *.bin or [a-g]*.b?? - see "glob.c" for details. The options and syntax of the utilities have not changed, except for 'ls' (see 5 below). The commands affected are: ls - list files and/or directories rm - delete files or directories cp - copy files mv - move files cat - concatenate and print files For example, you can now say things like: ls *.bin *.dat mv [a-f]*.bin bin rm ???.dat cat *.txt Note that wildcards can only be used in the file name part of a path, not in the directory part, so you cannot specify an argument like "/b??/*.*". For arguments that may be interpreted as files or directories, adding a trailing "/" ensures they will be treated as directories. For example: ls bin -- list just the entry "bin" (if it exists) ls bin/ -- list the contents of directory "bin" (if it exists) 5. The Catalyst "ls" command now uses a short listing format (similar to that used by Unix or the built-in "dir" command) by default. The previous default listing format can be specified using the "-l" option, and the previous long listing format (which used to be specified by "-l") must now be specified by "-l -l" or "-ll". Also fixed interactive mode, which was not working in some cases. 6. The Catalyst "cp", "mv" and "rm" commands now accept 'a' or 'A' (for "all") at the interactive prompt, to indicate that the command should assume "yes" for all subsequent files. 7. The catalyst "build_all" script now does "clean_all" before building, to ensure there are no binaries left from previous builds. Other Changes ------------- 1. Updated the file "catalina_compact.inc" in the target and target_p2 directories to add more notes about writing inline PASM in COMPACT programs. 2. The tiny library had a version of the C stdio function gets(). This prevented programs that used gets() from linking with the tiny library, because it ended up being multiply defined. The version in the tiny library has been renamed tiny_gets() in this library, but like the other tiny functions (i.e. tiny_printf() etc) it will be used automatically if a program is linked with the tiny library. Affected the Propeller 1 and Propeller 2. 3. Fixed a bug in the Propeller 1 SD Card plugin - multiple block writes must not be used on platforms that must disable the SD Card to use XMM RAM (e.g. because they share pins). Affected the Propeller 1 only. 4. Fixed a bug in Catalina's "unmanaged" file close function which meant that only a limited number of files could ever be opened. Did not affect the normal stdio open/close functions, but it affected the Catalyst "cat" command, which uses the unmanaged versions of open/close to save space. 5. Catalyst's example basic files are now all Unix format, not DOS format. This makes no difference to their functionality, but makes them easier to edit usng "vi" or list using "cat". 6. Fixed a bug in DOSFS which affected the ability to delete files with zero length (e.g. using the Catalyst "rm" command) - doing so may have corrupted the file system.Is that self-hosting?
EDIT: Never mind, found the docs:
Looking up Spinx I see it's been around since 2009. So self-hosted Spin programming has been an option for a while then. I've not seen anyone mention this before when the topic of self-hosting gets its regular airing.
Hi @evanh
No, it's not self-hosting because you cannot compile the Lua compiler itself on the Propeller - you have to compile that using Catalina. There is a version of Lua written entirely in Lua, but I think it would be too large to compile on the P2, and too slow to compile on the P1 even if you had megabytes of XMM RAM available. I have not tried it, and I don't intend to do so - I've never understood why people are so obsessed with self-hosted development on a microcontroller, when you can buy a full-blown computer that will do the job better and cheaper.
Still, Catalyst is fine for developing and executing Lua programs on the Propeller, and it's a fun way to learn Lua!
Why is it that as soon as you do a release, you notice a stupid little problem ... Murphey's Law, I guess ...
One of the Lua demos - fact.lua - was written when all Lua numbers were floating point. Recent versions of Lua have both floats and integers, but it is still an untyped language - so an expression like n = 1 makes n an integer, whereas n = 1.0 makes n a floating point.
So fact.lua should now look like this, with one minor change on line 18:
-- function closures are powerful -- traditional fixed-point operator from functional programming Y = function (g) local a = function (f) return f(f) end return a(function (f) return g(function (x) local c=f(f) return c(x) end) end) end -- factorial without recursion F = function (f) return function (n) if n == 0 then return 1.0 -- <<< changed 1 to 1.0 to return a float instead of an int else return n*f(n-1) end end end factorial = Y(F) -- factorial is the fixed point of F -- now test it function test(x) io.write(x,"! = ",factorial(x),"\n") end for n=0,16 do test(n) end -- added for Catalina/Catalyst, which may clear the screen on program exit print("\nPress any key to continue") any = io.stdin:read()I'll leave it to you to figure out how this program actually works, and how it can claim to be non-recursive.
Lua looks a lot like any other procedural language (Pascal, Modula, C, C++ etc) - but it is very much not. It is so expressive and powerful that it can seriously do your head in. That's probably why I like it
Well, without reading anything from the Web, first thing of note is the apparent function call to
factorial(x)and that that is the one thing that isn't defined as a function.Ok. here goes ... **spoiler alert! **...
It is best to start from the bottom ...
The line "factorial = Y(F)" tells you that factorial is what is returned from calling function Y with F as a parameter. But since Lua is an untyped language, we don't know what that is - at least not from just this line.
The lines "F = function(f) return function (n) ..." tells you that F is a function that accepts an argument f - and we learn a few lines down that f has to be a function (remember Lua is typeless - we only know f has to be a function because F uses it as such). But note that F itself does not call f - instead F uses f to generate another function, which accepts a parameter n - and that function would calculate factorials --- provided F was passed a function f that does exactly what the generated function does (big hint there!).
The lines "Y = function (g) ..." is where the magic happens. This tells us that Y is a function that accepts a parameter g, and it produces the "fixed point of g" - i.e. it calculates "fix g" = g(fix g) = g(g(fix g)) = ... etc ...
So, when "fix g" is F, Y returns a function that takes one numeric argument, and that function calculates the factorial of its numeric argument.
I warned you that Lua can do your head in! But it is a surprisingly small, clean and highly orthogonal language that can be used to do procedural programming, object-oriented programming, or (as in this case) functional programming.
RossH,
I have seen 'uncle' Murphy pay a visit too many times when either there was an insane schedule to meet or there was a piece of equipment on the verge of death.
Another thing that I have found is that it's best to wait a day or two before doing your final checks, because when your mind is fresh mistakes and/or problems are easier to spot.
Good advice. However, I generally only have limited time to devote to this at the moment, so I have to get it out there while I can!
I myself prefer to take my time and focus on quality, because I have found that if I just try to get it done, then sooner or later I will have to fix it.
Attached is a preview of what will be coming in the next release of Catalina. If I had realized how easy this was going to be, I would have held up the last release for it, but no matter - I am sure I will find a few more annoying bugs that also need fixing.
I have decided to make Lua the scripting language for Catalyst itself (an obvious thing to do really, but I had just not bothered, because I had not realized both how easy and how useful it would be until I had learned more about Lua).
You could of course always invoke Lua manually, and you still can - but now Catalyst will look for Lua programs to execute the same way it executes any normal Propeller binaries, so it is much easier to extend Catalyst's basic functionality using Lua and then invoke these programs seamlessly from the command line as if they plain old Propeller binaries.
The attached file includes a working prototype of the new versions of both Catalyst and Lua, compiled for the P2 EDGE or P2 EVAL boards (you need an SD Card and a VT100 terminal emulator such as putty or payload). A version of payload is included in the attachment. If you use putty you may need to tweak its line termination options.
It includes three demos designed to highlight the new Lua functionality, and illustrate how easily Lua now integrates with Catalyst:
Load the programs onto an SD Card, set up your P2 to boot from it, and then use the following payload command (where X is the USB port connected to your P2):
If you reboot your P2, you should see the Catalyst prompt:
As well as all the usual Catalyst commands, you can try the following new commands:
To see how easy stuff like this is to do, here is a complete (but quite mimimalist) version of a directory listing program, implemented in Lua:
-- print the file name only function print_entry(name, attr, size) print(" " .. name) end -- mount the DOSFS volume propeller.mount() -- scan the DOSFS directory specified in arg[1] and call print_entry -- on any files matching the wildcard pattern specified in arg[2] propeller.scan(print_entry, arg[1], arg[2])The real version (in list.lua) has a few more bells and whistles, but is basically the same program.
You may notice that this program uses two new functions that have been added to the Lua propeller module:
This is using essentially the same wildcard 'globbing' functionality now used in the new Catalyst utilities (ls, mv, cp, rm etc), and it makes it especially easy to write new file processing utilities in Lua.
So, is it worth rewriting all the existing Catalyst utilities (e.g. ls) as Lua scripts?
Well, in short "no". The C versions are faster and can run on any Propeller - 1 or 2, whereas the Lua versions could only run on a Propeller 2, or on a Propeller 1 with sufficient XMM RAM. And while loading Lua each time you need it does work ok on a Propeller 1, on some platforms - especially those that use FLASH memory as XMM RAM - it is painfully slow!
Lua is fun!
EDIT: Updated freq.lua in the attached zip file (to add garbage collection).
EDIT: I will leave this zip file here, because it is a stand-alone demo - but a slightly more advanced version of the same functionality has now been released as part of Catalina 5.3, so it is generally recommended to download that instead.
I have just posted Catalina 5.3 to SourceForge here.
This is a full release, primarily to update Catalyst to add Lua scripting capabilities.
Lua can now be used as a simple method of extending Catalyst functionality, as well as being a powerful high-level language for developing programs that can exploit the Propeller's unique capabilities.
Lua provides the kind of capabilities in a high-level language that I wanted the Propeller to have from day 1, but which were only previously available from low-level languages like Spin, Forth, C and C++.
Here is an extract of the README.TXT file:
RELEASE 5.3 New Functionality ----------------- 1. The Catalyst utilities cp, mv, rm, ls and cat now ignore volume id entries, except that the ls utility will still include them when long or very long listing output is specified. 2. Catalyst now has the ability to execute a once-only command on startup. This is similar to the existing AUTOEXEC.TXT file processing when the AUTODELETE option was also enabled. The AUTOEXEC.TXT and AUTODELETE functionality has been retained, but AUTODELETE is not enabled by default, which means that normally the AUTOEXEC.TXT is executed on every Propeller reboot. The new once-only execution functionality overrides this if the file EXECONCE.TXT exists. If it does then it will be executed and also deleted. This new functionality is used by the Lua propeller module "execute" function, described in point 3 below. It requires that the SD Card be writable. 3. The Lua propeller module now includes new propeller-specific functions. Lua analogues of the C DOSFS wildcard/globbing functions: mount() scan(function, directory, filename) The filename can include wildcards. See the Catalyst Reference Manual for details, and see list.lua for an example of using these functions. A Lua function to execute any Catalyst command (reboots the Propeller): execute(command, filename) For example: propeller.execute("list *.bas") The first parameter is the command (which may include parameters), and the second (optional) parameter is the file name to write this command to. This defaults to "EXECONCE.TXT", which means the command will only be executed once, but it can be used to write the command to any file. For instance: propeller.execute("lua", "AUTOEXEC.TXT") will cause the Propeller to execute "lua" on each reboot. To disable this from within Lua, just delete the file by executing a Lua command like: os.remove("AUTOEXEC.TXT") Lua analogues of the keyboard HMI functions: k_get, k_wait, k_new, k_ready, k_clear Lua analogues of the screen HMI functions: t_geometry, t_mode, t_setpos, t_getpos, t_scroll, t_color, t_color_fg, t_color_bg Lua analogues of the mouse HMI functions: m_button, m_abs_x, m_abs_y, m_delta_x, m_delta_y, m_reset, m_bound_limits, m_bound_scales, m_bound_x, m_bound_y Each of these HMI functions accepts the same parameters and returns the same values as their C counterparts. See the Catalina Reference Manual for details. Note that to save space, the mouse functions will not be included if the Catalina symbol NO_MOUSE is defined when Lua is compiled (which it will be if you use the "build_all" scripts to build Lua - to change this you can edit the file Makefile.Catalina to remove the "-C NO_MOUSE" option). You can detect whether mouse functions have been included from within Lua itself by testing if any of them are nil, such as: if (propeller.m_button) then -- mouse functions have been included else -- no mouse functions end 4. Catalyst will now try to execute commands as Lua scripts, in addition to just executing binary files. The order of priority for Catalyst command execution is now: 1. As a built-in command (e.g. "dir") 2. As a Lua script, adding a ".lua" extension if none is specified 3. As a binary file, using the command as the file name 4. As a binary file, adding the following extensions (in this order if the command does not specify an extension): .BIN .BIX (propeller 2 only) .XMM .SMM .LMM Note that all command types accept command line arguments, including Lua scripts. Several example Lua scripts are included: list.lua - a simple directory listing program (similar to Unix "ls"). find.lua - a simple file searching program (similar to Unix "find"). freq.lua - a simple word frequency counting program. You invoke Lua scripts from the Catalyst command line just like any other command. For example: list <-- list file details of all files list *.lua <-- list file details of Lua files find *.bas PRINT <-- find PRINT statements in all BASIC files freq *.txt <-- count word frequency in all text files Note that Lua scripts can be compiled to improve load and execution times, but they should still have the extension ".lua". As a consequence of this, the Lua versions of the Super Star Trek demo programs have changed name, so that executing the command "sst" at the Catalyst prompt will still invoke the C version ("sst.bin") rather than now executing the Lua version (which used to be called "sst.lua"): sst.lua --> now called star.lua sst-tos.lua --> now called star-tos.lua 5. The Propeller 2 Catalina command line argument processing has been modified to match that of the Propeller 1. On startup, all Catalina programs check if CogStore is running. If it is, the program fetches any arguments stored in it. Previously on the Propeller 2, if CogStore was not running then Catalina would set argc to zero and argv[0] to NULL - but some C programs expect argc to always be at least 1, so now if CogStore is not running argc will be set to 1 and argv[0] will point to a string with the value "null" (since without CogStore the real program name is not available). This functionality was already implemented on the Propeller 1. 6. The wildcard/globbing doDir() function in storage.h and storage.c has been extended to make it more useful - it now calls the file processing function with the file size and the DOSFS file attributes of each file in addition to the file name. One advantage of this is that programs that use doDir() do not need to use DOSFS functions to retrieve the file attributes themselves, and therefore only need to use standard C stdio file functions. 7. The build_catalyst script now detects whether it is being run in the current directory. If so, it builds catalyst in this directory, otherwise it builds it in the demos\catalyst folder in the Catalina installation tree (as it did in previous releases). The purpose of this is that if you copy the Catalyst folder to your own local user directory, you can build Catalyst locally and do not need to have write permission for the Catalina installation tree. Note that the build_all script already supported local builds, it was only the build_catalyst script that did not. 8. The build_utilities script now detects whether it is being run in the current directory. If so, it builds the utilities in this directory, otherwise it builds them in the utilities folder in the Catalina installation tree (as it did in previous releases). The purpose of this is that if you copy the utilities folder to your own local user directory, you can build the utilities locally and do not need to have write permission for the Catalina installation tree. Also, the build_utilities script has been modified to prompt whether the utilities binaries should also be copied to Catalina's bin directory, or simply left in the current directory. Copying the utilities to Catalina's bin directory is convenient since payload looks for them there if it does not find them in the current directory, but it requires write permission to the Catalina installation tree. If you do not have this permission, you can now copy the utilities folder to your own user directory, build the utilities there, then copy the binaries to each directory from which programs will be loaded - which may actually be a better solution if you have multiple Propeller platforms or configurations which need different versions of the utilities. Note that the utilities are required only for Propeller 1 platforms - there are currently none needed for any of the supported Propeller 2 platforms. 9. The build_all script in the utilities folder has been removed (it was deprecated quite a few releases ago). Use the build_utilities script instead. The main difference is that build_all used to build binaries for all CPUs of multi-CPU platforms (such as the TRIBLADEPROP) whereas the build_utilities script has to be re-run for each CPU. Other Changes ------------- 1. Implemented a workaround for an OpenSpin/Spinnaker bug that meant compiling Spin programs with too many short symbol names could fail unexpectedly. The current workaround is to simply use longer symbol names. Affected only Propeller 1 programs that used the Catalina Optimizer (which may generate many additional symbol names). 2. Fixed a typo in the DOSFS demo program that prevented it from compiling. 3. Updated the version of Lua used in payload and blackbox to Lua 5.4.4, which is the current version. Lua 5.1.5 is still included as a Catalyst demo program, but is no longer compiled by default (Lua 5.4.4 is now used everywhere) and Lua 5.1.5 is now deprecated and may be removed from a future release. 4. Since Lua is now compiled in COMPACT mode by default, the pre-compiled P2 demo versions of Catalyst (in P2_EVAL.ZIP and P2_VGA.ZIP) are now compiled in NATIVE mode, which improves execution speed, but at the expense of larger executables for some of the demo programs. If this is a problem, simply recompile Catalyst, specifying COMPACT mode as one of the parameters to the "build_all" script. 5. Updated the notes about compiling Catalyst - a memory model should only be specified as an argument to the "build_all" script when compiling for the Propeller 2, not for the Propeller 1.I have just updated the Catalyst spin demo (located in the "demos/catalyst/demo" folder of the Catalina installation tree) to add a version for the Propeller 2. Unzip the attached zip file over the contents of that folder (it is a complete replacement) or to a separate folder.
This demo shows how Spin programs can be executed from Catalyst and passed command-line arguments, just like C and Lua programs.
Demo.spin only supports the Propeller 1, but I have added Demo.spin2 for the Propeller 2. You can use the Propeller Tool to compile them both. I used version 2.7.0.
Ross.
EDIT: This is also included in the Catalina 5.3.1 patch, which is now available.
I have just posted Catalina 5.3.1 to SourceForge here.
This is a patch release that must be installed over Catalina 5.3 (Windows or Linux). The purpose of this patch is to fix a bug I found (finally!) in the Propeller 2 SD Card plugin. There are a few other minor additions and fixes. Here is the relevant portion of the README.TXT:
I have just posted Catalina 5.4 to SourceForge here.
This is a full release, primarily to update Catalina to add PSRAM support on the Propeller 2, and Catalyst to add HIRES_VGA support on the Propeller 1, and also add enhanced Catalyst scripting capabilities on both the Propeller 1 and 2. Plus a few significant bug fixes.
Here is an extract of the README.TXT file concerning the new functionality - the full file is too long to include here, but is attached:
RELEASE 5.4 New Functionality ----------------- 1. Roger Loh's 16 Bit PSRAM driver has been added as a Catalina plugin. It is supported on the Propeller 2 only. It is enabled by linking with the new psram library (i.e. adding -lpsram to the Catalina command). An example of its use has been added in "demos\examples\ex_psram.c". The configuration parameters for the driver must be specified in the platform files in the target_p2 directory, such as P2_EDGE.inc. You would compile the PSRAM example program with a command like: catalina -p2 -lc -lpsram -C P2_EDGE ex_psram.c The only tested platform is the P2 EDGE, and like the driver itself, Catalina's support for it should be considered a beta version until further notice. 2. As a demonstration of the use of the PSRAM plugin, Lua now has the option to store code in PSRAM on those platforms that support it, such as the P2 EDGE. This allows larger Lua programs to be executed at the cost of a slight speed reduction. Supported on the Propeller 2 only. To enable the use of PSRAM in Lua, specify ENABLE_PSRAM to the Catalyst or Lua build_all scripts. For example: build_all P2_EDGE SIMPLE VT100 ENABLE_PSRAM Note that PSRAM is supported only by the Lua execution engine (luax) which executes compiled Lua programs, and not for the interactive version (lua) that executes text programs or the Multiprocessing version (mlua or mluax). So if ENABLE_PSRAM is specified, only luax will be built. This means you may need to build Lua twice - once to build the lua programs that do not use PSRAM, and then again to build luax (only) to use PSRAM. For example, to compile Lua in directory "demos\catalyst\lua-5.4.4" and put the executables in "demos\catalyst\bin" and call the PSRAM version luaxp rather than overwrite the standard luax binary, you might use commands such as: cd demos\catalyst\lua-5.4.4 build_all P2_EDGE SIMPLE VT100 copy src\*.bin ..\bin\ build_all P2_EDGE SIMPLE VT100 ENABLE_PSRAM copy src\luax.bin ..\bin\luaxp.bin Note that specifying ENABLE_PSRAM is applicable only when using the Catalyst and Lua build_all scripts and Makefiles - it is not a general Catalina symbol that can be used on the Catalina command-line to enable PSRAM in other cases (which is done via the usual mechanism of linking the program with the psram library - i.e. adding -lpsram to the catalina command). Finally, note that for small Lua programs, the Hub RAM usage of the PSRAM version may not be much smaller than that of the non-PSRAM version - it may even be larger. This is not only because of the additional PSRAM support code required, it is also because the PSRAM version allocates a fixed amount of Hub RAM on startup to use as a PSRAM cache, and for small Lua programs the cache may be larger than the program being loaded. However, the amount of Hub RAM used for Lua code will never increase beyond the cache size no matter how big the program code gets. 3. Catalyst on the Propeller 1 can now use the HIRES_VGA option. However, Hub RAM limitations mean that Catalyst itself needs to be built as an EEPROM program, and some of the Catalyst utilities may only work if they are built as LARGE programs. This means that while Catalyst itself will work in HIRES_VGA mode on all platforms, some utilities (such as cp & mv) may only work in HIRES_VGA mode on platforms with XMM RAM. Alternatively, you might build Catalyst itself in HIRES_VGA mode, but the utilities in LORES_VGA mode. To facilitate this, two new options have been added that can be used with the Catalyst build_all scripts: EEPROM_CATALYST specifies that the Catalyst binary should be built as an EEPROM program. LARGE_UTILITIES specifies that the Catalyst utilities should be built as LARGE programs. Whether you need to specify one or both of these options can depend on the other options used. For instance, if you need to use the cache to access XMM RAM, then you will generally need to use both of these options to build Catalyst in HIRES_VGA mode. For example, here is how you might build Catalyst to use HIRES_VGA on the C3: build_all C3 FLASH CACHED_1K HIRES_VGA EEPROM_CATALYST LARGE_UTILITIES When you build Catalyst to use LORES_VGA or HIRES_TV HMI option, you may also find that catalyst.binary exceeds the size of Hub RAM and in that case you can specify the EEPROM_CATALYST option, but you may not need the LARGE_UTILITIES option - e.g: build_all C3 FLASH CACHED_1K HIRES_TV EEPROM_CATALYST Note that to program Catalyst into EEPROM when the CATALYST_EEPROM option is used, you will need to run the build_utilities script to build the EEPROM loader, and then use that with payload. For example, to build and load Catalyst to use HIRES_VGA on the C3, you might use commands like: cd demos\catalyst build_all C3 FLASH CACHED_1K HIRES_VGA EEPROM_CATALYST LARGE_UTILITIES build_utilities payload -o1 EEPROM ..\bin\catalyst.bin Note that these new options are applicable only when using the Catalyst build_all scripts and Makefiles - they are not Catalina symbol that can be used in other circumstances (i.e. specifying -C EEPROM_CATALYST when compiling catalyst.c manually will not have any effect. It is the Makefile that intercepts this symbol and instead uses -C EEPROM, but only when building catalyst.binary). 4. The Catalyst ONCE capability (i.e. to execute a command once on reboot) has been extended to execute MORE than one command. If enabled when building Catalyst (by setting both ENABLE_ONCE and ENABLE_MORE to 1) then the file (EXECONCE.TXT by default) can contain more than one command. The commands in the file will be executed in sequence on successive reboots. The Lua "execute" function can be used to easily add multiple commands to the file, one per line. For example, if the MORE capability is enabled, the Lua statement: propeller.execute("vi abc\n vi def") will cause the propeller to first reboot and execute the command "vi abc", and then when vi exits, the propeller will reboot and execute the command "vi def". This also allows a very basic scripting capability to be implemented. For example, if you have a file called commands.txt which contains all the commands you want executed, then executing the command: cp command.txt execonce.txt at the Catalyst prompt, or executing the Lua statement: propeller.execute("cp command.txt execonce.txt") will cause all the commands in the file to be executed in sequence. Note that this capability is enabled by default on the Propeller 2, and on the Propeller 1 unless the HIRES_VGA HMI option is used, because there is not enough Hub RAM available. If you want to enable it, you may need to disable something else, such as the capability to allow Lua commands to be executed directly from the command line (note that you can still execute them by specifying them as parameters to Lua - e.g. by entering a command like "lua list.lua" instead of just "list"). This can be disabled by editing "demos\catalyst\core\catalyst.h" and rebuilding Catalyst. 5. The internal cat, dir and help commands have been removed from Catalyst. The help and cat commands are now always external. The dir command has simply been dropped since the ls command is far more capable. However, if you prefer typing "dir" to "ls", then just make a copy of "ls.bin" called "dir.bin" - i.e: cp ls.bin dir.bin 6. Added a new Lua demo called "wild.lua" that can be used to add wildcard capability to a Catalyst command that accepts multiple file names. For example, if the command specified in wild.lua is "vi" (which it is by default) then luac -o xvi.lux wild.lua will create a command "xvi" which can then be used on the command line to invoke vi on multiple files - e.g: xvi ex*.lua will execute vi on all the Lua example files. 7. The length of a Catalyst command line has been increased to 300 characters on the Propeller 1, and 1200 characters on the Propeller 2. Hub RAM is limited on the Propeller 1, but 300 is enough to accommodate 24 MSDOS 8.3 filenames, which could be generated when using wildcards, and 1200 is the maximum that CogStore can store. This maximises the potential usefulness of the wildcard functionality added in the last few releases. 8. A new "getrand()" function has been added to the C library. It is defined in "propeller.h" and is implemented on both the Propeller 1 and the Propeller 2 (but differently - see below). A program that demonstrates the use of the function has been added in "demos\examples\ex_random.c" On the Propeller 1, the first time this function is called it calls srand() with the current CNT value and is therefore best called after some user input or other random source of delay. It then returns the result of 3 combined calls to rand() to make up 32 random bits (rand itself only returns 15 bits). On the Propeller 2, the first time this function is called it calls srand() with the result of the GETRND opcode, and also returns that value. Thereafter it just returns the result of the GETRND opcode. This means you can either use just this function, or use this function once to generate a seed for srand() and thereafter use rand(), which is what most traditional C programs would typically do. Note that rand() only returns a value between 0 and RAND_MAX (inclusive) (i.e. 0 .. 32727 on the Propeller) whereas getrand() returns 32 bits. To simulate rand() using getrand(), use an expression like: (getrand() % (RAND_MAX + 1)) 9. On the P2_EDGE, the base pin for VGA has been changed to 0 (was 32), and the base pin for USB has been changed to 8 (was 40). This makes it possible to use the P2-ES VGA and USB accessory boards with Catalina on the P2_EDGE. The base pins can be edited if required in the file P2_EDGE.inc in the target_p2 folder.Oops! A silly mistake in the Propeller 1 XMM startup code meant an XMM SMALL program loaded using Catalyst could not access command-line arguments. Does not affect other program types on the Propeller 1, or the Propeller 2.
To fix it, just drop the attached file over the existing one in Catalina's target directory and recompile any affected programs (you don't need to recompile either Catalina or Catalyst).
This will also be included in the next release, which is likely to be a very small patch release unless I find any other problems.
UPDATE: This file is now also included in Catalina patch release 5.4.1
Ross.
I have just posted Catalina 5.4.1 to SourceForge here.
This is a patch release that must be installed over Catalina 5.4 (Windows or Linux). The purpose of this patch is to fix the bug that was preventing XMM SMALL programs using command line arguments when loaded using Catalyst, plus a few other minor changes. Here is the relevant portion of the README.TXT:
RELEASE 5.4.1 New Functionality ----------------- 1. p2asm now supports the "IF_NN" and "IF_NOT_NN" (where N=0,1 or X) PASM instruction prefixes, as well as "IF_SAME" & "IF_DIFF". For example, the following two statements will generate the same PASM instruction: if_c_or_z mov r0, #1 if_not_00 mov r0, #1 Other Changes ------------- 1. Programs compiled in SMALL mode and loaded using Catalyst were not setting up the argc and argv parameters to the C main function correctly. Affected the Propeller 1 only. 2. The arg.c argument diagnostic program in the utilities folder was not working correctly on the Propeller 1. 3. It was not clear in the Catalyst documentation that when Catalyst is used to load XMM programs, then both Catalyst and the XMM programs must use the same caching and flash options. For example, if Catalyst is compiled using FLASH CACHED_1K then so must all XMM programs (but note that they can be SMALL or LARGE programs). This only applies to XMM programs, and not to LMM, CMM, SMM, Spin or Lua programs. Applies to the Propeller 1 only. 4. The Catalina Optimizer was failing to perform some optimizations when the optimization level was 3 or greater, so that -O3 could result in a larger code size than -O2. Affected all memory models except COMPACT on both the Propeller 1 and the Propeller 2.Just a quick - but important - update on Lua on the Propeller 2 ...
I have just realized that Lua can very easily take advantage of the Propeller 2's ability to read and write data to Hub RAM without requiring the address to be aligned on a particular boundary (e.g. you can read or write a long to any address, even if that address is not a multiple of 4).
This gets around a significant problem using Lua in memory constrained environments, which occurs because Lua is a dynamically typed language. That means that every value in Lua (even a simple integer) must be tagged to tell Lua what type it is (e.g. that the value should be interpreted as an integer and not as a float or a pointer to a string). This requirement means that on most processors, Lua requires 8 bytes to store each 4 byte value - 4 for the value and another 4 for the tag, because even though the tag only needs 1 byte, the alignment requirements of most processors force anything that follows the tag to the next 4 byte boundary, so effectively the tag also takes 4 bytes. For things like arrays this can be a real problem because it can double the memory required!
But the Propeller 2 can do this in 5 bytes rather than 8 bytes - 4 bytes for the value and 1 byte for the tag, at the trivial cost of one additional clock cycle per access. Chalk one up for the Propeller 2!
By itself, this can reduce a typical Lua program's memory requirement by 5-10% - more in some cases. When combined with the PSRAM support, it can reduce a typical Lua program's memory requirement by 20-30% - more in some cases.
This will be in the next release of Catalina. But it turns out to be so easy that if anyone is interested in a preview, just drop the attached files over the existing ones in demos\catalyst\lua-5.4.4\src and then rebuild Lua. Note that the alignment fix will work on all Propeller 2 platforms, but PSRAM is currently only supported on the P2 EDGE, and to enable it you must add ENABLE_PSRAM on the command line - e.g:
Ross.