Catalina - ANSI C for the Propeller 1 & 2
I guess I should start a new thread for Catalina, instead of continuing the separate ones for the P1 and P2. I will add new updates here.
Catalina is an open source ANSI C compiler. It support the Propeller 1 and Propeller 2.
Catalina is available here.
The current version is 5.1 on both Windows and Linux.
I am releasing Catalina 4.9.5. The main thing this release adds is enhanced Lua support and a couple of bug fixes. See the README extract below.
Lua is lots of fun, and well suited to the Propeller (especially the Propeller 2). For those unfamiliar with it, here is a description of Lua from lua.org:
In addition to adding Lua 5.4.3 (the latest version) I have upgraded both Catalina and Catalyst to use Lua 5.1.5, because this version adds the ability to not only run an interactive Lua interpreter and a Lua compiler, but also to execute pre-compiled Lua programs using less memory, which enables the execution of larger Lua programs on the Propeller.
Here is the relevant section of the README.TXT file.
Both Windows and Linux versions are now available.
I have had a question about integrating C with Lua. This is very easy to do, and there is a minimalist example (min.c) that is included with Lua-5.1.5 in Catalina 4.9.5. However, this example is not so easy to execute using the payload terminal emulator because when you run the program, payload terminates before it prints the output the program produces.
The min.c program reads stdin and expects the input to be terminated with an EOT (i.e. Ctrl-D) before it produces output to stdout and/or stderr. But Ctrl-D is also the signal to payload to terminate, so you never see the output. To fix this, I have added the ability for payload to send an EOT to the application without terminating. Just press Ctrl-D once to send EOT. To terminate payload you now have to enter TWO CONSECUTIVE Ctrl-D characters.
Copy the new payload executable over the existing payload.exe in your Catalina\bin directory. The amended source (payload.c) is also included, and will also be included in the next release of Catalina.
The program min.c is in the Lua-5.1.5\etc folder, but the easiest way to compile it is to copy it to the Lua src folder, then build the Lua library and then compile the min program. For example, you could use commands like:
To load and run the program, use the new payload in interactive mode:
payload -i -b230400 min.bin
Then in the payload window, type the following:
Make sure to press ENTER after the final line. Then press Ctrl-D. You should see output like:
Press Ctrl-D again to exit payload. The min.c program interprets any Lua statements entered via stdin, but it loads no Lua libraries - the "print" function you just executed is not the normal Lua print function - it is the C function in min.c. This minimalist example shows how easy it is to call Lua from C, and also C from Lua!
Here is the entire min.c program:
You can perhaps see why I find Lua so interesting!
I have just issued a small patch for Catalina - release 4.9.6. It is available on SourceForge. Please note that this is NOT a full release. It must be installed over an existing installation of Catalina 4.9.5. It includes all new sources and Windows binaries, but for Linux you will need to recompile Catalina from source after you install the patch.
The main purpose of release 4.9.6 is to tidy up a few outstanding issues before the next full release of Catalina, which may be a while away because it will introduce some significant changes. For example, the next full release will include a thread-safe version of malloc, and multithreading support for Lua.
Here is an extract from the README.TXT:
Just a quick progress report. I have just finished making Catalina's memory management functions (malloc, calloc, realloc, free etc) thread-safe, which also makes them safe to execute from multiple cogs and also from multiple memory models.
As a test I thought I'd try executing multiple parallel Lua interpreters. Turns out to be quite trivial.
Here is an example program which executes four Lua interpreters, each one running on a separate cog. Currently the Lua interpreters are pretty much independent, and can only communicate via C shared functions and global variables (such as is demonstrated by the "print" function in this example):
Here is the actual output produced when the above program is run (on a Propeller 2). It is pretty much what you would expect if all the Lua interpreters are in fact executing in parallel:
Of course, this program requires Catalina 5.0, which is not released yet!
The next step is to add multi-threading support to Lua (i.e. in addition to just multi-cog and multi-model support). With Catalina's multi-threading, this should be trivial, and should allow the execution of dozens of concurrent Lua interpreters, not bounded by the number of available cogs (as the example above is!).
However, I am not sure yet that this will be particularly useful. Ideally I would prefer to be able to run individual Lua functions as threads - i.e. to offer a much finer granularity of threading than just instances of the Lua interpreter.
Time will tell.
I don't suppose there is any chance for a Mac release?
I never did Mac releases, but I used to use a virtual Mac to make sure Catalina would compile and basically run. So the Linux version of Catalina used to be able to be compiled on a Mac, but we are going back quite a few years, and I doubt very much that it would do so now.
You're probably right that it won't compile out of the box if you haven't done anything to make that happen. I remember trying a while back and there were some problems that I didn't have time to resolve.
As expected, the multi-threaded version proved to be trivial, but worth doing to verify (before I dig deeper into making Lua multi-threaded) that once the memory management functions are made thread-safe, so is Lua.
Here is a very similar program to the multi-cog version above, but this one is not limited by the available cogs - it executes 20 Lua interpreters in separate threads:
The output is exactly what you would expect. Here are just the last few lines:
Now comes the hard part ...
A quick update on Catalina 5, and a note about a strange Windows 10 bug that affects Catalina's payload loader ...
First, the bug ...
The Windows 10 Command-Line interpreter (cmd.exe) crashes and burns if you execute payload in a command window with the "Wrap Text Output on Resize" option unchecked. This option is on the Layout tab of the cmd.exe Properties dialog box. Not only does cmd.exe crash, it leaves payload executing in the background, hanging onto the executable it was loading (which then cannot be updated) and it also takes out the Windows 10 USB subsystem. I have sometimes had to reboot Windows to get it all working again. Took me a while to track this one down, and I am still not sure exactly what the problem is, but the workaround is easy - just make sure this option is selected in cmd.exe if you use the payload loader.
Next, a quick update on Catalina 5 ...
All looking very promising. I now have Lua fully multithreaded. It turned out the easiest way to test out various existing Lua multi-threading extensions was to implement the Posix thread library (pthread). So now I have a pthread library implemented using Catalina's existing thread subsystem. Compared to Catalina's lean and mean threads, Posix threads are a bloated abomination designed by a committee to be impossible to use - but they are a common way of implementing platform independent multi-threading in C, so I gave in and implemented a thin binding.
Catalina's pthread.h is attached, which summarizes Catalina's level of support (which is fairly complete).
Here is a simple example of a producer/consumer problem implemented with pthreads:
and here is the output when compiled and executed with Catalina 5.0:
Catalina's threading subsystem is so efficient that Catalina can support pthreads on any Propeller 1 platform, as well as on the Propeller 2. When compiled in COMPACT mode, the above program takes less than 7k of Hub RAM on a Propeller 1 or 2.
I expect to release a first version of Catalina 5 as soon as I have finished documenting all the new features.
Not 100% happy with this yet, but here is an example of using the new "threads" package in Lua:
That's the complete program. One thing I like about Lua is that it is even more concise than C. There is very little unnecessary or obscure syntax.
You can probably guess the output this program produces, but here it is anyway:
Clearly, there is a lot more of the underlying pthreads capabilities that could be exposed. But I'm not yet sure how much is either necessary or desirable. This is probably too simple as it stands - it only has message passing as the mechanism for interaction between threads - but on the other hand pthreads is way too complex.
I'm open to suggestions as to what would constitute a good set of minimal "primitives" for Lua.
to me this looks like JMPRET on the P1.
It does not run in parallel but just passes the 'ball' but easy on the eyes, the source.
I agree it is hard to spot the difference with such simple examples. But not impossible. Here is another complete Lua program (this one is even simpler, and correspondingly less interesting - but it does exhibit true multi-threading):
And here is the actual output:
The formal name is cooperative multi-tasking.
Both pthreads and Catalina are pre-emptive. But native Lua supports only co-routines, which are cooperative. This is one reason adding multi-threading to Lua is non-trivial.
I have done some more investigation and more thinking and have decided on the following set of primitives:
The main reasons for this are practical - (1) is already implemented in the package (luaproc) I am basing my threads package on, and (2) and (3) are logical extensions of this message passing paradigm. There is already an implicit "mutex" in receiving a message (only one receiver can get each message), and an implicit "condition" in sending a message (sending blocks the sender until there is a receiver ready).
Nice. Sorry, I hadn't been reading and just reacted to Mike's observation.
@RossH I just got the 3Color Eink program working with Catalina (P1), but have some questions, if you don't mind.
Most of the work was moving variable definitions to the top of the functions, not such a big deal.
Rest of changes documented in the attached.
There are some things I don't understand...
Is "propeller.h" automatically included? If I try to include it, I get errors, looks like it's including propeller2.h for some reason...
Does #warning work? Should it?
There are some things in flexprop C's propeller.h that I don't think are here, right? Like:
getpin(), togglepin(), setpin()
I tried using Geany, but I seem to have forgotten the trick for making it work.
Get "Process failed (The system cannot find the file specified)" when trying to make or build.
Vaguely remember something about needing a .geany file for this to work, right?
Sounds like you're working on making Geany faster. That's good.
The errors you found are perfectly legitimate, and are due to differences between ANSI C and other C variants. Here are some explanations:
Yes, ANSI C requires all declarations to precede all code in the block. This rule was relaxed in C99, but with some consequences. If you want to see why this was not such a brilliant idea, try guessing what the following program produces before running it (it compiles and runs with gcc but will not compile with ANSI C):
The only place this feature ever made sense was in declaring the control variable for for loops in the loop (as in "for (int i =1; i < 100; i++)". Which, to be fair, is where it mostly gets used.
Yes, only #error is supported by ANSI C. AFAIK #warning (and #info) were added by specific compiler vendors and are not part of any C standard.
Problem: #include <propeller.h> -->C:\Program Files (x86)\Catalina_4.9\include/propeller2.h:100: redeclaration of cogstop previously declared at C:\Program Files (x86)\Catalina_4.9\include/catalina_cog.h:80
You should not include "propeller2.h" if compiling for the Propeller 1!
This is actually required by C. But some compilers are more relaxed about it than others.
Yes, Catalina offer many more library options than C - mostly to do with whether or not floating point and full streams support is included.
Try using the optimizer - "-O5"
The remaining errors are due to differences in the propeller-specific libraries and header files - we never standardized these on the P1 as we did on the P2.
I wasn't including propeller2.h... I got that error when including propeller.h. No idea why...
I find the functions: getpin(), togglepin(), setpin()
to be very useful. It was a bit of a chore to set particular pins without it.
Also, I don't seem to have direct access to, for example, DIRA. Am I missing something there?
That is odd!
I'll look at adding these to the next release.
Again, odd. These are defined in "propeller.h".
Can you post your source?
Code is posted already in top post of the seeed studio 3 color eink thread in this category.
I’m curious as to why including propeller.h doesn’t work…
The include was in platform.h …
Maybe has to to in main file?
The problem seems to be with "SPI.c", which includes "propeller2.h". If you remove that, you can include "propeller.h".
No idea why I did that.
Guess that explains it.
Deleted because there is a new version of Multi-threaded Lua available. See later in this thread.
Deleted because there is a new version of Multi-threaded Lua available. See later in this thread.
Catalina 5 is nearly ready for release, but the final packaging process always takes much longer than I expect, especially as there have been many more changes and improvements in Catalina than I originally planned. While I am getting it ready, attached is the final release candidate of a Multi-threaded version of Lua, compiled for the Propeller 2 Evaluation board.
It is a self-contained Lua development environment for the Propeller 2, and now includes a comprehensive tutorial and reference manual. You do not need Catalina at all to install or run it. The sources are not included - but they will be in the release.
If anyone has any suggestions for improvement of the tutorial, I may have time to incorporate them in the final release of Catalina 5.
EDIT: The attached file has been updated to reflect Catalina 5.0.2
I was just about to release Catalina 5 on SourceForge when I noticed that SourceForge is now flagging some previous Catalina releases as malware affected. I have sent the same files off to various online virus scanners and also virus checked my PC with multiple anti-virus programs (including the same ones SourceForge uses!) and they are all clean.
I am fairly sure this is a false positive by SourceForge. However, I'll spend some time trying to sort this out before I upload the new version
Ok. I seem to have SourceForge sorted, and have now uploaded a release candidate of Catalina 5.0 for Windows. A corresponding Linux release will follow when I get more time (very busy with flooding here in Australia at present!).
Also, I have updated the compiled version of multi-threaded Lua (see https://forums.parallax.com/discussion/comment/1536442/#Comment_1536442) to match the released version (which saves you having to compile it yourself).
Apart from bug fixes, this is about as far as I can go with multi-threaded Lua without making internal changes to Lua itself (which is something I have managed to avoid doing so far). But for a high-level language like Lua (it is a much higher level language than C, supporting both object-oriented and functional programming) the Propeller is quite memory constrained - so much so that code size and garbage collection can become significant issues for non-trivial programs.
Instead of frigging about trying to improve Lua's memory management, I may reconsider adding support for extended memory on the Propeller 2 - this would allow Lua to become much more useful without needing internal changes.
Here is the "New Functionality" section of the README file for Catalina 5.0:
Oops - just noticed a typo in the previous post, which is also in the README.TXT for Catalina 5.0 - togglepin(pin) sets pin to output (not input) and toggles its value. The software is correct, it is the README.TXT that is wrong.