Catalina is now on GitHub

RossH · 2024-07-21 06:22

The current Catalina release is now also available on GitHub - see here.

I tried using GitHub in the past, but it always seemed to require far more effort than it was worth - but the last few Catalina releases have all been minor bug fix releases, and my usual release process in such cases takes more time to build the release than actually finding or fixing the bugs.

So I am again going to try hosting Catalina on GitHub, and use it for minor releases. I will still make major releases available on SourceForge as well.

One reason I didn't like using GitHub for releases is that the installation process is much more difficult for Windows users - because GitHub does not provide an easy way to create Start Menu items etc, requiring you to do everything manually, which is a real pain. Or at least I have not figured out how to do it automatically (if there are any Git gurus out there who know how to do it, by all means let me know!).

However, I recently found a utility that allows me to create Windows Start Menu entries using a simple batch script, so I have bundled this into the GitHub release. This makes the Windows installation experience much better.

Note that the GitHub release contains only Windows binaries - if you want to install on Linux you will have to either use the SourceForge release or rebuild the GitHub release from source - but the Linux build process is at least fairly simple (whereas the Windows one is quite complex). See the new README.md and BUILD.TXT in the release for more details.

I'd appreciate it if people could let me know how the GitHub installation process goes on this thread, and keep GitHub issues off the main Catalina thread (at least until I decide whether or not I am going to stay with GitHub).

Ross.

macca · 2024-07-21 07:54

@RossH said:
One reason I didn't like using GitHub for releases is that the installation process is much more difficult for Windows users - because GitHub does not provide an easy way to create Start Menu items etc, requiring you to do everything manually, which is a real pain. Or at least I have not figured out how to do it automatically (if there are any Git gurus out there who know how to do it, by all means let me know!).

I don't define myself a "guru" but my Spin Tools IDE is hosted on GitHub and uses its runners to build the release packages. I'm not using an installer for the Windows package, but the Windows runner provides a number of pre-loaded package managers. See this for reference https://github.com/actions/runner-images/blob/main/images/windows/Windows2022-Readme.md

I did a quick look at the BUILD.TXT file and AFAIK msys2 is already installed on the Windows runner (I'm not sure about the versions), I think it is just a matter of writing the correct workflow script.

RossH · 2024-07-21 08:13

@macca said:

@RossH said:
One reason I didn't like using GitHub for releases is that the installation process is much more difficult for Windows users - because GitHub does not provide an easy way to create Start Menu items etc, requiring you to do everything manually, which is a real pain. Or at least I have not figured out how to do it automatically (if there are any Git gurus out there who know how to do it, by all means let me know!).

I don't define myself a "guru" but my Spin Tools IDE is hosted on GitHub and uses its runners to build the release packages. I'm not using an installer for the Windows package, but the Windows runner provides a number of pre-loaded package managers. See this for reference https://github.com/actions/runner-images/blob/main/images/windows/Windows2022-Readme.md

I did a quick look at the BUILD.TXT file and AFAIK msys2 is already installed on the Windows runner (I'm not sure about the versions), I think it is just a matter of writing the correct workflow script.

Thanks. I will investigate. Anything that simplifies the Windows build process is very welcome. Fortunately, most Catalina users will never have to do that. It's only me that has to endure that particular recurring nightmare.

P.S. I've just made some updates to the release, including the BUILD.TXT document, but they are for Linux, not Windows.

Wuerfel_21 · 2024-07-21 12:25

You generally never want to have any sort of build artifact (i.e. EXE files and the like) actually committed to your repository - they'll take up ever-increasing space on the repository and it's just unpleasant in general. What you actually want is to upload a "release" to github. This can be done manually, but you can also make it automatic.

I've set up the github actions stuff for the flexspin commandline tools. So on every commit it will:

run all "offline" tests ("does compiling this input give the expected ASM output" etc)
run all P1 test programs using a customized version of spinsim
build the tools for Windows
build the tools for musl/Linux
if commit has a tag like v1.2.3, create a release and upload the compiled tools (yes, this still works if the tag is added later)

Setting this up is principally very simple, it's just some YAML files (that really just call makefile tasks): https://github.com/totalspectrum/spin2cpp/tree/master/.github/workflows
In practice it's quite annoying because of the huge turnaround between changing something and the server coming back with a result.

macca · 2024-07-21 14:09

@RossH said:

@macca said:

@RossH said:
One reason I didn't like using GitHub for releases is that the installation process is much more difficult for Windows users - because GitHub does not provide an easy way to create Start Menu items etc, requiring you to do everything manually, which is a real pain. Or at least I have not figured out how to do it automatically (if there are any Git gurus out there who know how to do it, by all means let me know!).

I don't define myself a "guru" but my Spin Tools IDE is hosted on GitHub and uses its runners to build the release packages. I'm not using an installer for the Windows package, but the Windows runner provides a number of pre-loaded package managers. See this for reference https://github.com/actions/runner-images/blob/main/images/windows/Windows2022-Readme.md

I did a quick look at the BUILD.TXT file and AFAIK msys2 is already installed on the Windows runner (I'm not sure about the versions), I think it is just a matter of writing the correct workflow script.

Thanks. I will investigate. Anything that simplifies the Windows build process is very welcome. Fortunately, most Catalina users will never have to do that. It's only me that has to endure that particular recurring nightmare.

Oh.. I haven't realized that you have uploaded the binary artifacts to the repository (and that explains your statement about Github not providing an easy way to create Start Menu items...). Never do that!

The git repository should have only the source files and the instructions or scripts needed for a user to build the product.
Binary artifacts can be uploaded to the Releases pages, just like what you do with the Files pages on Sourceforge (except Releases also holds a source snapshot).
As a first step you can just do that, build the packages on your system and upload the binaries manually, then you can fully automate the process with workflow scritps.

Workflow scripts may be a bit complicated to write correctly, I suggest to create a private repository or a test/disposable repository to test them, and maybe start small with a simple "Hello World" program. Beware that the free account has 2000 minutes/month of actions time and that Windows runners have a 2x multiplier (effectively translates to 500 minutes/month limit). You can also self host the runner on your system (this doesn't have any limit), I'm using the self-hosted runner to build the Raspberry Pi packages.

Wuerfel_21 · 2024-07-21 15:08

@macca said:
Workflow scripts may be a bit complicated to write correctly, I suggest to create a private repository or a test/disposable repository to test them, and maybe start small with a simple "Hello World" program. Beware that the free account has 2000 minutes/month of actions time and that Windows runners have a 2x multiplier (effectively translates to 500 minutes/month limit). You can also self host the runner on your system (this doesn't have any limit), I'm using the self-hosted runner to build the Raspberry Pi packages.

Is offering windows runners a new thing? I could swear that wasn't a thing some time ago. I just made it cross-compile the windows builds from the normal linux environment. MinGW isn't pre-installed, so it'll have to do that every time, but for longer builds, it probably beats "paying" extra for the windows runner (does anyone actually need 2000 minutes/month? Seems high enough that for reasonable usage, no one actually has to buy the paid plan)

macca · 2024-07-21 18:22

@Wuerfel_21 said:

@macca said:
Workflow scripts may be a bit complicated to write correctly, I suggest to create a private repository or a test/disposable repository to test them, and maybe start small with a simple "Hello World" program. Beware that the free account has 2000 minutes/month of actions time and that Windows runners have a 2x multiplier (effectively translates to 500 minutes/month limit). You can also self host the runner on your system (this doesn't have any limit), I'm using the self-hosted runner to build the Raspberry Pi packages.

Is offering windows runners a new thing? I could swear that wasn't a thing some time ago. I just made it cross-compile the windows builds from the normal linux environment. MinGW isn't pre-installed, so it'll have to do that every time, but for longer builds, it probably beats "paying" extra for the windows runner (does anyone actually need 2000 minutes/month? Seems high enough that for reasonable usage, no one actually has to buy the paid plan)

I think it was made available for free accounts since couple of years at least, paid subscriptions always had Windows runners (AFAIK).
Recently it was made availble also the MacOS Mx (arm) runner (before it was available for paid subscriptions only), however it has an even worse 10x minutes mutiplier...
Consider that 2000 minutes are for the whole account, every project and every runner you need adds to the time. I think that it is not needed to upgrade the subscription, action minutes can also be purchased as needed.

RossH · 2024-07-22 00:52

Thanks for all the suggestions and advice.

I understand the point about not usually having binaries in a source code repository, but if I take them out altogether then GitHub has few advantages over building and bundling up the releases myself (as I currently do on SourceForge). Requiring end-users to build the release from source is generally ok for Linux end-users, but not for Windows end-users.

I can see how I could use a customized Windows "runner" to do the actual build and automated testing, and this avoids the issue of requiring Windows end-users to have all the required packages installed that are required to build Catalina (but not just to use it). Now that I have (I think) adequately addressed the Windows installation issue I will look into this in the longer term.

In the interim (as suggested by @Wuerful_21 and @macca) I will use GitHub for hosting the sources but put the binaries in a separately uploaded package associated with each release. This sounds like a reasonable compromise.

Ross.

RossH · 2024-07-22 07:32

Ok - I've created a new release on GitHub called v7.6.3-alpha-5 - the Windows binaries have been removed from the repository and added back in as an 'asset' associated with this release.

Let's see how this goes - the only issue I can see that is that if you just clone or download the repository, you may not realize there are pre-compiled binaries available for Windows. I added a suitable note to the README.md file, but who ever reads documentation?

I think we're nearly there. Any comments welcome!

Ross.

macca · 2024-07-22 09:08

@RossH said:
I understand the point about not usually having binaries in a source code repository, but if I take them out altogether then GitHub has few advantages over building and bundling up the releases myself (as I currently do on SourceForge). Requiring end-users to build the release from source is generally ok for Linux end-users, but not for Windows end-users.

Github as well as Sourceforge, Bitbucket, Gitlab and who knows what else, are basically source code repositories, the difference is made with the additional services they provide, either for free or paid. Downloadable files space, bug tracking, collaboration, wiki pages, discussion forums, etc. End users are not supposed to clone the repository to install the program, there are other facilities for that. As said, the repository should contains the source code and the instructions/scripts needed for an end user to build the product, if they can't or don't want to build it themselves then there should be other options.

In my opinion, GitHub currently has the most complete offering even for free accounts (put aside the polemics about Microsoft, IA scraping, etc. and, to be clear, I don't work for GitHub or Microsoft...). What I appreciate most is the build runners offering that allows to automatically build and publish releases, this is very helpful for multi-platform projects that needs to build on native systems. The ability to run custom Docker images also is a very useful feature, that allows to have a perfectly controlled build environment with the the needed tools. Other sites have similar features but with much more limitations.

@RossH said:
Ok - I've created a new release on GitHub called v7.6.3-alpha-5 - the Windows binaries have been removed from the repository and added back in as an 'asset' associated with this release.

Let's see how this goes - the only issue I can see that is that if you just clone or download the repository, you may not realize there are pre-compiled binaries available for Windows. I added a suitable note to the README.md file, but who ever reads documentation?

That's one way to point the users to the pre-built binaries.
For Spin Tools IDE I have a dedicated page on my web site https://www.maccasoft.com/spin-tools-ide/
Dropbox or Google Drive may be other options to distribute binaries.
Github includes wiki pages for each project, you can create a page that points to the binary installation files (not sure if there is a way to set the wiki as the project's home). I see you have a Patreon page, I can suggest to add a post pointing to the binary files (either public or reserved to subscribers, at your choice, of course). Definitely never ever use the source repository to distribute binaries (even third-party tools should be kept out of the repository if not absolutely necessary).

RossH · 2024-07-22 12:21

@macca said:

Github as well as Sourceforge, Bitbucket, Gitlab and who knows what else, are basically source code repositories ...

Yes, and I may continue to use GitHub for source code control and source code releases - it has improved significantly since last time used it. But I guess it is still not a good solution for distributing "ready to execute" binaries. And Catalina really needs both.

Ross.

RossH · 2024-07-23 11:36

Oops - the pre-release version still has a couple of GitHub hiccups ...

GitHub doesn't like empty folders
GitHub doesn't like Windows batch files

Now I remember some of the reasons I gave up on GitHub last time - it is really only "friendly" to Unix and Linux source code.

However, Google assures me these issues are fixable with a bit of frigging about. I will sort them out before making the GitHub release "official".

Ross.

macca · 2024-07-23 14:04

@RossH said:
Oops - the pre-release version still has a couple of GitHub hiccups ...

GitHub doesn't like empty folders

It is git, not github. Why do you need empty folders in the repository (and why can't you create them when needed) ?
If you absolutely need an empty folder the repository, just add a 0-bytes .keep file, like emptyFolder/.keep

GitHub doesn't like Windows batch files

What problems you have with batch files ?

Electrodude · 2024-07-23 14:28

@RossH said:
2. GitHub doesn't like Windows batch files

Now I remember some of the reasons I gave up on GitHub last time - it is really only "friendly" to Unix and Linux source code.

Yes, this is a problem with your Git client and not with Github. In a misguided attempt to make it more Windows-friendly, the Git for Windows people added a horrible setting core.autocrlf that is on by default on Windows. Make sure it is false, or it will do horrible things to line endings:

git config --global core.autocrlf false
git add --update --renormalize
git commit -m "Fix files damaged by core.autocrlf"

When I must use Windows, I just use normal Linux git inside WSL, which defaults to not butchering DOS and binary files, as one would expect.

EDIT: The rationale for this setting being on had to do with certain Windows editors apparently being unable to correctly handle Unix line endings, resulting in bad things like files with mixed line endings. When merging, Git can't just ignore whitespace changes because it would make everything look like a mess, so it often gives you merge conflicts over trivial-to-humans whitespace changes. Use a decent editor that doesn't mangle line endings and automatically removes or at least complains about whitespace at the end of lines, and you shouldn't have any of these problems.

RossH · 2024-07-24 00:11

Git changing line terminations is at least understandable, and this is what was damaging batch files. The solution to this one is easy - you just have to know to tell Git not to do it.

But the empty directories one is a bit baffling. Why should Git users have to make changes to their software to compensate for Git deficiencies? Especially when there is no real solution, just a bunch of workarounds - all of which turn out to be variations on "Don't do that, Git doesn't support it" - which is somewhat less than ideal.

Still, others seem to manage to live with Git's limitations, and I'm sure I will too. The hard part is finding them.

RossH · 2024-07-24 23:43

Another minor hurdle ... not really related to Git, more a Windows question ...

When I use a Windows installer to install Catalina, I give users the option of installing 'make' if they don't already have a version on their system (which most Windows users would not). This is easy and seamless to do using the installer, but not so easy for users to do manually.

To get the 'make' binary out of the Git repository, I need an easy alternative way for Windows users to install it. I could insist they manually install GnuWin or MinGW - but this seems a bit heavy-handed for users who just want to use Catalina (i.e. not rebuild it).

So I looked for alternatives and found ezwinports, which means Windows users can install 'make' just by executing the command winget ezwinports.make

This is the easiest method I have found so far for Windows 10 or 11 - but I have no experience with ezwinports.

Does anyone here have an opinion, or a better alternative?

Ross.

Rayman · 2024-07-25 21:41

@RossH Thanks for posting this! Have you posted the Catalina source before? I don't remember seeing it.

Think I'm seeing that it's based on LCC, probably explaining why it's so stable...

RossH · 2024-07-25 23:18

@Rayman said:
@RossH Thanks for posting this! Have you posted the Catalina source before? I don't remember seeing it.

Think I'm seeing that it's based on LCC, probably explaining why it's so stable...

Hello Rayman

Catalina's source has always been available - it is included in every binary distribution.

Yes, Catalina is based on LCC, which is a brilliant C compiler.

Ross.

RossH · 2024-07-28 23:44

Blast! I thought I had sorted out GitHub with release 7.7, but I've just realized the repository still has the Catalina Geany Windows binaries in it

Will sort this out in the next release and (like the Catalina Windows binaries) make them an optional binary asset instead.

Ross.

Catalina is now on GitHub

Comments