Unofficial Parallax Continuous Integration / Build Server
DavidZemon
Posts: 2,973
What is a "Continuous Integration" server?
Here's some snippets I found on the Internet:
Both quotes stolen from https://www.thoughtworks.com/continuous-integration
What is this thread for?
Many of you are already well aware that I have a public build server hosting numerous Propeller-related projects, including PropGCC and many others: https://ci.zemon.name?guest=1
Once in a while, there's an issue with it (a build failing, the server is down, whatever) or I have news to share regarding something about the server... now I have a place to deal with these topics.
I created this thread on this particular day because I have excellent news to share: NO MORE SLOW DOWNLOADS! I've just completed the migration of TeamCity's artifact storage from local storage to Amazon's AWS S3! Do you know what that means? It means you'll get files straight from Amazon's cloud instead of from a laptop in my basement. This is really important because I only have a 5 Mbps upload speed through my ISP and therefore you could only download at a maximum of 5 Mbps. But now? Oh, now it is quite a bit faster... I pay for 200 Mbps download speeds through my ISP and AWS saturates that no problem. It means PropGCC downloads in ~6 seconds instead of 4+ minutes.
For anyone who would like to see another project added to my server, comment below with a link to GitHub or some other public version control server and we can work together to get it added.
Here's some snippets I found on the Internet:
Continuous Integration (CI) is a development practice that requires developers to integrate code into a shared repository several times a day. Each check-in is then verified by an automated build, allowing teams to detect problems early.
By integrating regularly, you can detect errors quickly, and locate them more easily.
Continuous Deployment is closely related to Continuous Integration and refers to the release into production of software that passes the automated tests.
"Essentially, it is the practice of releasing every good build to users”, explains Jez Humble, author of Continuous Delivery.
By adopting both Continuous Integration and Continuous Deployment, you not only reduce risks and catch bugs quickly, but also move rapidly to working software.
Both quotes stolen from https://www.thoughtworks.com/continuous-integration
What is this thread for?
Many of you are already well aware that I have a public build server hosting numerous Propeller-related projects, including PropGCC and many others: https://ci.zemon.name?guest=1
Once in a while, there's an issue with it (a build failing, the server is down, whatever) or I have news to share regarding something about the server... now I have a place to deal with these topics.
I created this thread on this particular day because I have excellent news to share: NO MORE SLOW DOWNLOADS! I've just completed the migration of TeamCity's artifact storage from local storage to Amazon's AWS S3! Do you know what that means? It means you'll get files straight from Amazon's cloud instead of from a laptop in my basement. This is really important because I only have a 5 Mbps upload speed through my ISP and therefore you could only download at a maximum of 5 Mbps. But now? Oh, now it is quite a bit faster... I pay for 200 Mbps download speeds through my ISP and AWS saturates that no problem. It means PropGCC downloads in ~6 seconds instead of 4+ minutes.
For anyone who would like to see another project added to my server, comment below with a link to GitHub or some other public version control server and we can work together to get it added.
Comments
Not meaning to rain on the parade... but?
dgately
Due to the proprietary nature of the mac os, and the windows os, its advisable for users to install a virtual machine and install linux onto it.
I once was on windows, and found that i can shove linux into a software "machine", its pretty easy, they even make machines ready to go.
If I run linux and shove windows/mac into a machine.... simply put.. its not free, i must burn my key on it.
I can't get all the software to DEV (AND TEST IT) for windows or mac, for free.
https://www.virtualbox.org/wiki/Downloads
I mean, look at all the already built boxes you can download and plugin. All linux.
https://www.osboxes.org/virtualbox-images/
If the GIT has methods to compile windows, linux, osx, rpi, I think they should be run also, then those users will be able to just bypass the compile process and download the compile result (what ever that may be) targeted for their os.
From how the site looks, he says its on his server, so if he has a mac, then he would need to have it run all the same stuff and offer/upload the packages to his webserver.(if the git cannot cross)
It could probably be done, who wants to offer the mac? (lol, i have no idea what that involves)
To answer your question directly: not a lot. I do remember one time there was a project that failed to compile on my server which caused an automated email to be sent out. From there, I worked with the developer and quickly determined that he forgot to commit a new file to the repository. Easy mistake, we've all done it! But maybe, possibly, perhaps, there's a slim chance that if a CI server wasn't running and didn't catch that bug, you might have been the one to catch that mistake when you tried to compile the tool for yourself. So in that respect... maybe. Maybe you've been helped a tiny bit because the projects have stayed in good shape. Or... maybe just "better" shape... they'd be in good shape even without my server.... we have great developers in this community!
Nonetheless, I feel for you. I've wanted to do something about it ever since I first stood up the server. I've even considered hopping on Craigslist and buying an old Macbook solely for the sake of running a TeamCity agent on it... here's one for $100 near me... maybe I will now that they've gotten this cheap? Last time I looked, the cheapest I could find anything was closer to $200-$300.
However, if you or anyone else has a Macbook that is on most or all of the time, I'd LOVE to get it configured to run as a TeamCity agent. Nothing needs to change in your network's firewall and you don't need a custom domain name or anything; all communication is unidirectional from the TeamCity agent to the TeamCity server. You'd need to ensure that the machine has access to david.zemon.name:8111 and that Java 8 is installed. Following that, you'd download the build agent zip file from my server, extract it, and run the startup script. There's a touch more to it than that (like configuring the startup script to run at boot time), but not much. Full instructions here.
Once installed, I'd have to start creating new run configurations for each project to actually build the Mac binaries... no idea how to do that yet... but I'll happily learn!
yea, i looked at it.. don't do everything in there... lol
I would suggest ParallaxEsp. Output is ability to flash files to ParallaxWx using the built OTA file. The build folder contains files useful too, i would suggest running all scrips to initiate creation of files that require source. Encapsulate entire ParallaxESP folder once built and offer it for download as a compiled item, just don't call it the source. This allows any owner of a esp device sufficient size able to flash the esp using the /build/flash script, if I remember correctly. I need to rip out my breadboard and program a esp using serial again.
https://github.com/parallaxinc/Parallax-ESP
I don't know if these would make sense.. but... in order of preference.
Hardware -offer the entire git (after compile) as a package to download.
(only offer whats inside the git's folder once compile is done)
( make sure to search for scrips to run in subfolders like "build". Be careful they don't do bad stuff like install something on your build server)
https://github.com/parallaxinc/ParallaxEsp Most important = .ota file
https://github.com/parallaxinc/BadgeWX Most important = ??
https://github.com/parallaxinc/Flight-Controller Most important = ??
https://github.com/parallaxinc/BadgeHacker Most important = ??
Software
https://github.com/parallaxinc/BlocklyPropClient Most important = jar file.
https://github.com/parallaxinc/PropBASIC Most important = Installer packages
https://github.com/parallaxinc/PropellerIDE Most important = Installer packages
https://github.com/parallaxinc/BlocklyProp Most important =??
https://github.com/parallaxinc/Parallax-IDE Most important = Installer packages?
https://github.com/parallaxinc/propeller Most important =spider the websites and forum links it suggests to X??? deep? (what makes sense) and compress into package offer for daily archive (pick a time) download?
Well.. uhh... cause that's a lot of repositories? lol Maybe eventually... but let's take it one at a time please
Great idea!
...
Ugh... turns out, no one is providing binaries for that toolchain. So first I have to build the toolchain and then I can build Parallax-Esp. Hm... so the question is, do I want to add that toolchain to my build server too? Or do I want to just build the toolchain once and be done with it? Or do I create a Dockerfile that builds the toolchain and then use the Docker image to build Parallax-Esp? Choices...
Hah, I feel ya.
That's why I ended up using NixOS's build infrastructure to do my Mac builds. Which reminds me, I need to finish the build / integration / documentation.
(Now I'm going to get out of your thread other than to say that this is amazing, awesome work that you've been doing for us).
What about something you have listed... ?
ObexDownloader<--THIS.
(the latest head of ParallaxESP has failed for me on debian 64-bit and rpi i had to go back a commit)
https://github.com/parallaxinc/Flight-Controller
And this is why using the build system built into your IDE is a bad idea. Yes, some people hate the command line... I get it. Pushing a button in a GUI is easier. But I can't build the Flight Controller from my server because the build system doesn't have a command-line interface... the build system is SimpleIDE.
As for BadgeWX... same thing. The build system is a lot simpler... run a Spin compiler of your choice on the Spin file of your choice and *POOF* out pops a binary... except oh wait, you have to know where the Spin libraries are. Sooo... there is a build system, because there's some logic somewhere that determines where those libraries are. So changes will be required to BadgeWX to, or I put together some fragile Docker image with the libraries and compiler in the right place and hope nothing ever changes. Oh, and I have to write a million lines of code in TeamCity to individually compile each spin file.
https://github.com/parallaxinc/PropellerIDE
https://github.com/parallaxinc/BadgeHacker
BadgeHacker and PropellerIDE were written by @"Brett Weir" and I believe he has his own CI server builds configured in TravisCI. Hopefully he's still watching email from these forums and can provide a link, and guidance on what should be done (if anything) to maintain/modify those Travis builds if anyone else decides to start taking an active role in development of any those tools. Also, Brett, once that link is available, I'll happily include it at the top of this thread in my first post.
I looked through most of the others (though I notice you're constantly editing your list
https://github.com/parallaxinc/BlocklyProp
That's an interesting idea. I could certainly build the Node package and provide the installation archive as the build artifact. Not sure how much use it would get though since Parallax are already hosting this product themselves: http://blockly.parallax.com/blockly but it's easy enough to get it going, I'm not opposed to it.
ha
Yea.... I should take another look at that. Maybe tomorrow night? I've been staring at this forums for too long tonight.
Of what? From what I gathered from cpants homepage, it's a static code analysis tool, and largely aimed at Perl maybe?
(edit: actually, nope. It does dynamic analysis too)
It downloads, compiles, runs test suite for every perl package in CPAN. It publishes and harrasses^Wnotifies developers if their code doesn't compile or fails tests on multiple architectures. Latter versions of it even do documentation coverage etc...
The reason I mention it is because CPANTS was built *after* and *independently of* CPAN (which is the model we'd be looking at here).
None of this is hugely hard, it's just hand-cranking and we need to be sure that it's something that will be used before we start cranking the handle.
Test suites to assure confidence in tool testing should clearly be on the list.
Hey David, thanks for the shout-out! I was just reading my email and happened to see this, so I'll throw in my two cents. =P
In various Parallax projects, I used Travis for Linux/Mac builds and AppVeyor for Windows builds.
https://travis-ci.org/parallaxinc/PropellerIDE
https://ci.appveyor.com/project/bweir/propelleride
In general, the artifacts were published to GitHub releases, which is also available via API:
https://github.com/parallaxinc/PropellerIDE/releases
https://github.com/parallaxinc/PropBASIC/releases
https://github.com/parallaxinc/PropellerManager/releases
What gets published is managed by the relevant .travis.yml and appveyor.yml files in the project itself. Some docs on that:
https://docs.travis-ci.com/user/tutorial/
https://www.appveyor.com/docs/
Having build automation changes your entire workflow. Having builds available for all platforms at all times means you can spend more time doing more important things. Beyond build automation, there's also testing, static analysis, automated deployment, and all kinds of fun stuff you can do with a good CI service.
These days, I am a huge fan of GitLab, because of the powerful and flexible CI/CD pipelines you can bake into every project. The runner is written in Go and open-source, so you can build your own runner infrastructure and even execute CI jobs locally to test them (something I WISH I could do in Travis/AppVeyor).
In any case, automation is your friend, and CLI tools are fantastically easy to automate, whereas GUI tools are almost impossible. I always plan build systems to be driven from the command line for precisely that reason.
I think I have a spare iMac around the house, somewhere... Let me see if I can get something going.
dgately
AWESOME! Send me an email at ${first name}@${last name}.name whenever you're ready and we'll get it configured.
Sent a PM...
Woo hoo!!!
I only have a select few building for Mac at this time, but I'll get the others added soon. PropGCC I think is next on my list, as it will open the door to more, but it is more than I can do tonight.
In the meantime, take a look at OpenSpin, PropLoader, and Spin2Cpp!
@"Roy Eltham", you can now add a link to OpenSpin's readme for a Mac OS X build
edit: added it
I'll see about getting Windows builds added later. Maybe tomorrow.
thanks for doing this.
Enjoy!
Mike
Seems like you've got a lot of resources already corralled!
Me too. I've tried to be fairly vocal about taking part in that, and I'm fairly sure I was well heard, but I haven't heard anything from Parallax yet (other than what's been publicly posted on the forums). But I don't think anyone else has either... it's just taking a lot of time. Since they didn't get the boards out until right before Christmas, I'm sure they're now all on vacation (at least they should be!) for the holidays (at least New Year's day, for our non-Christian friends). My guess is that we might hear from Parallax somewhere between Jan 2 and 11.
Mac versions will be live soon hopefully. Maybe they'll be working by the time you read this, who knows
I've also updated the ObexDownloader and configured it to run weekly. We now have a WEEKLY BACKUP of every single Obex object
* Micro ATX case
* Micro ATX power supply (350W will be ample)
* Motherboard/CPU, preferably quad-core, but even a dual-core Intel would be an upgrade
* 8 GB of RAM (in whatever form the motherboard requires)
I already have an SSD and all peripherals.
For comparison's sake, this laptop is running an AMD A6-4400M, which is about on par with the Intel Core2 Duo P8400 @ 2.26GHz, according to cpubenchmark.net (https://www.cpubenchmark.net/cpu.php?cpu=AMD+A6-4400M+APU&id=29).
I do have some old regular ATX sized stuff, motherboard, ram, CPU, case, power supply, etc. Decent stuff, just a few gens old. Like an i7 quad core from like 1st/2nd gen era?
I also have a Mac mini from a few years back.
So, I'll PM Roy about arranging something.
Anyone have a small (or big) server in the cloud and want to run the TeamCity server? It wouldn't do any builds, just run the website. Not sure what the minimum hardware requirements are... but 1 CPU and 2 GB RAM would likely be okay, with 2 CPUs being ample. This is completely optional, but having the server-side running in a cloud with more than 5 Mbps upload (that's all I get from my house) would greatly improve perceived performance.