Unofficial Parallax Continuous Integration / Build Server

What is a "Continuous Integration" server?

Here's some snippets I found on the Internet:
Continuous Integration (CI) is a development practice that requires developers to integrate code into a shared repository several times a day. Each check-in is then verified by an automated build, allowing teams to detect problems early.

By integrating regularly, you can detect errors quickly, and locate them more easily.
Continuous Deployment is closely related to Continuous Integration and refers to the release into production of software that passes the automated tests.

"Essentially, it is the practice of releasing every good build to users”, explains Jez Humble, author of Continuous Delivery.

By adopting both Continuous Integration and Continuous Deployment, you not only reduce risks and catch bugs quickly, but also move rapidly to working software.

Both quotes stolen from https://www.thoughtworks.com/continuous-integration

What is this thread for?

Many of you are already well aware that I have a public build server hosting numerous Propeller-related projects, including PropGCC and many others: http://david.zemon.name:8111?guest=1

Once in a while, there's an issue with it (a build failing, the server is down, whatever) or I have news to share regarding something about the server... now I have a place to deal with these topics.

I created this thread on this particular day because I have excellent news to share: NO MORE SLOW DOWNLOADS! I've just completed the migration of TeamCity's artifact storage from local storage to Amazon's AWS S3! Do you know what that means? It means you'll get files straight from Amazon's cloud instead of from a laptop in my basement. This is really important because I only have a 5 Mbps upload speed through my ISP and therefore you could only download at a maximum of 5 Mbps. But now? Oh, now it is quite a bit faster... I pay for 200 Mbps download speeds through my ISP and AWS saturates that no problem. It means PropGCC downloads in ~6 seconds instead of 4+ minutes.

For anyone who would like to see another project added to my server, comment below with a link to GitHub or some other public version control server and we can work together to get it added.
David
PropWare: C++ HAL (Hardware Abstraction Layer) for PropGCC; Robust build system using CMake; Integrated Simple Library, libpropeller, and libPropelleruino (Arduino port); Instructions for Eclipse and JetBrain's CLion; Example projects; Doxygen documentation
CI Server: http://david.zemon.name:8111/?guest=1

Comments

  • 17 Comments sorted by Date Added Votes
  • Great, but as a die-hard macOS user, does this help me?

    Not meaning to rain on the parade... but?

    dgately
    Livermore, CA (50 miles SE of San Francisco)
  • Clock LoopClock Loop Posts: 1,575
    edited December 4 Vote Up0Vote Down
    dgately wrote: »
    Great, but as a die-hard macOS user, does this help me?

    Not meaning to rain on the parade... but?

    dgately

    Due to the proprietary nature of the mac os, and the windows os, its advisable for users to install a virtual machine and install linux onto it.

    I once was on windows, and found that i can shove linux into a software "machine", its pretty easy, they even make machines ready to go.

    If I run linux and shove windows/mac into a machine.... simply put.. its not free, i must burn my key on it.
    I can't get all the software to DEV (AND TEST IT) for windows or mac, for free.

    https://www.virtualbox.org/wiki/Downloads

    I mean, look at all the already built boxes you can download and plugin. All linux.
    https://www.osboxes.org/virtualbox-images/

    If the GIT has methods to compile windows, linux, osx, rpi, I think they should be run also, then those users will be able to just bypass the compile process and download the compile result (what ever that may be) targeted for their os.

    From how the site looks, he says its on his server, so if he has a mac, then he would need to have it run all the same stuff and offer/upload the packages to his webserver.(if the git cannot cross)
    It could probably be done, who wants to offer the mac? (lol, i have no idea what that involves)
    1. Silicon gel filled square. <---2.Sonics(ultra even). 3.Lazers. 4. ?
    https://hackaday.io/project/162734-a-trillion-year-clock
    54 propeller chips were connected to One, to rule them all....
    https://forums.parallax.com/discussion/127983/55-parallax-propeller-s-parallells-processing-of-permanent-perturbations/p1
  • DavidZemonDavidZemon Posts: 2,559
    edited December 4 Vote Up0Vote Down
    dgately wrote: »
    Great, but as a die-hard macOS user, does this help me?

    Not meaning to rain on the parade... but?

    dgately

    To answer your question directly: not a lot. I do remember one time there was a project that failed to compile on my server which caused an automated email to be sent out. From there, I worked with the developer and quickly determined that he forgot to commit a new file to the repository. Easy mistake, we've all done it! But maybe, possibly, perhaps, there's a slim chance that if a CI server wasn't running and didn't catch that bug, you might have been the one to catch that mistake when you tried to compile the tool for yourself. So in that respect... maybe. Maybe you've been helped a tiny bit because the projects have stayed in good shape. Or... maybe just "better" shape... they'd be in good shape even without my server.... we have great developers in this community! :smile:

    Nonetheless, I feel for you. I've wanted to do something about it ever since I first stood up the server. I've even considered hopping on Craigslist and buying an old Macbook solely for the sake of running a TeamCity agent on it... here's one for $100 near me... maybe I will now that they've gotten this cheap? Last time I looked, the cheapest I could find anything was closer to $200-$300.

    However, if you or anyone else has a Macbook that is on most or all of the time, I'd LOVE to get it configured to run as a TeamCity agent. Nothing needs to change in your network's firewall and you don't need a custom domain name or anything; all communication is unidirectional from the TeamCity agent to the TeamCity server. You'd need to ensure that the machine has access to david.zemon.name:8111 and that Java 8 is installed. Following that, you'd download the build agent zip file from my server, extract it, and run the startup script. There's a touch more to it than that (like configuring the startup script to run at boot time), but not much. Full instructions here.

    Once installed, I'd have to start creating new run configurations for each project to actually build the Mac binaries... no idea how to do that yet... but I'll happily learn!
    David
    PropWare: C++ HAL (Hardware Abstraction Layer) for PropGCC; Robust build system using CMake; Integrated Simple Library, libpropeller, and libPropelleruino (Arduino port); Instructions for Eclipse and JetBrain's CLion; Example projects; Doxygen documentation
    CI Server: http://david.zemon.name:8111/?guest=1
  • Clock LoopClock Loop Posts: 1,575
    edited December 4 Vote Up0Vote Down
    Why not do everything that parallax has in their git profile.
    yea, i looked at it.. don't do everything in there... lol

    I would suggest ParallaxEsp. Output is ability to flash files to ParallaxWx using the built OTA file. The build folder contains files useful too, i would suggest running all scrips to initiate creation of files that require source. Encapsulate entire ParallaxESP folder once built and offer it for download as a compiled item, just don't call it the source. This allows any owner of a esp device sufficient size able to flash the esp using the /build/flash script, if I remember correctly. I need to rip out my breadboard and program a esp using serial again.
    https://github.com/parallaxinc/Parallax-ESP


    I don't know if these would make sense.. but... in order of preference.

    Hardware -offer the entire git (after compile) as a package to download.
    (only offer whats inside the git's folder once compile is done)
    ( make sure to search for scrips to run in subfolders like "build". Be careful they don't do bad stuff like install something on your build server)

    https://github.com/parallaxinc/ParallaxEsp Most important = .ota file

    https://github.com/parallaxinc/BadgeWX Most important = ??

    https://github.com/parallaxinc/Flight-Controller Most important = ??

    https://github.com/parallaxinc/BadgeHacker Most important = ??

    Software

    https://github.com/parallaxinc/BlocklyPropClient Most important = jar file.

    https://github.com/parallaxinc/PropBASIC Most important = Installer packages

    https://github.com/parallaxinc/PropellerIDE Most important = Installer packages

    https://github.com/parallaxinc/BlocklyProp Most important =??

    https://github.com/parallaxinc/Parallax-IDE Most important = Installer packages?

    https://github.com/parallaxinc/propeller Most important =spider the websites and forum links it suggests to X??? deep? (what makes sense) and compress into package offer for daily archive (pick a time) download?

    1. Silicon gel filled square. <---2.Sonics(ultra even). 3.Lazers. 4. ?
    https://hackaday.io/project/162734-a-trillion-year-clock
    54 propeller chips were connected to One, to rule them all....
    https://forums.parallax.com/discussion/127983/55-parallax-propeller-s-parallells-processing-of-permanent-perturbations/p1
  • DavidZemonDavidZemon Posts: 2,559
    edited December 4 Vote Up0Vote Down
    Clock Loop wrote: »
    Why not do everything that parallax has in their git profile.

    Well.. uhh... cause that's a lot of repositories? lol Maybe eventually... but let's take it one at a time please
    Clock Loop wrote: »
    I would suggest ParallaxEsp.

    Great idea!

    ...

    Ugh... turns out, no one is providing binaries for that toolchain. So first I have to build the toolchain and then I can build Parallax-Esp. Hm... so the question is, do I want to add that toolchain to my build server too? Or do I want to just build the toolchain once and be done with it? Or do I create a Dockerfile that builds the toolchain and then use the Docker image to build Parallax-Esp? Choices...
    David
    PropWare: C++ HAL (Hardware Abstraction Layer) for PropGCC; Robust build system using CMake; Integrated Simple Library, libpropeller, and libPropelleruino (Arduino port); Instructions for Eclipse and JetBrain's CLion; Example projects; Doxygen documentation
    CI Server: http://david.zemon.name:8111/?guest=1
  • DavidZemon wrote: »
    Nonetheless, I feel for you. I've wanted to do something about it ever since I first stood up the server. I've even considered hopping on Craigslist and buying an old Macbook solely for the sake of running a TeamCity agent on it...

    Hah, I feel ya.

    That's why I ended up using NixOS's build infrastructure to do my Mac builds. Which reminds me, I need to finish the build / integration / documentation.

    (Now I'm going to get out of your thread other than to say that this is amazing, awesome work that you've been doing for us).
  • Clock LoopClock Loop Posts: 1,575
    edited December 4 Vote Up0Vote Down
    Scratch all the above... for now...

    What about something you have listed... ?

    ObexDownloader<--THIS.


    (the latest head of ParallaxESP has failed for me on debian 64-bit and rpi i had to go back a commit)
    1. Silicon gel filled square. <---2.Sonics(ultra even). 3.Lazers. 4. ?
    https://hackaday.io/project/162734-a-trillion-year-clock
    54 propeller chips were connected to One, to rule them all....
    https://forums.parallax.com/discussion/127983/55-parallax-propeller-s-parallells-processing-of-permanent-perturbations/p1
  • https://github.com/parallaxinc/BadgeWX
    https://github.com/parallaxinc/Flight-Controller


    And this is why using the build system built into your IDE is a bad idea. Yes, some people hate the command line... I get it. Pushing a button in a GUI is easier. But I can't build the Flight Controller from my server because the build system doesn't have a command-line interface... the build system is SimpleIDE.

    As for BadgeWX... same thing. The build system is a lot simpler... run a Spin compiler of your choice on the Spin file of your choice and *POOF* out pops a binary... except oh wait, you have to know where the Spin libraries are. Sooo... there is a build system, because there's some logic somewhere that determines where those libraries are. So changes will be required to BadgeWX to, or I put together some fragile Docker image with the libraries and compiler in the right place and hope nothing ever changes. Oh, and I have to write a million lines of code in TeamCity to individually compile each spin file.


    https://github.com/parallaxinc/PropellerIDE
    https://github.com/parallaxinc/BadgeHacker


    BadgeHacker and PropellerIDE were written by @Brett Weir and I believe he has his own CI server builds configured in TravisCI. Hopefully he's still watching email from these forums and can provide a link, and guidance on what should be done (if anything) to maintain/modify those Travis builds if anyone else decides to start taking an active role in development of any those tools. Also, Brett, once that link is available, I'll happily include it at the top of this thread in my first post.

    I looked through most of the others (though I notice you're constantly editing your list :wink: ) and many of them also have Travis files. For all of the builds that are running on Travis, it'd be great if we could get access to the artifacts produced by them rather than have me duplicate that effort.

    https://github.com/parallaxinc/BlocklyProp

    That's an interesting idea. I could certainly build the Node package and provide the installation archive as the build artifact. Not sure how much use it would get though since Parallax are already hosting this product themselves: http://blockly.parallax.com/blockly but it's easy enough to get it going, I'm not opposed to it.
    Clock Loop wrote: »
    Scratch all the above... for now...

    What about something you have listed... ?

    ObexDownloader<--THIS.

    ha :) you like that, do you?
    Yea.... I should take another look at that. Maybe tomorrow night? I've been staring at this forums for too long tonight.
    David
    PropWare: C++ HAL (Hardware Abstraction Layer) for PropGCC; Robust build system using CMake; Integrated Simple Library, libpropeller, and libPropelleruino (Arduino port); Instructions for Eclipse and JetBrain's CLion; Example projects; Doxygen documentation
    CI Server: http://david.zemon.name:8111/?guest=1
  • See CPANTS for a good example of an implementation of this.
  • __red__ wrote: »
    See CPANTS for a good example of an implementation of this.

    Of what? From what I gathered from cpants homepage, it's a static code analysis tool, and largely aimed at Perl maybe?
    David
    PropWare: C++ HAL (Hardware Abstraction Layer) for PropGCC; Robust build system using CMake; Integrated Simple Library, libpropeller, and libPropelleruino (Arduino port); Instructions for Eclipse and JetBrain's CLion; Example projects; Doxygen documentation
    CI Server: http://david.zemon.name:8111/?guest=1
  • __red____red__ Posts: 439
    edited December 4 Vote Up0Vote Down
    Yup
    (edit: actually, nope. It does dynamic analysis too)


    It downloads, compiles, runs test suite for every perl package in CPAN. It publishes and harrasses^Wnotifies developers if their code doesn't compile or fails tests on multiple architectures. Latter versions of it even do documentation coverage etc...

    The reason I mention it is because CPANTS was built *after* and *independently of* CPAN (which is the model we'd be looking at here).

    None of this is hugely hard, it's just hand-cranking and we need to be sure that it's something that will be used before we start cranking the handle.

    Test suites to assure confidence in tool testing should clearly be on the list.
  • DavidZemon wrote: »
    https://github.com/parallaxinc/BadgeWX
    https://github.com/parallaxinc/Flight-Controller


    And this is why using the build system built into your IDE is a bad idea. Yes, some people hate the command line... I get it. Pushing a button in a GUI is easier. But I can't build the Flight Controller from my server because the build system doesn't have a command-line interface... the build system is SimpleIDE.

    As for BadgeWX... same thing. The build system is a lot simpler... run a Spin compiler of your choice on the Spin file of your choice and *POOF* out pops a binary... except oh wait, you have to know where the Spin libraries are. Sooo... there is a build system, because there's some logic somewhere that determines where those libraries are. So changes will be required to BadgeWX to, or I put together some fragile Docker image with the libraries and compiler in the right place and hope nothing ever changes. Oh, and I have to write a million lines of code in TeamCity to individually compile each spin file.


    https://github.com/parallaxinc/PropellerIDE
    https://github.com/parallaxinc/BadgeHacker


    BadgeHacker and PropellerIDE were written by @Brett Weir and I believe he has his own CI server builds configured in TravisCI. Hopefully he's still watching email from these forums and can provide a link, and guidance on what should be done (if anything) to maintain/modify those Travis builds if anyone else decides to start taking an active role in development of any those tools. Also, Brett, once that link is available, I'll happily include it at the top of this thread in my first post.

    I looked through most of the others (though I notice you're constantly editing your list :wink: ) and many of them also have Travis files. For all of the builds that are running on Travis, it'd be great if we could get access to the artifacts produced by them rather than have me duplicate that effort.

    https://github.com/parallaxinc/BlocklyProp

    That's an interesting idea. I could certainly build the Node package and provide the installation archive as the build artifact. Not sure how much use it would get though since Parallax are already hosting this product themselves: http://blockly.parallax.com/blockly but it's easy enough to get it going, I'm not opposed to it.
    Clock Loop wrote: »
    Scratch all the above... for now...

    What about something you have listed... ?

    ObexDownloader<--THIS.

    ha :) you like that, do you?
    Yea.... I should take another look at that. Maybe tomorrow night? I've been staring at this forums for too long tonight.

    Hey David, thanks for the shout-out! I was just reading my email and happened to see this, so I'll throw in my two cents. =P

    In various Parallax projects, I used Travis for Linux/Mac builds and AppVeyor for Windows builds.
    https://travis-ci.org/parallaxinc/PropellerIDE
    https://ci.appveyor.com/project/bweir/propelleride

    In general, the artifacts were published to GitHub releases, which is also available via API:
    https://github.com/parallaxinc/PropellerIDE/releases
    https://github.com/parallaxinc/PropBASIC/releases
    https://github.com/parallaxinc/PropellerManager/releases

    What gets published is managed by the relevant .travis.yml and appveyor.yml files in the project itself. Some docs on that:
    https://docs.travis-ci.com/user/tutorial/
    https://www.appveyor.com/docs/

    Having build automation changes your entire workflow. Having builds available for all platforms at all times means you can spend more time doing more important things. Beyond build automation, there's also testing, static analysis, automated deployment, and all kinds of fun stuff you can do with a good CI service.

    These days, I am a huge fan of GitLab, because of the powerful and flexible CI/CD pipelines you can bake into every project. The runner is written in Go and open-source, so you can build your own runner infrastructure and even execute CI jobs locally to test them (something I WISH I could do in Travis/AppVeyor).

    In any case, automation is your friend, and CLI tools are fantastically easy to automate, whereas GUI tools are almost impossible. I always plan build systems to be driven from the command line for precisely that reason.
    LameStation, the game console parents want their kids to play!
    PropellerIDE, an integrated, cross-platform IDE for the Propeller
    PropellerManager, an event-driven C++ API for managing Propeller devices
  • DavidZemon wrote: »
    However, if you or anyone else has a Macbook that is on most or all of the time, I'd LOVE to get it configured to run as a TeamCity agent. Nothing needs to change in your network's firewall and you don't need a custom domain name or anything; all communication is unidirectional from the TeamCity agent to the TeamCity server. You'd need to ensure that the machine has access to david.zemon.name:8111 and that Java 8 is installed. Following that, you'd download the build agent zip file from my server, extract it, and run the startup script. There's a touch more to it than that (like configuring the startup script to run at boot time), but not much. Full instructions here.

    I think I have a spare iMac around the house, somewhere... Let me see if I can get something going.

    dgately
    Livermore, CA (50 miles SE of San Francisco)
  • DavidZemonDavidZemon Posts: 2,559
    edited December 4 Vote Up0Vote Down
    dgately wrote: »
    DavidZemon wrote: »
    However, if you or anyone else has a Macbook that is on most or all of the time, I'd LOVE to get it configured to run as a TeamCity agent. Nothing needs to change in your network's firewall and you don't need a custom domain name or anything; all communication is unidirectional from the TeamCity agent to the TeamCity server. You'd need to ensure that the machine has access to david.zemon.name:8111 and that Java 8 is installed. Following that, you'd download the build agent zip file from my server, extract it, and run the startup script. There's a touch more to it than that (like configuring the startup script to run at boot time), but not much. Full instructions here.

    I think I have a spare iMac around the house, somewhere... Let me see if I can get something going.

    dgately

    AWESOME! Send me an email at ${first name}@${last name}.name whenever you're ready and we'll get it configured.
    David
    PropWare: C++ HAL (Hardware Abstraction Layer) for PropGCC; Robust build system using CMake; Integrated Simple Library, libpropeller, and libPropelleruino (Arduino port); Instructions for Eclipse and JetBrain's CLion; Example projects; Doxygen documentation
    CI Server: http://david.zemon.name:8111/?guest=1
  • DavidZemon wrote: »
    AWESOME! Send me an email at ${first name}@${last name}.name whenever you're ready and we'll get it configured.

    Sent a PM...

    Livermore, CA (50 miles SE of San Francisco)
  • Thanks to @dgately, WE HAVE MAC OS X BUILDS RUNNING!

    Woo hoo!!!

    I only have a select few building for Mac at this time, but I'll get the others added soon. PropGCC I think is next on my list, as it will open the door to more, but it is more than I can do tonight.

    In the meantime, take a look at OpenSpin, PropLoader, and Spin2Cpp!

    @Roy Eltham, you can now add a link to OpenSpin's readme for a Mac OS X build :)
    David
    PropWare: C++ HAL (Hardware Abstraction Layer) for PropGCC; Robust build system using CMake; Integrated Simple Library, libpropeller, and libPropelleruino (Arduino port); Instructions for Eclipse and JetBrain's CLion; Example projects; Doxygen documentation
    CI Server: http://david.zemon.name:8111/?guest=1
  • Roy ElthamRoy Eltham Posts: 2,532
    edited December 5 Vote Up0Vote Down
    @DavidZemon Awesome!

    edit: added it :)
Sign In or Register to comment.