Shop OBEX P1 Docs P2 Docs Learn Events
Unofficial Parallax Continuous Integration / Build Server — Parallax Forums

Unofficial Parallax Continuous Integration / Build Server

DavidZemonDavidZemon Posts: 2,973
edited 2019-07-10 02:25 in General Discussion
What is a "Continuous Integration" server?

Here's some snippets I found on the Internet:
Continuous Integration (CI) is a development practice that requires developers to integrate code into a shared repository several times a day. Each check-in is then verified by an automated build, allowing teams to detect problems early.

By integrating regularly, you can detect errors quickly, and locate them more easily.
Continuous Deployment is closely related to Continuous Integration and refers to the release into production of software that passes the automated tests.

"Essentially, it is the practice of releasing every good build to users”, explains Jez Humble, author of Continuous Delivery.

By adopting both Continuous Integration and Continuous Deployment, you not only reduce risks and catch bugs quickly, but also move rapidly to working software.

Both quotes stolen from https://www.thoughtworks.com/continuous-integration

What is this thread for?

Many of you are already well aware that I have a public build server hosting numerous Propeller-related projects, including PropGCC and many others: https://ci.zemon.name?guest=1

Once in a while, there's an issue with it (a build failing, the server is down, whatever) or I have news to share regarding something about the server... now I have a place to deal with these topics.

I created this thread on this particular day because I have excellent news to share: NO MORE SLOW DOWNLOADS! I've just completed the migration of TeamCity's artifact storage from local storage to Amazon's AWS S3! Do you know what that means? It means you'll get files straight from Amazon's cloud instead of from a laptop in my basement. This is really important because I only have a 5 Mbps upload speed through my ISP and therefore you could only download at a maximum of 5 Mbps. But now? Oh, now it is quite a bit faster... I pay for 200 Mbps download speeds through my ISP and AWS saturates that no problem. It means PropGCC downloads in ~6 seconds instead of 4+ minutes.

For anyone who would like to see another project added to my server, comment below with a link to GitHub or some other public version control server and we can work together to get it added.
«1

Comments

  • Great, but as a die-hard macOS user, does this help me?

    Not meaning to rain on the parade... but?

    dgately
  • Clock LoopClock Loop Posts: 2,069
    edited 2018-12-04 02:01
    dgately wrote: »
    Great, but as a die-hard macOS user, does this help me?

    Not meaning to rain on the parade... but?

    dgately

    Due to the proprietary nature of the mac os, and the windows os, its advisable for users to install a virtual machine and install linux onto it.

    I once was on windows, and found that i can shove linux into a software "machine", its pretty easy, they even make machines ready to go.

    If I run linux and shove windows/mac into a machine.... simply put.. its not free, i must burn my key on it.
    I can't get all the software to DEV (AND TEST IT) for windows or mac, for free.

    https://www.virtualbox.org/wiki/Downloads

    I mean, look at all the already built boxes you can download and plugin. All linux.
    https://www.osboxes.org/virtualbox-images/

    If the GIT has methods to compile windows, linux, osx, rpi, I think they should be run also, then those users will be able to just bypass the compile process and download the compile result (what ever that may be) targeted for their os.

    From how the site looks, he says its on his server, so if he has a mac, then he would need to have it run all the same stuff and offer/upload the packages to his webserver.(if the git cannot cross)
    It could probably be done, who wants to offer the mac? (lol, i have no idea what that involves)
  • DavidZemonDavidZemon Posts: 2,973
    edited 2018-12-04 02:36
    dgately wrote: »
    Great, but as a die-hard macOS user, does this help me?

    Not meaning to rain on the parade... but?

    dgately

    To answer your question directly: not a lot. I do remember one time there was a project that failed to compile on my server which caused an automated email to be sent out. From there, I worked with the developer and quickly determined that he forgot to commit a new file to the repository. Easy mistake, we've all done it! But maybe, possibly, perhaps, there's a slim chance that if a CI server wasn't running and didn't catch that bug, you might have been the one to catch that mistake when you tried to compile the tool for yourself. So in that respect... maybe. Maybe you've been helped a tiny bit because the projects have stayed in good shape. Or... maybe just "better" shape... they'd be in good shape even without my server.... we have great developers in this community! :smile:

    Nonetheless, I feel for you. I've wanted to do something about it ever since I first stood up the server. I've even considered hopping on Craigslist and buying an old Macbook solely for the sake of running a TeamCity agent on it... here's one for $100 near me... maybe I will now that they've gotten this cheap? Last time I looked, the cheapest I could find anything was closer to $200-$300.

    However, if you or anyone else has a Macbook that is on most or all of the time, I'd LOVE to get it configured to run as a TeamCity agent. Nothing needs to change in your network's firewall and you don't need a custom domain name or anything; all communication is unidirectional from the TeamCity agent to the TeamCity server. You'd need to ensure that the machine has access to david.zemon.name:8111 and that Java 8 is installed. Following that, you'd download the build agent zip file from my server, extract it, and run the startup script. There's a touch more to it than that (like configuring the startup script to run at boot time), but not much. Full instructions here.

    Once installed, I'd have to start creating new run configurations for each project to actually build the Mac binaries... no idea how to do that yet... but I'll happily learn!
  • Clock LoopClock Loop Posts: 2,069
    edited 2018-12-04 05:03
    Why not do everything that parallax has in their git profile.
    yea, i looked at it.. don't do everything in there... lol

    I would suggest ParallaxEsp. Output is ability to flash files to ParallaxWx using the built OTA file. The build folder contains files useful too, i would suggest running all scrips to initiate creation of files that require source. Encapsulate entire ParallaxESP folder once built and offer it for download as a compiled item, just don't call it the source. This allows any owner of a esp device sufficient size able to flash the esp using the /build/flash script, if I remember correctly. I need to rip out my breadboard and program a esp using serial again.
    https://github.com/parallaxinc/Parallax-ESP


    I don't know if these would make sense.. but... in order of preference.

    Hardware -offer the entire git (after compile) as a package to download.
    (only offer whats inside the git's folder once compile is done)
    ( make sure to search for scrips to run in subfolders like "build". Be careful they don't do bad stuff like install something on your build server)

    https://github.com/parallaxinc/ParallaxEsp Most important = .ota file

    https://github.com/parallaxinc/BadgeWX Most important = ??

    https://github.com/parallaxinc/Flight-Controller Most important = ??

    https://github.com/parallaxinc/BadgeHacker Most important = ??

    Software

    https://github.com/parallaxinc/BlocklyPropClient Most important = jar file.

    https://github.com/parallaxinc/PropBASIC Most important = Installer packages

    https://github.com/parallaxinc/PropellerIDE Most important = Installer packages

    https://github.com/parallaxinc/BlocklyProp Most important =??

    https://github.com/parallaxinc/Parallax-IDE Most important = Installer packages?

    https://github.com/parallaxinc/propeller Most important =spider the websites and forum links it suggests to X??? deep? (what makes sense) and compress into package offer for daily archive (pick a time) download?

  • DavidZemonDavidZemon Posts: 2,973
    edited 2018-12-04 04:45
    Clock Loop wrote: »
    Why not do everything that parallax has in their git profile.

    Well.. uhh... cause that's a lot of repositories? lol Maybe eventually... but let's take it one at a time please
    Clock Loop wrote: »
    I would suggest ParallaxEsp.

    Great idea!

    ...

    Ugh... turns out, no one is providing binaries for that toolchain. So first I have to build the toolchain and then I can build Parallax-Esp. Hm... so the question is, do I want to add that toolchain to my build server too? Or do I want to just build the toolchain once and be done with it? Or do I create a Dockerfile that builds the toolchain and then use the Docker image to build Parallax-Esp? Choices...
  • DavidZemon wrote: »
    Nonetheless, I feel for you. I've wanted to do something about it ever since I first stood up the server. I've even considered hopping on Craigslist and buying an old Macbook solely for the sake of running a TeamCity agent on it...

    Hah, I feel ya.

    That's why I ended up using NixOS's build infrastructure to do my Mac builds. Which reminds me, I need to finish the build / integration / documentation.

    (Now I'm going to get out of your thread other than to say that this is amazing, awesome work that you've been doing for us).
  • Clock LoopClock Loop Posts: 2,069
    edited 2018-12-04 06:36
    Scratch all the above... for now...

    What about something you have listed... ?

    ObexDownloader<--THIS.


    (the latest head of ParallaxESP has failed for me on debian 64-bit and rpi i had to go back a commit)
  • https://github.com/parallaxinc/BadgeWX
    https://github.com/parallaxinc/Flight-Controller


    And this is why using the build system built into your IDE is a bad idea. Yes, some people hate the command line... I get it. Pushing a button in a GUI is easier. But I can't build the Flight Controller from my server because the build system doesn't have a command-line interface... the build system is SimpleIDE.

    As for BadgeWX... same thing. The build system is a lot simpler... run a Spin compiler of your choice on the Spin file of your choice and *POOF* out pops a binary... except oh wait, you have to know where the Spin libraries are. Sooo... there is a build system, because there's some logic somewhere that determines where those libraries are. So changes will be required to BadgeWX to, or I put together some fragile Docker image with the libraries and compiler in the right place and hope nothing ever changes. Oh, and I have to write a million lines of code in TeamCity to individually compile each spin file.


    https://github.com/parallaxinc/PropellerIDE
    https://github.com/parallaxinc/BadgeHacker


    BadgeHacker and PropellerIDE were written by @"Brett Weir" and I believe he has his own CI server builds configured in TravisCI. Hopefully he's still watching email from these forums and can provide a link, and guidance on what should be done (if anything) to maintain/modify those Travis builds if anyone else decides to start taking an active role in development of any those tools. Also, Brett, once that link is available, I'll happily include it at the top of this thread in my first post.

    I looked through most of the others (though I notice you're constantly editing your list :wink: ) and many of them also have Travis files. For all of the builds that are running on Travis, it'd be great if we could get access to the artifacts produced by them rather than have me duplicate that effort.

    https://github.com/parallaxinc/BlocklyProp

    That's an interesting idea. I could certainly build the Node package and provide the installation archive as the build artifact. Not sure how much use it would get though since Parallax are already hosting this product themselves: http://blockly.parallax.com/blockly but it's easy enough to get it going, I'm not opposed to it.
    Clock Loop wrote: »
    Scratch all the above... for now...

    What about something you have listed... ?

    ObexDownloader<--THIS.

    ha :) you like that, do you?
    Yea.... I should take another look at that. Maybe tomorrow night? I've been staring at this forums for too long tonight.
  • See CPANTS for a good example of an implementation of this.
  • __red__ wrote: »
    See CPANTS for a good example of an implementation of this.

    Of what? From what I gathered from cpants homepage, it's a static code analysis tool, and largely aimed at Perl maybe?
  • __red____red__ Posts: 470
    edited 2018-12-04 06:28
    Yup
    (edit: actually, nope. It does dynamic analysis too)


    It downloads, compiles, runs test suite for every perl package in CPAN. It publishes and harrasses^Wnotifies developers if their code doesn't compile or fails tests on multiple architectures. Latter versions of it even do documentation coverage etc...

    The reason I mention it is because CPANTS was built *after* and *independently of* CPAN (which is the model we'd be looking at here).

    None of this is hugely hard, it's just hand-cranking and we need to be sure that it's something that will be used before we start cranking the handle.

    Test suites to assure confidence in tool testing should clearly be on the list.
  • DavidZemon wrote: »
    https://github.com/parallaxinc/BadgeWX
    https://github.com/parallaxinc/Flight-Controller


    And this is why using the build system built into your IDE is a bad idea. Yes, some people hate the command line... I get it. Pushing a button in a GUI is easier. But I can't build the Flight Controller from my server because the build system doesn't have a command-line interface... the build system is SimpleIDE.

    As for BadgeWX... same thing. The build system is a lot simpler... run a Spin compiler of your choice on the Spin file of your choice and *POOF* out pops a binary... except oh wait, you have to know where the Spin libraries are. Sooo... there is a build system, because there's some logic somewhere that determines where those libraries are. So changes will be required to BadgeWX to, or I put together some fragile Docker image with the libraries and compiler in the right place and hope nothing ever changes. Oh, and I have to write a million lines of code in TeamCity to individually compile each spin file.


    https://github.com/parallaxinc/PropellerIDE
    https://github.com/parallaxinc/BadgeHacker


    BadgeHacker and PropellerIDE were written by @"Brett Weir" and I believe he has his own CI server builds configured in TravisCI. Hopefully he's still watching email from these forums and can provide a link, and guidance on what should be done (if anything) to maintain/modify those Travis builds if anyone else decides to start taking an active role in development of any those tools. Also, Brett, once that link is available, I'll happily include it at the top of this thread in my first post.

    I looked through most of the others (though I notice you're constantly editing your list :wink: ) and many of them also have Travis files. For all of the builds that are running on Travis, it'd be great if we could get access to the artifacts produced by them rather than have me duplicate that effort.

    https://github.com/parallaxinc/BlocklyProp

    That's an interesting idea. I could certainly build the Node package and provide the installation archive as the build artifact. Not sure how much use it would get though since Parallax are already hosting this product themselves: http://blockly.parallax.com/blockly but it's easy enough to get it going, I'm not opposed to it.
    Clock Loop wrote: »
    Scratch all the above... for now...

    What about something you have listed... ?

    ObexDownloader<--THIS.

    ha :) you like that, do you?
    Yea.... I should take another look at that. Maybe tomorrow night? I've been staring at this forums for too long tonight.

    Hey David, thanks for the shout-out! I was just reading my email and happened to see this, so I'll throw in my two cents. =P

    In various Parallax projects, I used Travis for Linux/Mac builds and AppVeyor for Windows builds.
    https://travis-ci.org/parallaxinc/PropellerIDE
    https://ci.appveyor.com/project/bweir/propelleride

    In general, the artifacts were published to GitHub releases, which is also available via API:
    https://github.com/parallaxinc/PropellerIDE/releases
    https://github.com/parallaxinc/PropBASIC/releases
    https://github.com/parallaxinc/PropellerManager/releases

    What gets published is managed by the relevant .travis.yml and appveyor.yml files in the project itself. Some docs on that:
    https://docs.travis-ci.com/user/tutorial/
    https://www.appveyor.com/docs/

    Having build automation changes your entire workflow. Having builds available for all platforms at all times means you can spend more time doing more important things. Beyond build automation, there's also testing, static analysis, automated deployment, and all kinds of fun stuff you can do with a good CI service.

    These days, I am a huge fan of GitLab, because of the powerful and flexible CI/CD pipelines you can bake into every project. The runner is written in Go and open-source, so you can build your own runner infrastructure and even execute CI jobs locally to test them (something I WISH I could do in Travis/AppVeyor).

    In any case, automation is your friend, and CLI tools are fantastically easy to automate, whereas GUI tools are almost impossible. I always plan build systems to be driven from the command line for precisely that reason.
  • DavidZemon wrote: »
    However, if you or anyone else has a Macbook that is on most or all of the time, I'd LOVE to get it configured to run as a TeamCity agent. Nothing needs to change in your network's firewall and you don't need a custom domain name or anything; all communication is unidirectional from the TeamCity agent to the TeamCity server. You'd need to ensure that the machine has access to david.zemon.name:8111 and that Java 8 is installed. Following that, you'd download the build agent zip file from my server, extract it, and run the startup script. There's a touch more to it than that (like configuring the startup script to run at boot time), but not much. Full instructions here.

    I think I have a spare iMac around the house, somewhere... Let me see if I can get something going.

    dgately
  • DavidZemonDavidZemon Posts: 2,973
    edited 2018-12-04 17:50
    dgately wrote: »
    DavidZemon wrote: »
    However, if you or anyone else has a Macbook that is on most or all of the time, I'd LOVE to get it configured to run as a TeamCity agent. Nothing needs to change in your network's firewall and you don't need a custom domain name or anything; all communication is unidirectional from the TeamCity agent to the TeamCity server. You'd need to ensure that the machine has access to david.zemon.name:8111 and that Java 8 is installed. Following that, you'd download the build agent zip file from my server, extract it, and run the startup script. There's a touch more to it than that (like configuring the startup script to run at boot time), but not much. Full instructions here.

    I think I have a spare iMac around the house, somewhere... Let me see if I can get something going.

    dgately

    AWESOME! Send me an email at ${first name}@${last name}.name whenever you're ready and we'll get it configured.
  • DavidZemon wrote: »
    AWESOME! Send me an email at ${first name}@${last name}.name whenever you're ready and we'll get it configured.

    Sent a PM...

  • Thanks to @dgately, WE HAVE MAC OS X BUILDS RUNNING!

    Woo hoo!!!

    I only have a select few building for Mac at this time, but I'll get the others added soon. PropGCC I think is next on my list, as it will open the door to more, but it is more than I can do tonight.

    In the meantime, take a look at OpenSpin, PropLoader, and Spin2Cpp!

    @"Roy Eltham", you can now add a link to OpenSpin's readme for a Mac OS X build :)
  • Roy ElthamRoy Eltham Posts: 2,996
    edited 2018-12-05 04:02
    @DavidZemon Awesome!

    edit: added it :)
  • Linux and Mac OS X builds for Spin2GUI have now been added to the CI server: http://david.zemon.name:8111/project.html?projectId=Spin2GUI
    I'll see about getting Windows builds added later. Maybe tomorrow.
  • @DavidZemon,

    thanks for doing this.

    Enjoy!

    Mike
  • No problem. For anyone that tried clicking the link earlier, it's fixed now. I had permissions set incorrectly
  • DavidZ and Brett, I hope your both on the list for the P2 Tools Meeting happening in the near future.
    Seems like you've got a lot of resources already corralled!
  • koehler wrote: »
    DavidZ and Brett, I hope your both on the list for the P2 Tools Meeting happening in the near future.
    Seems like you've got a lot of resources already corralled!

    Me too. I've tried to be fairly vocal about taking part in that, and I'm fairly sure I was well heard, but I haven't heard anything from Parallax yet (other than what's been publicly posted on the forums). But I don't think anyone else has either... it's just taking a lot of time. Since they didn't get the boards out until right before Christmas, I'm sure they're now all on vacation (at least they should be!) for the holidays (at least New Year's day, for our non-Christian friends). My guess is that we might hear from Parallax somewhere between Jan 2 and 11.
  • DavidZemon wrote: »
    koehler wrote: »
    DavidZ and Brett, I hope your both on the list for the P2 Tools Meeting happening in the near future.
    Seems like you've got a lot of resources already corralled!

    Me too. I've tried to be fairly vocal about taking part in that, and I'm fairly sure I was well heard, but I haven't heard anything from Parallax yet (other than what's been publicly posted on the forums). But I don't think anyone else has either... it's just taking a lot of time. Since they didn't get the boards out until right before Christmas, I'm sure they're now all on vacation (at least they should be!) for the holidays (at least New Year's day, for our non-Christian friends). My guess is that we might hear from Parallax somewhere between Jan 2 and 11.
    The last Parallax said about the tools meeting is that it would take place after people had the P2-Eval boards in hand. Now that that has happened, I guess the meeting can happen at any time but obviously we need to get past the holidays first.

  • Windows builds of Spin2Gui now available as well: http://david.zemon.name:8111/project.html?projectId=Spin2GUI&guest=1
  • Linux versions of p2gcc are live on TeamCity: http://david.zemon.name:8111/project.html?projectId=P2gcc&tab=projectOverview

    Mac versions will be live soon hopefully. Maybe they'll be working by the time you read this, who knows :smile:

    I've also updated the ObexDownloader and configured it to run weekly. We now have a WEEKLY BACKUP of every single Obex object :smiley:http://david.zemon.name:8111/project.html?projectId=ObexDownloader&tab=projectOverview
  • Curious if anyone has any computer hardware they'd want to donate to the cause? I'm currently using an old, cheap, horrendously slow laptop to run TeamCity, and I'd like to switch to a compact (Micro ATX) desktop. I'm looking for the following PC parts:

    * Micro ATX case
    * Micro ATX power supply (350W will be ample)
    * Motherboard/CPU, preferably quad-core, but even a dual-core Intel would be an upgrade
    * 8 GB of RAM (in whatever form the motherboard requires)

    I already have an SSD and all peripherals.

    For comparison's sake, this laptop is running an AMD A6-4400M, which is about on par with the Intel Core2 Duo P8400 @ 2.26GHz, according to cpubenchmark.net (https://www.cpubenchmark.net/cpu.php?cpu=AMD+A6-4400M+APU&id=29).
  • I don't have any Micro ATX stuff.

    I do have some old regular ATX sized stuff, motherboard, ram, CPU, case, power supply, etc. Decent stuff, just a few gens old. Like an i7 quad core from like 1st/2nd gen era?
    I also have a Mac mini from a few years back.
  • Even a first gen 920 is something like a 4x improvement according to cpubenchmark.net. I'm in no hurry... Let's give it a couple days to see if anyone has microatx. If no one else speaks up, that sounds great.
  • Probably cheaper to visit a local pawn shop, cheap and no shipping costs. Sometimes really great deals, other not so good. Pawn Stars and the internet/ebay/craigs list got them a bit more in tune with what they can charge. Or computer/electronic junkyard.
  • I've check on craigslist... lots of cheap Dell/HP machines out there, but I'd strongly prefer a standard (micro) ATX + PSU so I can do piecemeal upgrades in the future. I found this guy and got him down to $220 without HDD or GPU, which seems an okay deal, but I'll take free + shipping over $220 any day.

    So, I'll PM Roy about arranging something.

    Anyone have a small (or big) server in the cloud and want to run the TeamCity server? It wouldn't do any builds, just run the website. Not sure what the minimum hardware requirements are... but 1 CPU and 2 GB RAM would likely be okay, with 2 CPUs being ample. This is completely optional, but having the server-side running in a cloud with more than 5 Mbps upload (that's all I get from my house) would greatly improve perceived performance.
Sign In or Register to comment.