Shop OBEX P1 Docs P2 Docs Learn Events
Building PropGCC with Docker — Parallax Forums

Building PropGCC with Docker

DavidZemonDavidZemon Posts: 2,973
edited 2019-08-24 13:51 in Propeller 1
I really like this Docker thing. The crowds were right - it's a cool tool when used in the right places.

I've created a new Docker image (davidzemon/propgccbuilder - https://hub.docker.com/r/davidzemon/propgccbuilder/) which is capable of building PropGCC from source. I've so far tested the following:

Using David Betz's repository: https://github.com/dbetz/propeller-gcc
- master branch only
GCC 4
- Linux x86-64
- Windows 32-bit
- ARMv6 (Raspberry Pi)
GCC 6
- Linux x86-64
- Windows 32-bit
- ARMv6 (Raspberry Pi)

I have not tested the old version of PropGCC from Parallax's repository (https://github.com/parallaxinc/propgcc) and I do not know how to use Docker to build for Mac (or if that's even possible).

Okay, so you want to try this for yourself?


System Requirements
Docker
Git

Setup (Written for Linux, tweak for Windows where necessary)

(For Windows, check out @iseries reply to this thread: https://forums.parallax.com/discussion/comment/1455187/#Comment_1455187)

Start by cloning PropGCC and entering the Docker container
> mkdir propgcc-parent
> cd propgcc-parent
> git clone --recursive https://github.com/dbetz/propeller-gcc.git
> docker pull davidzemon/propgccbuilder
> docker run -it -u $(id -u):$(id -g) -v `pwd`:`pwd` -w `pwd` davidzemon/propgccbuilder

Note that the "-u $(id -u):$(id -g)" is probably not necessary for Windows users. Excluding that argument on Linux will yield a bunch of files owned by root and that'd just be a pain wouldn't it?

At this point, you are now inside the Docker container and should be able to follow the instructions in the README. The important part about doing this in a Docker container is that all system requirements have already been installed, namely: a version of GCC that doesn't throw any errors when building GCC 4.x, MinGW, ARMv6 toolchain, texinfo.

Building PropGCC for Linux
The following assumes you are in the Docker container as configured above
> cd propeller-gcc/gcc
> ./contrib/download_prerequisites
> cd ..
> make GCCDIR=gcc4 -j4
> make GCCDIR=gcc4 INSTALL=`pwd`/../propgcc4-linux-bin install
> exit
("-j4" can be changed for your rig, but serves as a good default)
At this point, you'll be dropped to your host command prompt and PropGCC with GCCv4 for 64-bit Linux will have been installed to propgcc-parent/propgcc4-linux-bin. You can then move that wherever necessary on your host box.

Building PropGCC with GCCv6 is almost identical of course:
> cd propeller-gcc/gcc
> ./contrib/download_prerequisites
> cd ..
> make GCCDIR=gcc -j4
> make GCCDIR=gcc INSTALL=`pwd`/../propgcc-linux-bin install
> exit

Cross-Compiling PropGCC
Note for cross-compiling PropGCC, you MUST have already compiled (or downloaded) PropGCC for Linux (even if your host computer is Windows, the Docker image is always Linux and therefore you must have compiled PropGCC for Linux). The Linux version of PropGCC must also be based off of the same version of GCC as you are wanting to cross-compile. If you now want to build PropGCC with GCCv4 for Windows, then you must first build/download PropGCC with GCCv4 for Linux. If you want to build PropGCC with GCCv6 for Raspberry Pi, then you must first build/download PropGCC with GCCv6 for Linux.

So, let's assume you want to build PropGCC with GCCv6 for Windows and you've already built PropGCC with GCCv6 for Linux, as described above. We're starting at your host command prompt in the propgcc-parent directory, not from within the Docker container:
> docker run -it -u $(id -u):$(id -g) -v `pwd`:`pwd` -v `pwd`/propgcc-linux-bin:/opt/parallax:ro -w `pwd`  davidzemon/propgccbuilder
> cd propeller-gcc
> make GCCDIR=gcc CROSS=win32 -j4
> make GCCDIR=gcc CROSS=win32 INSTALL=`pwd`/../propgcc-win32-bin install
> exit

Voila! In propgcc-parent/propgcc-win32-bin, you now have Windows binaries for PropGCC built with the latest GCC 6 (or whatever version of GCC it is at the time of you reading this).

Some notes on cross-compiling:
The key here is the extra argument to Docker: "-v `pwd`/propgcc-linux-bin:/opt/parallax:ro". This has four parts:
"-v" is short for "--volume", and it says that a directory on your host computer should be available in the Docker container
The value is then split into three parts with the colons:
"`pwd`/propgcc-linux-bin" means that the folder on our host computer which is going to be available in Docker is the PropGCC binaries. This is the ONLY configurable part. You can change this part of the argument to whatever you need, based on wherever you have installed/downloaded/compiled PropGCC.
"/opt/parallax" is the target directory in the Docker container. One of the steps to creating the Docker image was adding "/opt/parallax/bin" to $PATH, which is why things "just work" with this arg.
"ro" means read only. Not entirely necessary, but a good safety measure that prevents accidental mucking around with your working PropGCC binaries.





I do not expect this to be used by the masses. The masses should just download whichever binaries they need either from Parallax directly or from my build server (https://ci.zemon.name?guest=1). However, if you want to play around with compiling PropGCC, this should make it much more reliable than trying to figure out compatible versions of GCC, since newer versions of GCC (such as what ships in Ubuntu 17.10) throw errors when trying to build PropGCC.



That's all folks! Let me know if you have questions.
«1

Comments

  • DavidZemonDavidZemon Posts: 2,973
    edited 2018-04-25 19:11
    A final note: yes the builds on my build server are currently failing, despite having switched to this supposedly magnificent new way of doing things. I believe I've encountered a bug in TeamCity and, as soon as JetBrains gets back to me (https://youtrack.jetbrains.com/issue/TW-54774), we should be in good working order again.
  • Thanks for putting this together! I'll have to try it. I've never used Docker.
  • You say one of the prerequisite is that Docker be installed. I went to look for a Docker installer and found one for a community edition (presumably free) and an enterprise edition. Should I install the community edition? Or is there some other open source version of Docker that I should be installing?
  • David Betz wrote: »
    You say one of the prerequisite is that Docker be installed. I went to look for a Docker installer and found one for a community edition (presumably free) and an enterprise edition. Should I install the community edition? Or is there some other open source version of Docker that I should be installing?

    The free version is all you need. However, if you're on a Debian-based distro, simply "apt-get install docker.io" will install it. You likely want to add yourself to the "docker" group and reboot though, else you'll need to run docker with sudo every time.
  • DavidZemon wrote: »
    David Betz wrote: »
    You say one of the prerequisite is that Docker be installed. I went to look for a Docker installer and found one for a community edition (presumably free) and an enterprise edition. Should I install the community edition? Or is there some other open source version of Docker that I should be installing?

    The free version is all you need. However, if you're on a Debian-based distro, simply "apt-get install docker.io" will install it. You likely want to add yourself to the "docker" group and reboot though, else you'll need to run docker with sudo every time.
    I see there is a Mac version of Docker. I wonder if I can use your Docker image with the Mac version?

  • Just what exactly is in your Docker container? Is it a full install of Ubuntu?
  • David Betz wrote: »
    DavidZemon wrote: »
    David Betz wrote: »
    You say one of the prerequisite is that Docker be installed. I went to look for a Docker installer and found one for a community edition (presumably free) and an enterprise edition. Should I install the community edition? Or is there some other open source version of Docker that I should be installing?

    The free version is all you need. However, if you're on a Debian-based distro, simply "apt-get install docker.io" will install it. You likely want to add yourself to the "docker" group and reboot though, else you'll need to run docker with sudo every time.
    I see there is a Mac version of Docker. I wonder if I can use your Docker image with the Mac version?

    Yes - the whole idea of docker is that anywhere Docker is installed, the images will be identical, whether I'm running Linux and you're on Mac, or I'm on Linux and you're on some custom OS (good luck getting Docker installed on your custom OS though :P ).
    David Betz wrote: »
    Just what exactly is in your Docker container? Is it a full install of Ubuntu?

    The base Ubuntu images in Docker are extremely stripped down, so no. Basically what you get is the Ubuntu folder structure, apt-get, and the Ubuntu mirrors configured for apt-get. Aside from that... not much more than ls and cp :P. What I've added to it can be seen from my Dockerfile: https://github.com/DavidZemon/PropGCCBuilder/blob/master/Dockerfile
  • Cool! It's building on my Mac. Do I have to follow all of these steps each time? For example, will I have to do the download_prerequisites step if I've just changed some of the sources in the propeller-gcc directory and want to rebuild?
  • This looks very nice. Thanks for setting this up, David!

    Eric
  • David Betz wrote: »
    Cool! It's building on my Mac. Do I have to follow all of these steps each time? For example, will I have to do the download_prerequisites step if I've just changed some of the sources in the propeller-gcc directory and want to rebuild?

    You do not have to repeat the steps each time. Any file changed in the mounted volume (everything under propgcc-parent) will remain after the container exits. So, the downloaded prereqs and all the build output will still be there when you launch the next container and remount the same volume.
    ersmith wrote: »
    This looks very nice. Thanks for setting this up, David!

    Eric

    Glad you like it! I'm really enjoying being able to run the latest and greatest bleeding edge OS while still being able build old apps. Docker definitely helps to solve the legacy software problem.
  • David Betz wrote: »
    Cool! It's building on my Mac. Do I have to follow all of these steps each time? For example, will I have to do the download_prerequisites step if I've just changed some of the sources in the propeller-gcc directory and want to rebuild?
    The build completed. Now I can unplug my Linux machine. :-)

  • David Betz wrote: »
    David Betz wrote: »
    Cool! It's building on my Mac. Do I have to follow all of these steps each time? For example, will I have to do the download_prerequisites step if I've just changed some of the sources in the propeller-gcc directory and want to rebuild?
    The build completed. Now I can unplug my Linux machine. :-)

    OH NO! Don't do that! Linux is the answer. Second only to 42, of course.
  • DavidZemon wrote: »
    David Betz wrote: »
    David Betz wrote: »
    Cool! It's building on my Mac. Do I have to follow all of these steps each time? For example, will I have to do the download_prerequisites step if I've just changed some of the sources in the propeller-gcc directory and want to rebuild?
    The build completed. Now I can unplug my Linux machine. :-)

    OH NO! Don't do that! Linux is the answer. Second only to 42, of course.
    Just kidding actually. I think about switching over to Linux entirely off and on but I still like my Mac. I kind of even like my Microsoft Surface Pro running Windows.

  • Heater.Heater. Posts: 21,230
    Best is my Surface Pro running Ubuntu and Debian in the Linux Subsystem For Windows. Now if only they would put a Linux kernel under there they would have a great OS.
  • Heater. wrote: »
    Best is my Surface Pro running Ubuntu and Debian in the Linux Subsystem For Windows. Now if only they would put a Linux kernel under there they would have a great OS.
    Why not run VirtualBox with Linux in a VM?

  • David Betz wrote: »
    Heater. wrote: »
    Best is my Surface Pro running Ubuntu and Debian in the Linux Subsystem For Windows. Now if only they would put a Linux kernel under there they would have a great OS.
    Why not run VirtualBox with Linux in a VM?

    WSL is much lighter weight than a VM
  • DavidZemon wrote: »
    David Betz wrote: »
    Heater. wrote: »
    Best is my Surface Pro running Ubuntu and Debian in the Linux Subsystem For Windows. Now if only they would put a Linux kernel under there they would have a great OS.
    Why not run VirtualBox with Linux in a VM?

    WSL is much lighter weight than a VM
    Yes but he said he wanted a real Linux kernel underneath. I've found VirtualBox works pretty well on my Windows, Mac, and Linux boxes.

  • Heater.Heater. Posts: 21,230
    I have been using VM's running Linux on Windows a lot over the years. It's was the only way I could get anything useful done when I had to use a Windows machine. That or Cygwin.

    The WSL has some advantages:

    As noted above it is much lighter than a VM.
    The whole Windows machine is available as /mnt/c so no messing with setting up shares.
    No messing with setting up networking.
    Cutting and pasting just works.

    The WSL has some disadvantages:

    It does not have a Linux kernel under it. So anything that needs that is off the table. A recent annoying example of that is that one cannot generate core dumps.
    No X Windows. I guess one could get a Windows X server running for that. Generally I don't need X Windows.

    I still have Cygwin installed. It's getting less and less use...







  • DavidZemon wrote: »
    However, if you want to play around with compiling PropGCC, this should make it much more reliable than trying to figure out compatible versions of GCC, since newer versions of GCC (such as what ships in Ubuntu 17.10) throw errors when trying to build PropGCC.

    Thanks for sharing, I'll give docker a try.

    I'm currently compiling PropGCC with Ubuntu 16.04 and gcc 5.4.0 without problems, I don't remember if I have applied a patch for it. If interested I can try to build the original sources and see what happens.

    Also, I have added targets for 64 bit Windows. I don't have a PC with 64 bit Windows to test but compile works fine. If interested the patches are for the makefiles to use the 64 bit version of mingw cross compiler.
  • macca wrote: »
    DavidZemon wrote: »
    However, if you want to play around with compiling PropGCC, this should make it much more reliable than trying to figure out compatible versions of GCC, since newer versions of GCC (such as what ships in Ubuntu 17.10) throw errors when trying to build PropGCC.

    Thanks for sharing, I'll give docker a try.

    I'm currently compiling PropGCC with Ubuntu 16.04 and gcc 5.4.0 without problems, I don't remember if I have applied a patch for it. If interested I can try to build the original sources and see what happens.

    Also, I have added targets for 64 bit Windows. I don't have a PC with 64 bit Windows to test but compile works fine. If interested the patches are for the makefiles to use the 64 bit version of mingw cross compiler.
    Are these patches for the Makefile in the propeller-gcc repository or are they for the original propgcc in the Parallax GitHub account?

  • David Betz wrote: »
    macca wrote: »
    DavidZemon wrote: »
    However, if you want to play around with compiling PropGCC, this should make it much more reliable than trying to figure out compatible versions of GCC, since newer versions of GCC (such as what ships in Ubuntu 17.10) throw errors when trying to build PropGCC.

    Thanks for sharing, I'll give docker a try.

    I'm currently compiling PropGCC with Ubuntu 16.04 and gcc 5.4.0 without problems, I don't remember if I have applied a patch for it. If interested I can try to build the original sources and see what happens.

    Also, I have added targets for 64 bit Windows. I don't have a PC with 64 bit Windows to test but compile works fine. If interested the patches are for the makefiles to use the 64 bit version of mingw cross compiler.
    Are these patches for the Makefile in the propeller-gcc repository or are they for the original propgcc in the Parallax GitHub account?

    I always use dbetz/propeller-gcc but should be applicable for the parallax repository too, they only add ifeq CROSS... lines to set the cross compilers.
  • macca wrote: »
    David Betz wrote: »
    macca wrote: »
    DavidZemon wrote: »
    However, if you want to play around with compiling PropGCC, this should make it much more reliable than trying to figure out compatible versions of GCC, since newer versions of GCC (such as what ships in Ubuntu 17.10) throw errors when trying to build PropGCC.

    Thanks for sharing, I'll give docker a try.

    I'm currently compiling PropGCC with Ubuntu 16.04 and gcc 5.4.0 without problems, I don't remember if I have applied a patch for it. If interested I can try to build the original sources and see what happens.

    Also, I have added targets for 64 bit Windows. I don't have a PC with 64 bit Windows to test but compile works fine. If interested the patches are for the makefiles to use the 64 bit version of mingw cross compiler.
    Are these patches for the Makefile in the propeller-gcc repository or are they for the original propgcc in the Parallax GitHub account?

    I always use dbetz/propeller-gcc but should be applicable for the parallax repository too, they only add ifeq CROSS... lines to set the cross compilers.
    Can you submit a pull request for my propeller-gcc repository?

  • David Betz wrote: »
    Can you submit a pull request for my propeller-gcc repository?

    I can try, does it work for the submodules ? The patch involves all tools.

    I'm attaching it here, just in case.
  • macca wrote: »
    David Betz wrote: »
    Can you submit a pull request for my propeller-gcc repository?

    I can try, does it work for the submodules ? The patch involves all tools.

    I'm attaching it here, just in case.
    I suspect you have to submit the patch for each submodule independently. I'll take a look at your zip file to see if I can manually merge.

  • dgatelydgately Posts: 1,620
    edited 2018-04-27 03:19
    David Betz wrote: »
    David Betz wrote: »
    Cool! It's building on my Mac. Do I have to follow all of these steps each time? For example, will I have to do the download_prerequisites step if I've just changed some of the sources in the propeller-gcc directory and want to rebuild?
    The build completed. Now I can unplug my Linux machine. :-)
    David, when you exec: docker pull davidzemon/propgccbuilder, on macOS, do you get "I have no name!@...." in front of all prompts?
    $ docker run -it -u $(id -u):$(id -g) -v `pwd`:`pwd` -w `pwd` davidzemon/propgccbuilder
    I have no name!@49844b3b824b:/Users/myUserName/source/propgcc-parent$ ls
    

    And, did you build as gcc4 or gcc? (I get errors building gcc4, currently still building as gcc without an error, yet...).

    EDIT: Wow, gcc (gccv6) completed its build:
    I have no name!@dc0dedd21a67:/Users/myUserName/source/propgcc-parent/propgcc-linux-bin/bin$ ./propeller-elf-gcc --version
    propeller-elf-gcc (propellergcc-alpha_v1_9_0_) 6.0.0 20150430 (experimental)
    Copyright (C) 2015 Free Software Foundation, Inc.
    This is free software; see the source for copying conditions.  There is NO...
    

    dgately
  • dgately wrote: »
    David Betz wrote: »
    David Betz wrote: »
    Cool! It's building on my Mac. Do I have to follow all of these steps each time? For example, will I have to do the download_prerequisites step if I've just changed some of the sources in the propeller-gcc directory and want to rebuild?
    The build completed. Now I can unplug my Linux machine. :-)
    David, when you exec: docker pull davidzemon/propgccbuilder, on macOS, do you get "I have no name!@...." in front of all prompts?
    $ docker run -it -u $(id -u):$(id -g) -v `pwd`:`pwd` -w `pwd` davidzemon/propgccbuilder
    I have no name!@49844b3b824b:/Users/myUserName/source/propgcc-parent$ ls
    

    Yes, this is normal. This is because, inside the docker image, there is no user with the UID specified via the "-u" parameter. Weird that Linux even functions when you're logged in as a user whose UID doesn't exist... but it works :D

    What error did you get for GCC 4?
  • dgately wrote: »
    David Betz wrote: »
    David Betz wrote: »
    Cool! It's building on my Mac. Do I have to follow all of these steps each time? For example, will I have to do the download_prerequisites step if I've just changed some of the sources in the propeller-gcc directory and want to rebuild?
    The build completed. Now I can unplug my Linux machine. :-)
    David, when you exec: docker pull davidzemon/propgccbuilder, on macOS, do you get "I have no name!@...." in front of all prompts?
    $ docker run -it -u $(id -u):$(id -g) -v `pwd`:`pwd` -w `pwd` davidzemon/propgccbuilder
    I have no name!@49844b3b824b:/Users/myUserName/source/propgcc-parent$ ls
    

    And, did you build as gcc4 or gcc? (I get errors building gcc4, currently still building as gcc without an error, yet...).

    EDIT: Wow, gcc (gccv6) completed its build:
    I have no name!@dc0dedd21a67:/Users/myUserName/source/propgcc-parent/propgcc-linux-bin/bin$ ./propeller-elf-gcc --version
    propeller-elf-gcc (propellergcc-alpha_v1_9_0_) 6.0.0 20150430 (experimental)
    Copyright (C) 2015 Free Software Foundation, Inc.
    This is free software; see the source for copying conditions.  There is NO...
    

    dgately
    Yes, I got the "I have no name!" prompt. I thought Docker had failed at first but then realized it was just a silly prompt. I built gcc4 using exactly the commands that David suggested. I haven't tried building gcc yet.

  • DavidZemon wrote: »
    What error did you get for GCC 4?

    I'm currently trying a new make after a make clean, but here's what I get with make, right now after previous make attempts:
    /Users/myUserName/source/propgcc-parent/propeller-gcc4-build/gcc/./gcc/xgcc -B/Users/myUserName/source/propgcc-parent/propeller-gcc4-build/gcc/./gcc/ -B/Users/myUserName/source/propgcc-parent/propeller-gcc4-build/target/propeller-elf/bin/ -B/Users/myUserName/source/propgcc-parent/propeller-gcc4-build/target/propeller-elf/lib/ -isystem /Users/myUserName/source/propgcc-parent/propeller-gcc4-build/target/propeller-elf/include -isystem /Users/myUserName/source/propgcc-parent/propeller-gcc4-build/target/propeller-elf/sys-include    -g -O2 -mcmm -O2  -g -O2 -DIN_GCC -DCROSS_DIRECTORY_STRUCTURE  -W -Wall -Wwrite-strings -Wcast-qual -Wstrict-prototypes -Wmissing-prototypes -Wold-style-definition  -isystem ./include  -Os -fomit-frame-pointer -Wno-missing-prototypes -g  -DIN_LIBGCC2 -D__GCC_FLOAT_NOT_NEEDED -fno-stack-protector -Dinhibit_libc -Os  -I. -I. -I../../.././gcc -I/Users/myUserName/source/propgcc-parent/propeller-gcc/gcc4/libgcc -I/Users/myUserName/source/propgcc-parent/propeller-gcc/gcc4/libgcc/. -I/Users/myUserName/source/propgcc-parent/propeller-gcc/gcc4/libgcc/../gcc -I/Users/myUserName/source/propgcc-parent/propeller-gcc/gcc4/libgcc/../include  -DHAVE_CC_TLS -DUSE_EMUTLS -o _gcov_ior_profiler.o -MT _gcov_ior_profiler.o -MD -MP -MF _gcov_ior_profiler.dep -DL_gcov_ior_profiler -c /Users/myUserName/source/propgcc-parent/propeller-gcc/gcc4/libgcc/../gcc/libgcov.c
    xgcc: error trying to exec '/Users/myUserName/source/propgcc-parent/propeller-gcc4-build/gcc/./gcc/as': execv: Bad file descriptor
    make[4]: *** [_gcov_indirect_call_profiler.o] Error 1
    make[4]: *** Waiting for unfinished jobs....
    make[4]: Leaving directory `/Users/myUserName/source/propgcc-parent/propeller-gcc4-build/gcc/propeller-elf/cmm/libgcc'
    make[3]: *** [multi-do] Error 1
    make[3]: Leaving directory `/Users/myUserName/source/propgcc-parent/propeller-gcc4-build/gcc/propeller-elf/libgcc'
    make[2]: *** [all-multi] Error 2
    make[2]: Leaving directory `/Users/myUserName/source/propgcc-parent/propeller-gcc4-build/gcc/propeller-elf/libgcc'
    make[1]: *** [all-target-libgcc] Error 2
    make[1]: Leaving directory `/Users/myUserName/source/propgcc-parent/propeller-gcc4-build/gcc'
    make: *** [/Users/myUserName/source/propgcc-parent/propeller-gcc4-build/gcc/libgcc-built] Error 2
    

    dgately

  • DavidZemon wrote: »

    BTW: Once I got gccv6 to build, I was only able to exec propeller-elf-gcc within Docker (because of course, it is an x86 elf executable, which won't run under macOS native). So, I've got to figure out how to create a macOS variant, similar to how you created a WIN variant? Or, am I totally off here?

    dgately
  • dgately wrote: »
    DavidZemon wrote: »

    BTW: Once I got gccv6 to build, I was only able to exec propeller-elf-gcc within Docker (because of course, it is an x86 elf executable, which won't run under macOS native). So, I've got to figure out how to create a macOS variant, similar to how you created a WIN variant? Or, am I totally off here?

    dgately
    When you're running inside of David's Docker container you're essentially running under Ubuntu Linux, right? So it should be able to run Linux x86 executables.
Sign In or Register to comment.