Anyone interested in using Conan (and Docker?) with the Propeller?

DavidZemon

Since this forum has shown such active and enthusiastic responses to complex tools in the Parallax ecosystem, I thought I'd talk about two more: Conan and Docker. I recently started a new job and have begun using these two tools - and I like them a lot.



Conan https://www.conan.io/
Finally, a package manager for the C/C++ world that is build system agnostic. It's very new and the C/C++ world is extremely complex (there's a reason nothing like it has become hugely popular despite the languages being so old), so it isn't perfect. But... it's pretty darned cool. In short, a package maintainer writes "recipes" and compiles the package using that recipe for as many "profiles" (a profile might be Linux/x86_64 and another might be Windows/x86) as possible. The resulting binaries are uploaded to a (hopefully public) repository. When another user wants to use the package, Conan downloads the recipe for them and checks their "profile." If the end-user's profile matches a binary in the repo, then the binary is downloaded and used. If not, then the recipe contains all information necessary to download the package source code and re-compile for the user's profile, and Conan will automatically do just that.

Imagine if a new Propeller user could just install Conan and then let Conan download all necessary tools: PropGCC, Simple Libraries, PropWare (see what I did there? "necessary" tools ), PropLoader, etc. From there, they'd just have to write their code and run! How cool!

My experience with Conan is with Linux only and x86_64 and ARM. Conan works well in Windows (so I've read, and expect to be true since it is Python) and, if I can get cross-compiling well for ARM, it should work just fine for the Propeller too.

So, the result is a lot of extra complexity for anyone that understands how all of it works together, but the user should see significantly less complexity because Conan will just take care of it all.



Docker https://www.docker.com/
Another tool that is massively more complex than anyone on these forums has asked for. There are a number of ways this could be leveraged for our community:
1) Alternative to Conan: Ship a Docker image with PropGCC, Simple Libraries, PropLoader (is this even useful in a Docker image??? I have no idea...), PropWare, etc. All a user has to do is pull the Docker image and they're off to the races!
2) Combine with Conan: Ship a Docker image that has Conan pre-installed along with the Propeller profile. Use Conan as described above, but without the worry of installing Conan or maintaining the profile. I don't know that this adds much value... either way you have to install one of the tools (assuming the user doesn't already have one of them) and Docker is certainly more complex (and larger) than Conan.
3) Purely for the sake of creating artifacts for end-users to download. I've already started this: my build server (http://david.zemon.name:8111?guest=1) has switched to Docker for cross-compiling OpenSpin and Spin2cpp for the Raspberry Pi and I plan to make the necessary configuration changes soon so that PropGCC for RPi is also built via Docker. Going down this route has the added benefit of being able provide to any Propeller user well-tested Docker images for compiling PropGCC from source! No more lengthy instructions explaining what to do with texinfo and how to download the ARM cross-compiler; just run "docker run --rm -v `pwd`:/opt/project -w /opt/project dockcross/linux-armv6 make -j9999" and pretty soon you'll have a copy of PropGCC!



I'm willing to put the work in for any of this that folks are interested in - I don't think it will be too hard. Just let me know if any of this sounds intriguing.



Update

Docker image in the works: https://hub.docker.com/r/davidzemon/propworld. Has full suite of Propeller tools but lacks any instructions or helper scripts for using them.

pmrobert

David, that sounds very intriguing. With P2 on the near horizon this approach could be quite helpful. Thanks!

Mike R...

msrobots

@DavidZemon,

like you I used TeamCity for automated builds, but mostly for the duplicate runner and stuff like automated testing. Over the years I stopped using it because of changing infrastructure at work and stricter deployment rules.

Most Windows users like downloadable binaries and I guess for schools/education having a certain revision on all computers, easy to install (and reinstall) complete, might be a godsend thing.

But if you could provide a complete image where PropGCC and all the other stuff compiles without days of hair pulling even windows user could get used to do it.

Recently there where some posts about compiling PropGCC and not current versions used on Parallax side, a common Docker Image with not just PropGCC + libraries + PropWare, but all of the P1 and hopefully P2 stuff would be perfect.

Currently a new user has to search for all the goodies, just finding out by reading the forum that spin2gcc exists, or PropellerIDE, or PropWare...

Hard Disk space and download volume is not really a problem so building one Development Docker would be a wonderful idea.

Lets make a list,

GCC - added
cmake - added
PropGCC
SimpleIDE
PropellerIDE
PropLoader
PropWare
spin2cpp
spinsim
Gears?
OpenSpin
PropTool
BST?
whatever BLOCKLY could use local

most of it should be available on GIT somewhere.

That is not too bad to put in one Docker.

So yes I do find this intriguing.

Did not know about conan, will look at it now...

Enjoy!

Mike

DavidZemon

In my phone so this is short and unresearched. I don't think GUI apps are feasible via docker. Cmd only

DavidZemon

Okay, home now and researching GUI apps in Docker. Looks like, for Linux, it is pretty straightforward. Install your GUI app in the Docker image (duh), make sure you have the necessary X libs (hello bloat), and then run the docker image with a volume mounted for a special X file: "-v /tmp/.X11-unix:/tmp/.X11-unix:ro". Not too hard. But on Windows? Well now you need all the bloat that this Docker solution implies as well as some X server like Xming or VcXsrv. I don't see many people going for that, though I certainly don't mind giving it a shot if people really do want it.

msrobots

Yeah, I was already guessing that, but was not sure.

But I looked at conan between this posts, and that is a really nice find, thanks David.

The server side is more Linux centric, but the client side seems to work fine with windows, even supporting MSYS, Mingw, and Visual Studio to build with.

Its build around GIT and can supply binaries when a build is not possible (aka PropTool), at my first glance this is very interesting. Even for my work. I need to test this.

It's MIT license.

Thanks again,

Mike

DavidZemon

msrobots wrote: »

Yeah, I was already guessing that, but was not sure.

But I looked at conan between this posts, and that is a really nice find, thanks David.

The server side is more Linux centric, but the client side seems to work fine with windows, even supporting MSYS, Mingw, and Visual Studio to build with.

Its build around GIT and can supply binaries when a build is not possible (aka PropTool), at my first glance this is very interesting. Even for my work. I need to test this.

It's MIT license.

Thanks again,

Mike

For a couple years now I've been looking for a C/C++ package manager. Having worked with Java (Maven) and JavaScript (NPM) at my first job and already being familiar with Python's pip, I just couldn't believe that no one had a (good) solution for C/C++ yet! But finally, Conan is here to try and rectify that. There have been other attempts in the past that fell through (biicode made an attempt https://github.com/biicode), and others that are thriving but require specific build systems (buckaroo https://buckaroo.readthedocs.io/en/latest/, gradle https://docs.gradle.org/current/userguide/native_software.html)... but it will likely take decades for an opinionated package manager to dominate in the C/C++ ecosystem due to the momentum of Make, CMake, AutoTools, etc. About a year ago I actually created a Maven plugin for my old company that was a wrapper around CMake. It was pretty cool! It used all of Maven's dependency management/resolution to download binary artifacts and then handed the build off to CMake. It relied on your project using CPack to create the artifact for Maven, but then Maven would come back into the picture and push that artifact up to your standard Maven repository such as Nexus or Artifactory. I was very happy with it . But now we (the C/C++ community, not my old company... poor them :P ) have Conan, and I like it much more.

Feel free to PM me any Conan questions. I've been working with it extensively over the last few weeks, trying to get a demo ready, so I'm pretty darned familiar with it at this point.

msrobots

OK, simple things first, I have a fresh installed Win10 here I can play with, and have nothing on it yet. And another one with VS2017/SQL2017 So we can play with them.

Q: Python 2.7 or Python 3.0? conan said 32 not 64 bit on windows. Any preference for 2.7 or 3.0?



Mike

DavidZemon

Update: Just learned that Docker supports a "--device" flag that can be used to mount devices into the Docker image. For instance: "--device/dev/ttyUSB0" should make it so you can program your Prop board from Docker. This is looking mighty promising for command line users at least.

DavidZemon

I haven't run into any problems with Python 2 or 3 (and I have to use both since our Docker images at work use 2 by default, but my host machine uses 3). So, if I were I'd go with 3 since 2 is quite old by now and 3 has a couple nice features.

ALSO! Check this out:

david@balrog:~/reusable/Documents/Programming/desktop/propworld$ docker run -it --rm -w /opt/project --device=/dev/ttyUSB0 davidzemon/propworld
root@2f599da56f61:/opt/project# cd /usr/PropWare/Examples/Hello/
root@2f599da56f61:/usr/PropWare/Examples/Hello# cmake .
-- Found PropWare: /usr/PropWare (found version "3.0.0.162") 
-- The C compiler identification is GNU 4.6.1
-- The CXX compiler identification is GNU 4.6.1
-- Check for working C compiler: /opt/parallax.gcc4_1.9.0/bin/propeller-elf-gcc
-- Check for working C compiler: /opt/parallax.gcc4_1.9.0/bin/propeller-elf-gcc -- works
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Detecting C compile features
-- Detecting C compile features - done
-- Check for working CXX compiler: /opt/parallax.gcc4_1.9.0/bin/propeller-elf-gcc
-- Check for working CXX compiler: /opt/parallax.gcc4_1.9.0/bin/propeller-elf-gcc -- works
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - failed
-- Detecting CXX compile features
-- Detecting CXX compile features - failed
-- Configuring done
-- Generating done
-- Build files have been written to: /usr/PropWare/Examples/Hello
root@2f599da56f61:/usr/PropWare/Examples/Hello# make debug
Scanning dependencies of target Hello_Demo
[ 33%] Building CXX object CMakeFiles/Hello_Demo.dir/Hello_Demo.cpp.obj
[ 66%] Building SPIN2DAT object CMakeFiles/Hello_Demo.dir/pst.spin.dat
[100%] Linking C executable Hello_Demo.elf
Code size  = 4688
Total size = 4924
[100%] Built target Hello_Demo
Scanning dependencies of target debug
Propeller Version 1 on /dev/ttyUSB0
Patching __cfg_txpin with 0000001e
Patching __cfg_baudrate with 0001c200
Loading /usr/PropWare/Examples/Hello/Hello_Demo.elf to hub memory
4688 bytes sent                  
Verifying RAM ... OK
[ Entering terminal mode. Type ESC or Control-C to exit. ]
Hello, world! 000 0x00
Hello, world! 001 0x01
Hello, world! 002 0x02
Hello, world! 003 0x03
Hello, world! 004 0x04
Hello, world! 005 0x05
Hello, world! 006 0x06
[100%] Built target debug
root@2f599da56f61:/usr/PropWare/Examples/Hello#

Now that is cool!

msrobots

first bad hit, Docker for Windows10 does not support Windows 10 Home, needs Pro or Enterprise or Windows Server 2016.

Mike

DavidZemon

msrobots wrote: »

first bad hit, Docker for Windows10 does not support Windows 10 Home, needs Pro or Enterprise or Windows Server 2016.

Mike

Well's terrible. Does Docker run in WSL? I've heard that Docker is actually Linux only, and that the Windows installer is actually just a Linux virtual machine with Docker already installed. So perhaps Docker also works from WSL?

DavidZemon

Okay, the Docker image that produced the above output (the "Hello, world! 001 0x01") is now available for download from Docker Hub: https://hub.docker.com/r/davidzemon/propworld/

I will continue making some updates to decrease the image size and increase the usability, such as writing a wrapper script to invoke it with some appropriate flags.

msrobots

Good Idea!

Have WSL installed one on of the W10 machines, but, but, but I do have no clue at all about Linux. Nada, Zilch.

JCL? no problem, Bash, hmmm....

I just installed that WSL to get GnuCOBOL running, not yet successful.

But GCC works and everything compiles, make. make install, .,.. all fine, it just starts still GnuCOBOL V.1.x instead of V2.x I tried to install.

My Linu Fu is very small...

Mike

DavidZemon

Assuming you went with the Ubuntu flavor of WSL, just run this:

sudo apt-get install docker.io
sudo usermod -a -G docker <your linux username>

Reboot

Then try to run the command I did above.

msrobots

In between I did install Python 3.0 and conan. Besides small issues with to long pathnames of windows standard install, flawless.

Looked up Docker and WSL, no go there for W10 Home.

Needs a windows Docker version to connect as client, WSL Ubuntu can not run docker directly

But conan works, I guess.

give me some test for this. say your OpenSpin compiled with Visual Studio Generateor and not cmake.

or something else....

[edit] installing cmake, but cmake does not like VS2017?]

Mike

msrobots

'sudo apt-get install docker.io' seem to fail on WSL with missing packages.

michael@Inspirion:/mnt/c/Users/Michael$ sudo apt-get install docker.io
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following additional packages will be installed:
  bridge-utils cgroupfs-mount containerd runc ubuntu-fan
Suggested packages:
  mountall aufs-tools debootstrap docker-doc rinse zfs-fuse | zfsutils
The following NEW packages will be installed:
  bridge-utils cgroupfs-mount containerd docker.io runc ubuntu-fan
0 upgraded, 6 newly installed, 0 to remove and 172 not upgraded.
Need to get 12.3 MB/16.4 MB of archives.
After this operation, 83.6 MB of additional disk space will be used.
Do you want to continue? [Y/n] y
Err:1 http://archive.ubuntu.com/ubuntu xenial-updates/universe amd64 runc amd64 1.0.0~rc2+docker1.12.6-0ubuntu1~16.04.1
  404  Not Found [IP: 2001:67c:1360:8001::21 80]
Err:2 http://archive.ubuntu.com/ubuntu xenial-updates/universe amd64 docker.io amd64 1.12.6-0ubuntu1~16.04.1
  404  Not Found [IP: 2001:67c:1360:8001::21 80]
Err:3 http://archive.ubuntu.com/ubuntu xenial-updates/main amd64 ubuntu-fan all 0.9.2
  404  Not Found [IP: 2001:67c:1360:8001::21 80]
E: Failed to fetch http://archive.ubuntu.com/ubuntu/pool/universe/r/runc/runc_1.0.0~rc2+docker1.12.6-0ubuntu1~16.04.1_amd64.deb  404  Not Found [IP: 2001:67c:1360:8001::21 80]

E: Failed to fetch http://archive.ubuntu.com/ubuntu/pool/universe/d/docker.io/docker.io_1.12.6-0ubuntu1~16.04.1_amd64.deb  404  Not Found [IP: 2001:67c:1360:8001::21 80]

E: Failed to fetch http://archive.ubuntu.com/ubuntu/pool/main/u/ubuntu-fan/ubuntu-fan_0.9.2_all.deb  404  Not Found [IP: 2001:67c:1360:8001::21 80]

E: Unable to fetch some archives, maybe run apt-get update or try with --fix-missing?

DavidZemon

Okay, so Docker is really just a no go for Propeller + Windows. Ain't happening. But... the image is out there for anyone in the Linux world that wants to play with it.

On to Conan. Now this one is truly command-line only. (Obviously you can edit your files in your own editor, but we won't be packaging SimpleIDE in Conan.)

First step is get PropGCC built as a Conan package so that other packages can depend on it. Annndd already we're running into a problem: PropGCC doesn't compile on super new versions of GCC. Back to Docker! Now I get to build a Docker image, probably Ubuntu 16.04, that has an older version of GCC so that I can compile PropGCC! Woo hoo!