For anyone building anything using cmake: please forget that gnu make exists. It's highly counter-productive. Use ninja everywhere. It's the defacto modern build tool. Once you experience its performance, going back to gnu make will sound and feel ridonkulous. make tries to be a general purpose tool, and its code base and performance bears that fact. ninja tries to do one job only and does it extremely well.`cmake -G Ninja` all the way. Thank me later Yes, you need to install ninja.
I've tried Ninja before. It made an insignificant difference in both of the two projects I tried it on.
For anyone building anything using cmake: please forget that gnu make exists. It's highly counter-productive. Use ninja everywhere. It's the defacto modern build tool. Once you experience its performance, going back to gnu make will sound and feel ridonkulous. make tries to be a general purpose tool, and its code base and performance bears that fact. ninja tries to do one job only and does it extremely well.`cmake -G Ninja` all the way. Thank me later Yes, you need to install ninja.
I've tried Ninja before. It made an insignificant difference in both of the two projects I tried it on.
Nor do I think the exact build system you use matters here (and isn't relevant to this discussion)... I used Ninja at work, couldn't really tell a difference in what it actually does vs make and I know make. But if anyone wants to use Ninja instead, feel free to! Just change the generator type when running cmake, as kuba wrote above.
Been quiet so quick update: I’ve rewritten how instructions are formatted to support conditions (if_z, etc) and effect flags (wz, etc) as operands to each instruction. This has created a unique thing situation that hasn’t been tested in LLVM before: instructions where the instruction mnemonic doesn’t come first and there are optional operands (the effect flag). As a result, something in the assembly parser isn’t happy. I think this is an actual bug in LLVM and I’ve isolated to where it happens, I just need to figure out a good fix. Hopefully this weekend, I’ll be able to post next version which fixes that, as well as supports more of the c library, particularly dynamic memory allocation (with locks and such)
Wait, doesn't x86 have instruction prefixes (REP, LOCK, etc), too?
Maybe, I don't know x86, but I do the know the x86 backend in LLVM is HUGE and has lots of custom parsing code, so it's likely the target-specific code handles this rather than the LLVM internals. The constructs that I'm using for having optional arguments AND instruction prefixes seems to not work. Anyway, I know where to fix the problem, just need to figure out how.
I wonder if there is a way to treat each variation of the instruction with different trailing argument (like wz, wc, wcz) more like just 3 new/different instructions, one of which you will choose to emit depending on what needs to be done.
I wonder if there is a way to treat each variation of the instruction with different trailing argument (like wz, wc, wcz) more like just 3 new/different instructions, one of which you will choose to emit depending on what needs to be done.
Yes that’s totally possible, and is how I initially did it for a few specific instructions, but would be a bit of a headache to write all those out for every instruction--i've got them working as operands, just need to clean a few things up.
latest version of compiler and library: https://github.com/ne75/p2llvm/releases/tag/v0.3, implementing things from above with instruction/effect flags. Next going to keep expanding c standard library functionality and adding instructions as needed.
Fwiw, this LLVM backend could also open the door to support for Go (golang) via TinyGo. I went ahead and made this to help track the work involved: https://github.com/tinygo-org/tinygo/issues/1336
Is there any info on what would be required to get this fork upstreamed into the main LLVM repo? Perhaps there a discussion that should be started with the LLVM maintainers on this? Just from a cursory check it seems like they do contributions via patches http://llvm.org/docs/Contributing.html#how-to-submit-a-patch
@DavidZemon made this suggestion near the top of the thread and I think it's an import aspect of helping make this effort successful. It might be worth submitting a patch of the current code to the main LLVM project to see what their feedback is?
Fwiw, this LLVM backend could also open the door to support for Go (golang) via TinyGo. I went ahead and made this to help track the work involved: https://github.com/tinygo-org/tinygo/issues/1336
Is there any info on what would be required to get this fork upstreamed into the main LLVM repo? Perhaps there a discussion that should be started with the LLVM maintainers on this? Just from a cursory check it seems like they do contributions via patches http://llvm.org/docs/Contributing.html#how-to-submit-a-patch
@DavidZemon made this suggestion near the top of the thread and I think it's an import aspect of helping make this effort successful. It might be worth submitting a patch of the current code to the main LLVM project to see what their feedback is?
I agree, I do want to get it into mainline LLVM, but I think that requires a) it be more complete, b) have the C std library more implemented than it currently is (which I'm working on) and c) someone else to help me push that effort forward. My branch of this is now a proper fork so we can merge in new LLVM features as they are made before putting in the patch request. I anticipate only 1 place where I made a bandaid fix in the actual LLVM code that needs to be properly addressed, but the rest I think would be able to be merged in without issue.
In other news, the FILE driver interface from propgcc has been ported over to the dev branch and it seems to work. I haven't tested it much, but I modified the SimpleSerialDriver with smart pins to be used as the default for stdin/stdout/stderr as it was on P1.
So once it is good shape will there be distributable binaries for the different plaforms in the end? After all the efforts I did to try to build it and ultimately failing myself with Yosemite, I guess some people won't easily be able to use these tools otherwise, depending on their systems. Basically will anyone who wants to use LLVM for P2 always need to build their own toolchain, or once it is part of mainline LLVM does that mean we will get binaries (e.g. for MacOS X)?
So once it is good shape will there be distributable binaries for the different plaforms in the end? After all the efforts I did to try to build it and ultimately failing myself with Yosemite, I guess some people won't easily be able to use these tools otherwise, depending on their systems. Basically will anyone who wants to use LLVM for P2 always need to build their own toolchain, or once it is part of mainline LLVM does that mean we will get binaries (e.g. for MacOS X)?
Yes I fully intend to provide a binary you can directly use without compiling anything, although it will be tough to support older OS’s if I’m running newer ones with newer libraries. Each tag I have on GitHub current contains the binaries I compiled on macOS, are you able to run those? If not, I’ll need to figure out a way to make it compatible with older versions of macOS/OS X, but that won’t be as high a priority for me.
I also have no intention of taking down my CI server any time soon, so you'll be able to find Linux binaries there for the foreseeable future. My guess is, since the build system is CMake, that I could pretty easily cross-compile for Windows as well (as opposed to other projects where that has historically taken a significant development effort).
I also have no intention of taking down my CI server any time soon, so you'll be able to find Linux binaries there for the foreseeable future. My guess is, since the build system is CMake, that I could pretty easily cross-compile for Windows as well (as opposed to other projects where that has historically taken a significant development effort).
Which branch does your CI server pull from? master? And do you mind if I just pull the binaries you build and include them in my tags?
For windows, I think I’ll leave that effort to someone else who really wants it. I haven’t written code on Windows in like 7-8 years and am super unfamiliar with how to port these things over
I also have no intention of taking down my CI server any time soon, so you'll be able to find Linux binaries there for the foreseeable future. My guess is, since the build system is CMake, that I could pretty easily cross-compile for Windows as well (as opposed to other projects where that has historically taken a significant development effort).
Which branch does your CI server pull from? master? And do you mind if I just pull the binaries you build and include them in my tags?
For windows, I think I’ll leave that effort to someone else who really wants it. I haven’t written code on Windows in like 7-8 years and am super unfamiliar with how to port these things over
It builds both dev and master branches. And you're most welcome to pull binaries direct from the CI server, or link to them, whatever you'd like.
I tried a Windows build... it died pretty quickly. Maybe cross compiling with clang would be better than MinGW. An idea for another time.
@n_ermosh, current dev branch is failing on the CI server. Does it build for you? Maybe this is another GCC v. LLVM difference, and it would work if I built it with clang instead?
Sorry for the long delay. Yes, it built successfully now
There was a failed build when i woke up this morning, but that's because I had forgotten to restrict the "linux" build configuration to Linux agents. So it tried to build on a Mac agent and failed... what a surprise :P
Sorry for the quiet, work has been super busy lately.
I'm currently working on setting up several example programs to demonstrate the various features, especially those related to LLVM. One of these will be my LVDS driver, which I can just compile with fastspin and include a binary since it's all assembly anyway, but I want to see how close I can get LLVM's assembly parser to mimic fastspin's assembly parser to keep it all in one tool. Not the end of the world if it doesn't work.
I also emailed the LLVM devs about merging this into mainline LLVM, haven't heard back, but from what I've read online, one thing they desire is unit tests. I have no idea how to go about writing unit tests for this, especially without a solid P2 emulator that can be used to verify the code runs as expected in a controlled environment with software feedback of the chip's state. If anyone has ideas, let me know. I started thinking about writing a Python-based P2 emulator, but quickly decided against that much scope creep on this project.
You probably first want to write simple unit tests that exercises most C functionality to prove the generated code works without obvious error. You may have already done a lot of that. How stringent they would want their unit tests to be I'm not sure. You could possibly write expect scripts and have something automated that uses printf with output you can match on to tell that the code did the correct thing.
for loops
if
case statements
goto
function calls
function pointers
assignment statements
pointer increment/decrements
expressions
etc...
Also some heap allocation stuff would be good to test.
It could just about go on endlessly though if you wanted to test all possible functionality and need to look for memory corruption or modifications outside of valid register areas etc. Just start simple and build up over time.
I'd be figuring what unit tests look like for other embedded systems that llvm supports. Without knowing what those look like though, I might hand write an assembly file and verify that clang generates the right listing. Then somehow acquire a valid binary (fastspin? Trial and error with llvm? Whatever) and verify that file matches what llvm's assembler outputs. Since llvm isn't responsible for loading into the p2, I would say your tests don't need to verify anything past writing the correct file to disk.
I've released v0.4 on GitHub. This release introduces more instructions, support for constructors in the startup code, and most importantly, cogex functions. A function can be marked with the "cogmain" attribute, so it will get compiled with appropriate prologue to be started in a cog and run directly using the _coginit() function. This allows for fast applications to be surrounded by higher level C++ code. Currently, there's no support for bundling multiple functions together to run in one cog, or for cog functions to call another cog function (but it can call a hub function). However, that is likely fairly simple to do by specifying section names correctly and adding some special support in the backend for generating a call instruction with a relative address (which is easy to do).
Additionally, a series of examples was introduced to the git repo to provide some very basic starting points for new users of this toolchain. The LVDS example is a (relatively) simple program demonstrating running a cog function within a C++ class where timing is extremely crucial, but it can interact with external hub-based code running in other cogs. This is pretty much a direct port of the LVDS driver I demoed before, but now in a nice and portable C++ class. I also want to write and HDMI, USB, and Ethernet example, but that will come some time later.
This will likely be my last post on this topic for a while. Now that is in a complete enough state to unlock a lot of projects, I'm going to go work on one of those . However, if you are using this tool and find a bug, have a feature request, or want to make improvements, please please reach out to me, I want to see this project really come to completion. I've added a laundry list of things that still need work, I'll knock them out as I find a need for those features.
Fwiw, this LLVM backend could also open the door to support for Go (golang) via TinyGo. I went ahead and made this to help track the work involved: https://github.com/tinygo-org/tinygo/issues/1336
I have another llvm-compiled language in mind so thank you for sharing this. I'll be following with great interest and trying to apply the same principles to mine.
I know how to get the LLVM compiler to spit out LLVM IR, but I don't think that getting it into a p2 executable is as simple as just piping the LLVM IR into the backend - is it?
Fwiw, this LLVM backend could also open the door to support for Go (golang) via TinyGo. I went ahead and made this to help track the work involved: https://github.com/tinygo-org/tinygo/issues/1336
I have another llvm-compiled language in mind so thank you for sharing this. I'll be following with great interest and trying to apply the same principles to mine.
I know how to get the LLVM compiler to spit out LLVM IR, but I don't think that getting it into a p2 executable is as simple as just piping the LLVM IR into the backend - is it?
In principle it is that simple. I did minimal work on the front end just to get a more seamless and familiar “compile + link in one step” workflow, but the way LLVM is structured, the IR is used in the middle specifically to separate the front end from the backend. Try it and let me know if it works. The part I’m not sure on is how to link in the startup code that is written in C++ and compiled with clang. That might be the part that breaks the workflow and requires more effort to get it running.
I am on a windows system using the Visual Studio 2019 build. I see the targets listed but don't see how to add the P2 as a target?
I successfully built the LLVM system in Visual Studio with no errors.
Mike
Are you trying to build LLVM itself or build a P2 application using the LLVM you've successfully built? I've never done this on Windows, but will do my best to help you out there.
When compiling LLVM, you need to specify P2 as an experimental target to build. When compiling a program with your newly built clang, specify --target=p2. I've attached an example cmake toolchain file.
Because I'm on windows I didn't run your python script but the LLVM script which does not have P2 set as a target and does not get added to the build process for Visual Studio.
Don't know how to add that target to the build process.
Comments
I've tried Ninja before. It made an insignificant difference in both of the two projects I tried it on.
Nor do I think the exact build system you use matters here (and isn't relevant to this discussion)... I used Ninja at work, couldn't really tell a difference in what it actually does vs make and I know make. But if anyone wants to use Ninja instead, feel free to! Just change the generator type when running cmake, as kuba wrote above.
Maybe, I don't know x86, but I do the know the x86 backend in LLVM is HUGE and has lots of custom parsing code, so it's likely the target-specific code handles this rather than the LLVM internals. The constructs that I'm using for having optional arguments AND instruction prefixes seems to not work. Anyway, I know where to fix the problem, just need to figure out how.
Yes that’s totally possible, and is how I initially did it for a few specific instructions, but would be a bit of a headache to write all those out for every instruction--i've got them working as operands, just need to clean a few things up.
Is there any info on what would be required to get this fork upstreamed into the main LLVM repo? Perhaps there a discussion that should be started with the LLVM maintainers on this? Just from a cursory check it seems like they do contributions via patches http://llvm.org/docs/Contributing.html#how-to-submit-a-patch
@DavidZemon made this suggestion near the top of the thread and I think it's an import aspect of helping make this effort successful. It might be worth submitting a patch of the current code to the main LLVM project to see what their feedback is?
I agree, I do want to get it into mainline LLVM, but I think that requires a) it be more complete, b) have the C std library more implemented than it currently is (which I'm working on) and c) someone else to help me push that effort forward. My branch of this is now a proper fork so we can merge in new LLVM features as they are made before putting in the patch request. I anticipate only 1 place where I made a bandaid fix in the actual LLVM code that needs to be properly addressed, but the rest I think would be able to be merged in without issue.
In other news, the FILE driver interface from propgcc has been ported over to the dev branch and it seems to work. I haven't tested it much, but I modified the SimpleSerialDriver with smart pins to be used as the default for stdin/stdout/stderr as it was on P1.
So once it is good shape will there be distributable binaries for the different plaforms in the end? After all the efforts I did to try to build it and ultimately failing myself with Yosemite, I guess some people won't easily be able to use these tools otherwise, depending on their systems. Basically will anyone who wants to use LLVM for P2 always need to build their own toolchain, or once it is part of mainline LLVM does that mean we will get binaries (e.g. for MacOS X)?
It sort of appears that should be the case...
https://releases.llvm.org/download.html
Yes I fully intend to provide a binary you can directly use without compiling anything, although it will be tough to support older OS’s if I’m running newer ones with newer libraries. Each tag I have on GitHub current contains the binaries I compiled on macOS, are you able to run those? If not, I’ll need to figure out a way to make it compatible with older versions of macOS/OS X, but that won’t be as high a priority for me.
Which branch does your CI server pull from? master? And do you mind if I just pull the binaries you build and include them in my tags?
For windows, I think I’ll leave that effort to someone else who really wants it. I haven’t written code on Windows in like 7-8 years and am super unfamiliar with how to port these things over
It builds both dev and master branches. And you're most welcome to pull binaries direct from the CI server, or link to them, whatever you'd like.
I tried a Windows build... it died pretty quickly. Maybe cross compiling with clang would be better than MinGW. An idea for another time.
Build failure: https://ci.zemon.name/buildConfiguration/P2llvm_LlvmLinuxX64/7792?guest=1&showLog=7792_4356_4267.4356
There was a failed build when i woke up this morning, but that's because I had forgotten to restrict the "linux" build configuration to Linux agents. So it tried to build on a Mac agent and failed... what a surprise :P
I'm currently working on setting up several example programs to demonstrate the various features, especially those related to LLVM. One of these will be my LVDS driver, which I can just compile with fastspin and include a binary since it's all assembly anyway, but I want to see how close I can get LLVM's assembly parser to mimic fastspin's assembly parser to keep it all in one tool. Not the end of the world if it doesn't work.
I also emailed the LLVM devs about merging this into mainline LLVM, haven't heard back, but from what I've read online, one thing they desire is unit tests. I have no idea how to go about writing unit tests for this, especially without a solid P2 emulator that can be used to verify the code runs as expected in a controlled environment with software feedback of the chip's state. If anyone has ideas, let me know. I started thinking about writing a Python-based P2 emulator, but quickly decided against that much scope creep on this project.
for loops
if
case statements
goto
function calls
function pointers
assignment statements
pointer increment/decrements
expressions
etc...
Also some heap allocation stuff would be good to test.
It could just about go on endlessly though if you wanted to test all possible functionality and need to look for memory corruption or modifications outside of valid register areas etc. Just start simple and build up over time.
Additionally, a series of examples was introduced to the git repo to provide some very basic starting points for new users of this toolchain. The LVDS example is a (relatively) simple program demonstrating running a cog function within a C++ class where timing is extremely crucial, but it can interact with external hub-based code running in other cogs. This is pretty much a direct port of the LVDS driver I demoed before, but now in a nice and portable C++ class. I also want to write and HDMI, USB, and Ethernet example, but that will come some time later.
This will likely be my last post on this topic for a while. Now that is in a complete enough state to unlock a lot of projects, I'm going to go work on one of those
I have another llvm-compiled language in mind so thank you for sharing this. I'll be following with great interest and trying to apply the same principles to mine.
I know how to get the LLVM compiler to spit out LLVM IR, but I don't think that getting it into a p2 executable is as simple as just piping the LLVM IR into the backend - is it?
In principle it is that simple. I did minimal work on the front end just to get a more seamless and familiar “compile + link in one step” workflow, but the way LLVM is structured, the IR is used in the middle specifically to separate the front end from the backend. Try it and let me know if it works. The part I’m not sure on is how to link in the startup code that is written in C++ and compiled with clang. That might be the part that breaks the workflow and requires more effort to get it running.
Trying to build LLVM but I'm lost.
I am on a windows system using the Visual Studio 2019 build. I see the targets listed but don't see how to add the P2 as a target?
I successfully built the LLVM system in Visual Studio with no errors.
Mike
Are you trying to build LLVM itself or build a P2 application using the LLVM you've successfully built? I've never done this on Windows, but will do my best to help you out there.
When compiling LLVM, you need to specify P2 as an experimental target to build. When compiling a program with your newly built clang, specify
--target=p2. I've attached an example cmake toolchain file.Because I'm on windows I didn't run your python script but the LLVM script which does not have P2 set as a target and does not get added to the build process for Visual Studio.
Don't know how to add that target to the build process.
Mike