Rust on P2: It's rough, but it works

n_ermosh

I got Rust working on P2. This is rather lengthy and manual process to set it up right now, but it does work and steps to get running are below. Plan for about 3 hours to go through it all. @DavidZemon looking to you to help clean things up

Prerequisites

1. Unix. Not planning to figure out how to do this on Windows any time soon
2. Can successfully compile p2llvm (the LLVM compiler suite and P2 support libraries)
3. p2llvm installed somewhere. you'll need the linker (ld.lld) and libp2.a from the installation

Building Rust

1. Clone the rust fork: https://github.com/ne75/rust
2. Within the repo, run ./x.py setup. x.py is the primary tool used to bootstrap the rust compiler, which is built in stages. This will create a config.toml. This is used to actually configure the build.
   • During the run, it will ask what you are installing for. I selected (c), probably what you should do to.
3. Fill out config.toml. I've attached the one I used, which is for an arm64 Mac.
   • The main thing to change for other hosts is to change which LLVM targets are built. set targets to your host. LLVM will be built and used to build rust during the bootstrapping process, so it needs to be able to build binaries for your host, as well as P2. Basically, set this to X86 for intel processors, Aarch64 for Apple ARM.
   • This sets the install directory to /opt/p2rust. Change as desired.
   • See config.toml.example for a detailed documentation of this file, available keys, etc.
4. Update the submodules (including pulling LLVM). git submodule update --init
   • You can skip this by changing submodules to true in the configuration file
5. Build rustc with ./x.py build.
   • This will build LLVM and the several stages of the compiler.
   • In theory, you shouldn't need to rebuild LLVM if you build and install the rust branch for p2llvm, but I haven't figured out the details there, so for now, just rebuild llvm.
6. Install rustc: ./x.py install
7. Install cargo: ./x.py install cargo
8. Add the rust install path (/opt/p2rust/bin or wherever) to your PATH
9. Build libcore:
   • cd library/core
   • cargo build --target p2
   • cd ../..
   • This build is not optimized (which would be done by adding --release to the build command). I need to figure out how to make rustc not try to emit jump tables, which the backend doesn't currently support.
10. Build compiler builtins
    • cd compiler_builtins
    • cargo build
    • cd ..
11. Copy everything to the sysroot
    • mkdir /opt/p2rust/lib/rustlib/p2/lib/
    • cp target/p2/debug/libcore.rlib /opt/p2rust/lib/rustlib/p2/lib/
    • cp compiler_builtins/target/p2/debug/libcompiler_builtins.rlib /opt/p2rust/lib/rustlib/p2/lib/
12. Copy in libp2 from an llvm build (included in the attached zip)
    • cp <path to libp2.a> /opt/p2rust/lib/rustlib/p2/lib

Building an Application

1. Use the attached blink program as a starting point, or create a new crate with cargo new.
2. Fill out the build script, build.rs with the script attached, and cargo.toml
   • update path in build.rs to the linker script
   • update path in .cargo/config.toml to the linker installation.
3. Write your application
4. Compile with cargo build --release (YMMV with --release, if it tries to use jump tables, same as above)
5. Load with loadp2 (compiled binary is in target/p2/debug or target/p2/release)

I tried all of these instructions out as I was writing them, so they should work okay. Reach out if there are glaring problems, hopefully this is enough to get going.

An unrelated aside

I think this effort demonstrates the importance of the LLVM effort and why getting a fully featured P2 target in the LLVM system is necessary. In just 3 days, I was able to add support for a language I don't even know to this chip. Which is kind of incredible and shows the power of LLVM. In principle any language that uses LLVM as a backend is now on the quick horizon. I think that's pretty awesome.

DavidZemon

DavidZemon

@n_ermosh

p2llvm installed somewhere. you'll need the linker (ld.lld) and libp2.a from the installation

I didn't see anywhere in the instructions that ld.lld was referenced. Is libp2.a the only thing we need from the p2-llvm fork?

n_ermosh

When you actually build an application, you'll need to provide the path to ld.lld in the .cargo/config configuration.

octetta

Very nice... on a parallel path, if I were smarter and slept less, I'd be trying to make Zig work with what y'all have been doing. The P2 gets mentioned on the Zig/Discord list a few times.

n_ermosh

It’s probably not that hard either. I had pretty much no idea what I was doing going into this and it turns out it wasn’t that bad

Tubular

Great going, congrats Nikita

DavidZemon

@n_ermosh
just fyi, i was able to compile p2rust successfully locally and am working on getting the CI server to build it. I have all the steps configured but the build is failing. No time to investigate, but will try to get it going soon. Once I get it working, I'll extract it to a template which can then be re-used for building a mac version.

n_ermosh

Sounds great! Does it make sense to try to integrate with p2llvm in some way? So that one p2llvm is built once to get the llvm binaries and P2 libraries, and have rust build off that, rather than its build its own?

DavidZemon

@n_ermosh said:
Sounds great! Does it make sense to try to integrate with p2llvm in some way? So that one p2llvm is built once to get the llvm binaries and P2 libraries, and have rust build off that, rather than its build its own?

Oh absolutely. Let me know when you have that working

n_ermosh

The only hiccup is that there a few rust-specific changes to core LLVM libraries, so if I merge those into the main p2llvm branch, we will unlikely be able to merge into upstream llvm. This should be solved once rust upgrades to LLVM 14 (currently they use 13, but p2llvm is 14). However those aren’t features that p2llvm uses, so might be able to get away with one branch for everything.

I also want to keep the core P2 library constant between the two languages. I.e don’t write a separate libp2 implementation in rust, just make bindings to the existing functions. So that as I upgrade/fix libp2 in C, those changes carry over instantly to the rust side.

__deets__

Just leaving a “hell yeah” here, as this is something I’m very excited about!

DavidZemon

Wowza this is slow progress. Ran some more builds last night - figured out that it was failing on my 8-core VM (the CI build agent) despite having passed on my 16-core host. Switched the CI server to build with "-j1" overnight and it finally got passed that step then failed on disk space lol. Increased the VM's disk size this morning, bumped the build parallelism to -j4 and after an hour it finally failed because pkg-config wasn't installed. Sigh. Another build was just triggered. Hopefully this one will work!

n_ermosh

There's a way to turn on incremental builds that might be helpful until you get all the build kinks sorted out--but I have noticed the builds are annoyingly slow, especially with a debug build of LLVM...

DavidZemon

Woohoo! I got a successful build of p2rust Folks can go download the Linux x86 version from https://ci.zemon.name/project/P2rust?mode=builds&guest=1

@n_ermosh, regarding that ld.lld prereq. Linux versions of p2llvm haven't built on the CI server for a little while and I haven't taken the time to dig into it. But you mentioned that llvm is getting built as part of p2rust as well. So I did a search in the p2rust directory for any reference to ld.lld and founnd a bunch:

teamcity@teamcity-agent:~/BuildAgent/work/6725556b473544a4$ find . -name ld.lld
./src/llvm-project/clang/test/Driver/Inputs/basic_cross_linux_tree/usr/x86_64-unknown-linux-gnu/bin/ld.lld
./src/llvm-project/clang/test/Driver/Inputs/hexagon_tree/Tools/bin/ld.lld
./src/llvm-project/llvm/utils/lit/tests/Inputs/lld-features/ld.lld
./build/x86_64-unknown-linux-gnu/lld/bin/ld.lld
./build/x86_64-unknown-linux-gnu/lld/build/bin/ld.lld

Any chance one of those is what we're looking for?

The artifact available in build #1 from the above Linux build contains libp2.a (pulled from the Mac release of p2llvm) but does not yet include a copy of ld.lld since that's an exe compiled for the host system and therefore I can't use the one from the Mac build.

n_ermosh

Yeah you can pull ld.lld from rust’s build (one of the last two from that list you gave). Only thing to watch out for is any shared libraries that it relies on—I’m not sure if it’s a completely standalone binary or not.

DavidZemon

@n_ermosh said:
Yeah you can pull ld.lld from rust’s build (one of the last two from that list you gave). Only thing to watch out for is any shared libraries that it relies on—I’m not sure if it’s a completely standalone binary or not.

Awesome, thanks. Seems to be mostly statically linked, so that should work out very nicely:

david@balrog:~/reusable/Code/Rust/rust/build/x86_64-unknown-linux-gnu/lld/bin$ ldd lld
    linux-vdso.so.1 (0x00007fffe1ef1000)
    libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x00007ffa3d7f6000)
    libstdc++.so.6 => /lib/x86_64-linux-gnu/libstdc++.so.6 (0x00007ffa3d5dd000)
    libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007ffa3d4f9000)
    libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007ffa3d4df000)
    libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007ffa3d2b7000)
    /lib64/ld-linux-x86-64.so.2 (0x00007ffa416af000)

DavidZemon

@n_ermosh, Linux llvm build is working on the CI server again. The only thing I really had to change was removing the default value for -j and force it to -j4. Same solution I had to apply to p2rust. I think there is a dependency issue with p2llvm's build system that is causing it to fail with -j8 but not -j4 or -j16. Not sure if that is unique to p2llvm or common to upstream as well.

Anyway, llvm builds are now available again for Linux (including WSL... in case anyone is interested) and the p2rust build should be available in an hour or so (one that includes lld this time). Once that build finishes, I'll download it and give it a try to make sure it can compile a P2 program. My new p2 board has arrived yet, so I'll post it here for someone else to try running once it's ready.

n_ermosh

nice! I'm still digging through issues in the backend that clang didn't cause before, but rust now is. something with jump analysis... but they only crop up when optimization is turned on.

DavidZemon

Okay, still working out some issues with the generated artifact. New build is running which fixes the path of libp2.a, removes duplicate copies of lld (what should have been symlinks ended up being copies) and adds source code for Rust standard library to the installable archive (helpful for IDEs).

I'm running into an issue trying to use the toolchain from CLion though. When I first ran this on TeamCity, I didn't think too much of it and just changed the build config around a little to avoid the problem. But I know CLion works with the default Rust toolchain, so this is specific to p2llvm or p2rust. Somewhere rustc is being invoked with "rustc -vV" and it ends up getting interpretted as a single command named "rustc -vV" instead of the command "rustc" with the argument "-vV". This, of course, throws a "file not found" error.

This was originally happening in the TeamCity build step for building the P2 core library. I had it configured as a executable with arguments:

Exe: env
Arguments: PATH="%env.DESTDIR%%PREFIX%/bin:$PATH" cargo build --target p2

And I fixed it by changing to custom script with value

set +x
export PATH="%env.DESTDIR%%PREFIX%/bin:$PATH"
cargo build --target p2

I kept cleaning up old build logs, so I don't have that failure log anymore, but CLion fails with a similar error:

Fetching target specific `cfg` options failed. Fallback to host options.

Execution failed (exit code 101).
/opt/p2rust/bin/cargo rustc -Z unstable-options --print cfg
stdout : 
stderr : error: could not execute process `rustc -vV` (never executed)

Caused by:
  No such file or directory (os error 2)

n_ermosh

Yeah thats what I don't like about cargo--it relies on rustc being in your path and doesn't use proper subprocess execution to invoke rustc, resulting in the errors like above.

Also I think I got everything updates to be able to build core in release mode (with optimizations). The solution is hacky--there's a assertion that will sometimes get triggered, but only in a debug build. In a release build that assertion is skipped over entirely. Not great to not figure out whats causing it but it still compiles...so I'll call it good until that code doesn't work

__deets__

Using all my cores.... I hope I can contribute to Rust on the P2. After spending quite some time staring at SPIN, I'm ready for a language that supports some nicer abstractions and types.

__deets__

Ok, just to follow up: it works! I got the blinking program to work! Which is of course amazing. Now there are a metric ton of stuff to understand 😅

__deets__

So I tried getting a more elaborate example to work. Success has been mixed. I managed to call into the libp2 uart functions, and produced output. Great! But I then tried to spin up a second cog, and that failed in myserious ways. It appears as if Rust creates code to initialize the stack that makes little sense to me. First my Rust program:

#![no_main]
#![no_std]
#![feature(asm_experimental_arch)]

use core::arch::asm;
use core::panic::PanicInfo;

#[link(name = "p2")]
extern "C" {
    fn _uart_init(rx: u32, tx: u32, baud: u32);
    fn _uart_putc(c: u8, rx: u32);
    fn cogstart(main: extern fn()->(), par: u32, stack: *mut u32, stacksize: u32) -> u32;
}

#[no_mangle]
fn memset(_s: *mut u32, _c: u32, _n: u32)
{
}

/// This function is called on panic.
#[panic_handler]
fn panic(_info: &PanicInfo) -> ! {
    loop {}
}

#[no_mangle]
extern fn blink2()
{
    unsafe {
        asm!{"dirh #57"};

        loop {
            asm!{"outnot #57"};
            asm!{"waitx ##4000000"};
        }
    }
}

#[no_mangle]
pub extern fn main() {
    let mut stack: [u32; 100] = [0; 100];
    unsafe {
        _uart_init(10, 12, 2000000);
        _uart_putc(66, 12);
        _uart_putc(13, 12);
        _uart_putc(10, 12);

        let stack_ref = &mut stack;
        let stack_ptr = stack_ref as *mut u32; // and a pointer, created from the reference
        let _res = cogstart(blink2, 0, stack_ptr, 100);

        asm!{"dirh #56"};

        loop {
            asm!{"outnot #56"};
            asm!{"waitx ##1000000"};
            _uart_putc(65, 12);
            _uart_putc(13, 12);
            _uart_putc(10, 12);
        }
    }
}

The memset deliberately does nothing, because I don't care about the stack having a special format.

However this is the main assembly:

00000a00 <main>:
     a00: 28 06 64 fd            setq #3
     a04: 61 a1 67 fc            wrlong r0, ptra++
     a08: 61 df 67 fc            wrlong r31, ptra++
     a0c: 00 00 00 ff            augs #0
     a10: 9c f1 07 f1            add ptra, #412 
     a14: f8 a1 03 f6            mov r0, ptra   # this looks like allocating the 100 longs on the stack. Great!
     a18: 90 a1 87 f1            sub r0, #400   
     a1c: 00 a2 07 f6            mov r1, #0 
     a20: 90 a5 07 f6            mov r2, #400   
     a24: 33 02 c0 fd            calla #\__unorddf2   # This is the inexplicable part to me. Here I'd expect a memset call or something similar, as that's what the compiler complained about.
     a28: 0a a0 07 f6            mov r0, #10    
     a2c: 0c a2 07 f6            mov r1, #12    
     a30: 42 0f 00 ff            augs #3906
     a34: 80 a4 07 f6            mov r2, #128   
     a38: e8 0b c0 fd            calla #\_uart_init
     a3c: 42 a0 07 f6            mov r0, #66    
     a40: 0c a2 07 f6            mov r1, #12    
     a44: 34 0c c0 fd            calla #\_uart_putc
     a48: 0d a0 07 f6            mov r0, #13    
     a4c: 0c a2 07 f6            mov r1, #12    
     a50: 34 0c c0 fd            calla #\_uart_putc
     a54: 0a a0 07 f6            mov r0, #10    
     a58: 0c a2 07 f6            mov r1, #12    
     a5c: 34 0c c0 fd            calla #\_uart_putc
     a60: f8 a5 03 f6            mov r2, ptra   
     a64: 90 a5 87 f1            sub r2, #400   
     a68: ff 47 00 ff            augs #18431
     a6c: 6c a4 67 fc            wrlong r2, #108
     a70: ff 47 00 ff            augs #18431
     a74: 68 a4 67 fc            wrlong r2, #104
     a78: 05 00 00 ff            augs #5
     a7c: e4 a0 07 f6            mov r0, #228   
     a80: 00 a2 07 f6            mov r1, #0 
     a84: 64 a6 07 f6            mov r3, #100   
     a88: a4 0b c0 fd            calla #\cogstart
     a8c: ff 47 00 ff            augs #18431
     a90: 64 de 67 fc            wrlong r31, #100
     a94: 41 70 64 fd            dirh #56   
     a98: 4f 70 64 fd            outnot #56 
     a9c: a1 07 80 ff            augd #1953
     aa0: 1f 80 64 fd            waitx #64  
     aa4: 41 a0 07 f6            mov r0, #65    
     aa8: 0c a2 07 f6            mov r1, #12    
     aac: 34 0c c0 fd            calla #\_uart_putc
     ab0: 0d a0 07 f6            mov r0, #13    
     ab4: 0c a2 07 f6            mov r1, #12    
     ab8: 34 0c c0 fd            calla #\_uart_putc
     abc: 0a a0 07 f6            mov r0, #10    
     ac0: 0c a2 07 f6            mov r1, #12    
     ac4: 34 0c c0 fd            calla #\_uart_putc
     ac8: cc ff 9f fd            jmp #-52

000002cc <__unorddf2>:
     2cc: 54 19 c0 fd            calla #\__unordXf2__
     2d0: 2e 00 64 fd            reta   

00001b04 <__unordXf2__>:
    1b04: 28 02 64 fd            setq #1
    1b08: 61 a1 67 fc            wrlong r0, ptra++
    1b0c: ff ff 3f ff            augs #4194303
    1b10: ff a3 07 f5            and r1, #511   
    1b14: 00 c0 3f ff            augs #4177920
    1b18: 00 a2 1f f2            cmp r1, #0 wcz
    1b1c: 01 a2 07 16   if_nc_and_nz     mov r1, #1 
    1b20: 00 a2 07 e6   if_c_or_z    mov r1, #0 
    1b24: ff ff 3f ff            augs #4194303
    1b28: ff a1 07 f5            and r0, #511   
    1b2c: 00 c0 3f ff            augs #4177920
    1b30: 00 a0 1f f2            cmp r0, #0 wcz
    1b34: 01 a0 07 16   if_nc_and_nz     mov r0, #1 
    1b38: 00 a0 07 e6   if_c_or_z    mov r0, #0 
    1b3c: d1 a1 43 f5            or r0, r1  
    1b40: d0 df 03 f6            mov r31, r0    
    1b44: 28 02 64 fd            setq #1
    1b48: 5f a1 07 fb            rdlong r0, --ptra  
    1b4c: 2e 00 64 fd            reta   

Below the main are the two code snippets that apparently the code jumps into. I can't make head or tail of them.

Any pointers on how to get to the bottom of this? I have successfully used objdump to extract the assembly, but any other tipps on how to progress are highly appreciated.

n_ermosh

Great progress so far!

Yeah the call to unorddf2 seems to be incorrect, and I'd guess that that's a mistake in the linking process. I'd disassemble the intermediate object file the compiler should be generating before linking to the standard library (I'm assuming that's how rustc works?), and see what it's trying to call. the actual operand will be 0 so it won't list the correct function name, but you can look at the relocation table (the -t flag I think) in the object file and see what the function is supposed to be after linking. From there, we can figure out why it's linking this function instead.

Also, I'd make sure to pull from my llvm-project/master branch into my llvm-project/rust branch and recompile. There have been some changed (including around the linker) that might be leading to this.

And, if the startup code is being called correctly, you might be able to use my debugger. It's super duper hacky right now but useful for single-stepping code and seeing what's going on.

__deets__

It seems that didn't work:

18:06 $ cd library/core
✔ ~/software/vc/p2rust/library/core [master|✚ 1…39] 
18:10 $ cargo build --target p2
error: failed to run `rustc` to learn about target-specific information

Caused by:
  process didn't exit successfully: `rustc - --crate-name ___ --print=file-names --target p2 --crate-type bin --crate-type rlib --crate-type dylib --crate-type cdylib --crate-type staticlib --crate-type proc-macro --print=sysroot --print=cfg` (exit status: 1)
  --- stderr
  error: Error loading target specification: Could not find specification for target "p2". Run `rustc --print target-list` for a list of built-in targets

I saw that there are commits in the p2rust submodule that I didn't find in your upstream master, so maybe that's the cause? Can you comment on that?

I also figured out how to get IR, if you care about that.

n_ermosh

IR is useful--it can help see what it's feeding the P2 target backend.

Yeah, the rust branch has a few specific things that aren't in master (I use LLVM 14, rust is still on LLVM 13 and they aren't backwards compatible. Once rust upgrades to 14, it should be the same), but master has new features for how startup and library linking works that are probably needed, so you should pull those into the submodule, and get the latest libp2 and libc.

For clarity, we have the following branches:

ne75/p2llvm/master: the main p2llvm repo. Contains the P2 llvm branch, libp2, libc, etc.
ne75/llvm-project/master: The main llvm branch for p2llvm. Contains all the P2-specific code. Use this for C/C++.
ne75/llvm-project/rust: The llvm branch that contains p2 code and a few things to make rust compile correctly. Use this for rust.
ne75/rust/master: The rust branch that contains the P2 target. Use this to actually build rustc.

I've gone ahead and merged the newer things from ne75/llvm-project/master into ne75/llvm-project/rust, and updated ne75/rust with the latest ne75/llvm-project/rust. Let me know if you are able to build it.

__deets__

Ok, fetching the rust branch, and building everything restored the basic functionality.

The cogstart still does not work. The IR looks good (to my rather uninitiated eye), but there is something odd in the disassembly: the memset became a weird number, #\547. Instead of the existing (and simple "reta") memset.

Find the assorted files attached.

n_ermosh

okay interesting. So it's doing what it's supposed to, but doing it wrong

Basically, special runtime functions (such as memset) are loaded into the LUT on cog start. As a result, they need to be placed at a special memory address so that the linker correctly computes their LUT RAM address, rather than their usual flash memory address. So, if you look at where that memset function is (0xa8c), that linker will convert that to LUT address 0x223 = 547 (which is incorrect, but expected behavior). The compiler expects all of these special functions to be continuous starting at address 0x200 in hub memory, and it copies the 512 longs starting at 0x200 into LUT RAM on cog startup, and the linker converts the byte-address to a long-address during linking for each call to one of these functions. The fact that the memset function is empty is also odd--are you defining an empty one in your source?

Now, for why this is happening in the first place. My best guess is you have an out-of-date libc--I think I actually remember memset not being properly implemented and leading to this exact bug not too long ago. I've attached my very recent build of libc and libp2 libraries so you can try that.

There might be a future development where I make this more flexible to avoid issues like these--it's very opaque currently and without knowing the compiler you'd probably have no way of figuring it out. Maybe have a runtime lookup of where these functions ended up and call them that way instead of having the linker try to figure out what the address will be when the program runs.

__deets__

Oh yes, the memset is in fact empty. I had to define it because Rust doesn’t allow uninitialized arrays. But I know it’s stack, so no need to clear it. That’s why I preliminarily defined it as empty, so that’s correct.

I’ll try your libs soon, I also can also build them myself though - I have your other LLVM project running, to. I just prefer Rust over C++. And why do we need this lib in the first place? Does rust depend on a libc or newlib or something, and that’s what you basically define?

Any hints on how to get deeper into this? I did peek around, but this is a LOT of moving pieces. I’d love to be able to help.

I also wonder why these functions need to be in the LUT. Can’t they be in the main memory, as the rest of the code ? We are using hubexec, don’t we? I

iseries

@deets ,

By putting memcpy and memset into LUT and then using rdfast and wrfast we can make them fly much faster than hub ram.

Mike