Host (PC) file system access
ersmith
Posts: 6,053
Here is an object to allow the P2 to read/write files on the host PC, using the 9P protocol. This code should compile with any Spin2 compiler (I have tested with flexspin and PNut). FlexProp users will be familiar with the 9P file server, as we have had access to this from C and BASIC for a long time. This object lets Spin2 users also have access.
The main object is hostfs.spin2. It is based on the Parallax flash file system, and has a very similar interface. On the P2 side you mount the file system, passing in a method that says how to communicate with the PC (samples are provided for serial communication, but in theory ethernet or some other transport could be used). Then you can open files and read/write using POSIX like calls.
On the PC side you need to run a file server. Loadp2 (which comes with flexprop) has a built in serial file server, as does my port of proploader. To use this do something like:
loadp2 -b2000000 myprogram.bin -9.
Here -b2000000 specifies the serial baud rate, and -9. specifies the directory to be served to the P2. -9. means to expose the current directory to the P2; you could also do something like -9C:\ to give access to all of the C: drive, although that's probably not a great idea.
EDIT: updated the version to 102 to reflect some fixed bugs.
Comments
Nice. Completes the set.
Hi @ersmith
I just tried your 9P file demo out to save me from writing logs to SD card each time I do a video capture etc but I have run into a problem with your demo posted above. For some reason the file created seems to have bad file permissions on the host (hello.txt file gets created with execute only, see below). I had to manually change its permissions before the host would let me see it and the demo itself failed could not read back the file. Am using a Mac Mini ARM64 running OS X Ventura.
Do you know why this should be happening/how to fix?
Thanks,
Roger.
❯ loadp2 -b 2000000 -t -9. demo.binary
9P file system test
Mount succeeded!
u9fs: badly sized message type 30: expected 30 got 0
last unix error: Bad file descriptor
open returned -12
open: error -12
❯ ls
demo.binary ers_fmt.spin2 hostfs.spin2 jm_nstr.spin2 README.md
demo.p2asm hello.txt hostfs_doc.txt jm_parser.spin2 serial_9p.spin2
demo.spin2 host_file_demo.spin2 jm_fullduplexserial.spin2 LICENSE serial_sendrecv.spin2
❯ more hello.txt
hello.txt: Permission denied
❯ ls -l hello.txt
.--------x roger staff 15 B Mon Sep 23 15:21:26 2024 hello.txt
❯ chmod +r hello.txt
❯ ls -l
.rw-r--r-- roger staff 12 KB Mon Sep 23 15:20:45 2024 demo.binary
.rw-r--r-- roger staff 43 KB Mon Sep 23 15:20:45 2024 demo.p2asm
.rw-r--r-- roger staff 1.3 KB Tue Oct 17 20:57:54 2023 demo.spin2
.rw-r--r-- roger staff 3.8 KB Tue Oct 3 17:18:58 2023 ers_fmt.spin2
.r--r--r-x roger staff 15 B Mon Sep 23 15:21:26 2024 hello.txt
.rw-r--r-- roger staff 21 KB Tue Oct 17 21:01:14 2023 host_file_demo.spin2
.rw-r--r-- roger staff 31 KB Mon Oct 16 17:27:54 2023 hostfs.spin2
.rw-r--r-- roger staff 12 KB Wed Oct 18 20:29:38 2023 hostfs_doc.txt
.rw-r--r-- roger staff 22 KB Sat Sep 30 13:24:04 2023 jm_fullduplexserial.spin2
.rw-r--r-- roger staff 8.8 KB Sat Sep 30 13:24:04 2023 jm_nstr.spin2
.rw-r--r-- roger staff 10 KB Sun Oct 1 17:10:12 2023 jm_parser.spin2
.rw-r--r-- roger staff 1.3 KB Wed Oct 18 20:14:04 2023 LICENSE
.rw-r--r-- roger staff 6.0 KB Sat Nov 4 08:44:32 2023 README.md
.rw-r--r-- roger staff 3.9 KB Sun Oct 15 10:31:16 2023 serial_9p.spin2
.rw-r--r-- roger staff 1.5 KB Tue Oct 17 20:50:40 2023 serial_sendrecv.spin2
❯ cat hello.txt
Hello, world!
❯
EDIT: also found I'm running a reasonably old loadp2 in case that may relate to this issue. I will try to see if I can find a newer version and retest this.
loadp2 - a loader for the propeller 2 - version 0.057 Apr 16 2023
EDIT2: No, just rebuilt the latest flexprop loadp2 submodule master branch and got the same result with this version of loadp2
loadp2 - a loader for the propeller 2 - version 0.076 Sep 23 2024
@rogloh Looks like there were some serious typos in the version of the HostFs I uploaded. I guess nobody is using it
. Here's a revised version. WIth this one the demo.spin2 works as expected. The interactive demo (host_file_demo.spin2) mostly works but has trouble writing to files, I haven't been able to track down the bug yet.
Thanks Eric, will take a look soon at this to see if it helps and report back in this post.
Update: fixed, see below.
Hi Eric(@ersmith)
Don't get discouraged that people here haven't immediately started using this.
If they are like me they are concentrating on other matters but keeping an eye on what you and others are working on.
I know I certainly am as it could be handy at some future time, thanks for your efforts.
I am not clear on how to use this on a Windows PC, do I need to use Linux subsystem for Windows?
I assumed the following was Linux when I first read it, not so sure now, does this work on Windows too?
It seems to be using the "." to specify the current directory like in DOS?
So that would share the directory the "server" is able to access and read / write files from?
Is it possible for windows to write to this file and then for the P2 to receive that information and visa versa?
So what is "myprogram.bin" exactly?
I noticed that @rogloh used this in his comment
I might need to refer to the "loadp2" docs because there is now a "-t" switch that you don't mention and the order of the switches is different,
I am not too clear on how to use this at this stage but I can see the promise.
Well, I first posted it almost a year ago
. I think most people have just been using the C library version of the host file system.
No. You can run loadp2 directly from the Windows command line. Or you can use the FlexProp GUI, which will run it for you (and has a menu option for selecting the directory for the server to read from).
Yes.
Yes.
It's the P2 compiled binary (output of PNut or flexspin).
The order of the switches generally doesn't matter. "-t" means to enter terminal mode. It's implied by "-9", so it's not really needed.
Regards,
Eric
@ersmith , I just quickly tried your updated code and the demo seems to work now on my Mac Mini M2 machine. The permissions are also okay too. Thanks for fixing it.
❯ l -t -b2000000 -9. demo.binary
( Entering terminal mode. Press Ctrl-] or Ctrl-Z to exit. )
9P file system test
Mount succeeded!
open returned 0
read: returned 15 bytes
buffer contents: Hello, world!
❯ ls -l
.rw-r--r-- roger staff 12 KB Fri Sep 27 15:25:23 2024 demo.binary
.rw-r--r-- roger staff 43 KB Fri Sep 27 15:25:23 2024 demo.p2asm
.rw-r--r-- roger staff 1.3 KB Tue Oct 17 20:57:54 2023 demo.spin2
.rw-r--r-- roger staff 3.8 KB Tue Oct 3 17:18:58 2023 ers_fmt.spin2
.rw------- roger staff 15 B Fri Sep 27 15:26:16 2024 hello.txt
.rw-r--r-- roger staff 21 KB Wed Sep 25 21:40:38 2024 host_file_demo.spin2
.rw-r--r-- roger staff 31 KB Wed Sep 25 21:39:18 2024 hostfs.spin2
.rw-r--r-- roger staff 12 KB Wed Oct 18 20:29:38 2023 hostfs_doc.txt
.rw-r--r-- roger staff 22 KB Sat Sep 30 13:24:04 2023 jm_fullduplexserial.spin2
.rw-r--r-- roger staff 8.8 KB Sat Sep 30 13:24:04 2023 jm_nstr.spin2
.rw-r--r-- roger staff 10 KB Sun Oct 1 17:10:12 2023 jm_parser.spin2
.rw-r--r-- roger staff 1.3 KB Wed Oct 18 20:14:04 2023 LICENSE
.rw-r--r-- roger staff 6.0 KB Sat Nov 4 08:44:32 2023 README.md
.rw-r--r-- roger staff 3.9 KB Sun Oct 15 10:31:16 2023 serial_9p.spin2
.rw-r--r-- roger staff 1.5 KB Tue Oct 17 20:50:40 2023 serial_sendrecv.spin2
Unfortunately, despite the demo above working, one thing I've not been able to do is to write more than 512 bytes in a file. After the initial write I just get E_IO errors on the next ones. Maybe it's the same bug you have not been able to track down.
' first write some data handle := fs.open(@"hello.txt", "w") if handle < 0 send("open returned ", fmt.dec(handle)) return fs.write(handle, @msg, strsize(@msg)) repeat r from 1 to 1*1024*2 '1MB <--- loop and write in bursts of 512 bytes. fs.write(handle, @msg, 512) if r//1000 == 0 send(".") fs.close(handle)Result is file size of only 512 bytes. It's like that value is special - even though the original 15 bytes written is being added to the output file size is just truncated at 512 bytes total. Might be a hint to the problem?
Actually what's wrong is that it keeps re-writing the bytes at the same offset of the file, so the file ends up containing only the last 512 bytes written (in your example). This was due to a missing line in the write() method. A fixed version is attached.
Thanks,
Eric
Great, seems to be writing large files now. Cheers for that.
I timed it, certainly not the most efficient way to transfer bulk data. On a 2Mbps serial port it gets about 220kbps so just around 11% actual throughput when I timed it with the snippet below. This is probably more useful for small file writes/transfers, but it is certainly convenient to have real file access on the host from the P2. I also ran without the dots printing each 64kB thinking that might possibly have somehow interfered a little and slowed it down, but at that frequency of printing it didn't make any real difference to the throughput.
The serial USB client stuff Chris Gadd did might be the best focus for me now. I'd like to transfer files about 4-5MB or so and still be more convenient than repeatedly swapping uSD cards from P2-EVAL in and out (which will eventually break something IMO).
' first write some data handle := fs.open(@"hello.txt", "w") if handle < 0 send("open returned ", fmt.dec(handle)) return fs.write(handle, @msg, strsize(@msg)) t1:=getsec() repeat r from 1 to 1024*2 fs.write(handle, @msg, 512) if r//128 == 0 send(".") fs.close(handle) t2:=getsec() send(13,10, "Time taken for 1 Megabyte transfer to host = ",fmt.dec(t2-t1)," seconds, rate = ",fmt.dec(1024*1024*8/(t2-t1))," bps",13,10)❯ l -t -b 2000000 -9. demo.binary
( Entering terminal mode. Press Ctrl-] or Ctrl-Z to exit. )
9P file system test
Mount succeeded!
................
Time taken for 1 Megabyte transfer to host = 38 seconds, rate = 220752 bps
open returned 0
read: returned 100 bytes
buffer contents: Hello, world!
Hello, world!
@rogloh I think 512 byte sized buffers are on the small side -- you may get better throughput with larger ones. With a 1K buffer it goes about twice as fast. Still not great though, I agree. You can get a further speedup by increasing the maximum transfer size in hostfs.c. Loadp2 can handle up to 8K buffers (not sure why that restriction is there, to be honest). I've attached a version 3 here which with 4K block size which gets 932067 bps at 2Mbaud, which is at least respectable.
Yeah 933067 is more like it and over a 4x speedup from my result. Remember that with start/stop bits the peak will only ever get to 1.6Mbps out of 2Mbps anyway. I'll have another go when I can.
Looking at the P2 write code I'm a little surprised it needs to send 4k blocks to gain that amount performance. This means if I have my math right, on a 180MHz P2 each write is still burning approximately (1.6-0.933)/0.933 * 40960/2Mbps = 14.6ms of time between bursts. That is a lot of P2 clocks @180MHz for the little overhead work being done framing the packet. It must be spending most of the time just waiting for the host to reply with the 32 bit long. The host seems slow if it takes 14.6ms to reply with the long when running on a modern PC/Mac/Linux system. Maybe this plan9 thing on the host is writing character by character or something dumb.
I took a quick look in loadp2 plan9 code which seems to be doing a lot of serial message delineation decode stuff with timeouts on reads. There was a timeout value of 1000 buried in there (1 sec), not sure, maybe it's set too slow to detect end of frame of something like that.
Update: Also if I'm not misinterpreting the u9fs code (which I could well be) then the response to the write call seems to send back as much data as gets written to disk??? I am gonna hook up my logic analyzer to the serial pins and look at that.
No, it seems to send just the short acks, but does show a 18.6ms gap between acks. There's a long delay at the receiver in loadp2/u9fs somewhere. You can see that after the ack reply from the host the P2 is ready to send almost immediately. This was still with 512 byte transfers.
If I change the size to 4k it does improve as expected but we still see a gap (this time 36ms or 2x what we had before). I'm wondering if it's some sort of tx data buffer pacing issue on the host. A tx flush operation after the ack message goes out may improve this...will try that in a bit...
No combination of tcflush, tcdrain or fsync after a Twrite message was processed seemed to make any difference. It's like something is limiting the rate of transmit messages going back out to the P2 from the host to happen only on 18ms boundaries or some such thing.
UPDATE: Nope.
UPDATE2: When I tried out other buffer sizes it looks like it's actually always delayed by 16ms from the end of the packet to the reply (constant time) so the fact that it looked like 2x on 18ms boundaries before was just a coincidence.
❯ l -t -b 2000000 -9. demo.binary
( Entering terminal mode. Press Ctrl-] or Ctrl-Z to exit. )
9P file system test
Mount succeeded!
Press a key to start...
................
Time taken for 1 Megabyte transfer to host = 10 seconds, rate = 838860 bps
open returned 0
read: returned 100 bytes
buffer contents: Hello, world!
Hello, world!
Just read this, another coincidence with this 16 ms value perhaps...
Ok so I found the cause. Once I reduced the FTDI latency timer to 2ms using the code linked below I increased the performance significantly. Getting 1.42Mbps now with FTDI latency=2ms and 1.47Mbps with latency=1ms.
❯ ~/Applications/flexprop/loadp2/build/loadp2 -t -9. -b 2000000 demo.binary
( Entering terminal mode. Press Ctrl-] or Ctrl-Z to exit. )
9P file system test
Mount succeeded!
Press a key to start...
................
Time taken for 1 Megabyte transfer to host = 5.901 seconds, rate = 1421797 bps
open returned 0
read: returned 100 bytes
buffer contents: Hello, world!
Hello, world!
See this link for details...
https://openbci.com/forum/index.php?p=/discussion/3108/driver-latency-timer-fix-for-macos-11-m1-m2
Source code for building a tool for setting FTDI latency on Linux/MacOS is here:
https://gist.github.com/mkeeter/c43c3990ecdb8dcb6547ac3dbac8e881
Windows users can apparently set the FTDI latency timer in some control panel setting.
You can see the delay between end of received write command to sending the ack reply is now much shorter and performance is increased. This was with 4k transfers and latency =2ms.
@rogloh : Interesting. I wonder why FTDI chose such a relatively large latency for a default? Seems like it could cause problems in a lot of applications. Anyway, thanks for looking in to this!
No worries Eric. Yeah this is probably a tradeoff FTDI made between performance of the serial port vs performance of an old (slow) PC or other USB capable device when USB interrupts are firing off every 1ms or so running their driver. These days I wouldn't expect it to slow a PC down that much...I suspect it should be DMA'd directly into memory and barely affects the host, though that probably depends on the controller.
UPDATE: also found this document which sheds more light...
https://www.ftdichip.com/Documents/AppNotes/AN232B-04_DataLatencyFlow.pdf
One final data point, with the BAUD rate set to 3000000 and the latency timer set to 1 and 8kB transfer bursts and the P2 running at 300MHz I was able to push to it achieve this which is decent:
2.2075/2.4 is ~92% throughput on a 3Mbps serial link.