and a USB keyboard (via @garryj's single COG USB driver, ported to C by me)..
Nice.
How does the C version compare with the Spin and p2asm versions in garryj's thread.
Should the C version be included in that thread's first post, with notes on Size/speed/ when to choose each version ?
The C version is a mechanical translation using spin2cpp. It's not necessary (at all) with fastspin, but I needed it because I wanted to compile the code with the Risc-V version of gcc. So it's probably not generally useful, although I suppose it might with some tweaking be usable under PropGCC or Catalina.
Ah, ok. Does that mean your version runs the USB code, via the Risc-V emulator code ? I'd have expected that to be too different in timing ?
and a USB keyboard (via @garryj's single COG USB driver, ported to C by me)..
Nice.
How does the C version compare with the Spin and p2asm versions in garryj's thread.
Should the C version be included in that thread's first post, with notes on Size/speed/ when to choose each version ?
The C version is a mechanical translation using spin2cpp. It's not necessary (at all) with fastspin, but I needed it because I wanted to compile the code with the Risc-V version of gcc. So it's probably not generally useful, although I suppose it might with some tweaking be usable under PropGCC or Catalina.
Ah, ok. Does that mean your version runs the USB code, via the Risc-V emulator code ? I'd have expected that to be too different in timing ?
The Spin/C part runs in Risc-V emulation. The PASM part runs in another COG, natively.
Hi Eric
II'm experiencing an issue with your self hosted upython when using paste mode.
After pasting code and hitting ctrl-D it crashes and video is lost too.
I have a test cog blinking an led and its still running aftr the crash.
I can paste the same code into my self hosted upython and all is well.
BTW I have found that the 230400 baud rate is too fast for upython pasting and requires inserting breaks to avoid lost characters. (upython reporst syntax errors)
Hi Eric
II'm experiencing an issue with your self hosted upython when using paste mode.
After pasting code and hitting ctrl-D it crashes and video is lost too.
I have a test cog blinking an led and its still running aftr the crash.
I can paste the same code into my self hosted upython and all is well.
Are you sure you didn't hit ctrl-D twice? I think that shuts down upython. I wasn't able to reproduce this issue, although I've also changed the serial driver a bit (see below).
BTW I have found that the 230400 baud rate is too fast for upython pasting and requires inserting breaks to avoid lost characters. (upython reports syntax errors)
I've added a buffered serial driver that should do better at keeping up with pasted data (at least for the first 512 characters). The updated upython (v10) is in the first post, as usual.
Hi Eric
Sadly it's not a double ctrl-D isuue with V9.
Repeated tests gave same outcome, screen and upython crashes.
V10 behaves differently again and crashes during download.
It appears to be stuck in a buffer loop cycling the same characters.
Here's a test sample that works on my V8 based self hosted upython and on my other micropython boards.
Cheers
Brian
#boyer-moore algorithm test
bad_table = []
def setup(x):
global bad_table
bad_table = [len(x)]*128
for i in range(len(x)):
q = len(x) - i - 1
if q < 1:
q = 1
w = ord(x[i])
print(i,q,w)
bad_table[int(w)] = q
m="the quick brown fox jumps over the lazy dog"
def find(s):
setup(s)
t=0
z=0
while True:
print(m)
print(" "*t,end="")
print(s)
psx = True
for u in range(len(s)-1,-1,-1):
z += 1
if s[u] != m[t+u]:
t += bad_table[ord(m[t+u])]
psx = False
break
if t + len(s) > len(m):
print("No match found!")
break
if psx:
print(z," comparisons made")
break
find("over")
Sorry Brian! There was a typo in the serial receive code that could cause it to get stuck repeating characters. I've uploaded v11 which does run your example correctly. Thanks for the repo case.
I've just posted v12 (see the first post in this thread). This one has preliminary support for microSD cards on the P2 Eval board (i.e. it expects the SD card to be SPI driven on pins 58-61) and FAT file systems. I haven't done a lot of testing yet, but it's worked with three different cards I've tried with.
execfile, open, close, read, and write all seemed to do the right thing. So now with a VGA monitor, SD card, and USB keyboard you can do Python development entirely on your P2 Eval board, and even save your work.
Again, I haven't done a lot of testing so be *very* careful (don't try it with any SD Cards containing data you care about!).
One of the cards wasn't automatically detected, so I had to manually mount it with:
import pyb
import os
sd=pyb.SDCard()
sd.power(1)
os.mount(sd, '/sd')
but once that was done it seemed to work OK.
The updated source code is posted to my github micropython repository.
I haven't had time yet to organize the micropython build environment, but it's basically a standard micropython build using an rv32im RISC-V toolchain and linked with the RISC-V emulator from my github repository (https://github.com/totalspectrum/riscvemu).
I haven't used Mu yet; I had a lot of trouble getting it installed under Linux. I didn't realize that the raw repl would be useful, but it makes sense. I'm preparing a new release now, I'll add it in.
I built the RISC-V toolchain myself from the official repository, but with the latest micropython github you can probably use any common rv32 toolchain, since the emulator (the p2trace one, at least) now supports compressed instructions. You'll also need fastspin to build the JIT compiler from RISC-V to P2 instructions.
I've updated the first post with v13, which now includes:
- raw repl support
- automatically runs "main.py" from the SD card at startup
- better ANSI terminal support in the VGA code (so ANSI escape codes for things like underline and blinking work, as well as the standard 256 ANSI colors)
- a faster RISC-V JIT engine, which also supports compressed code. Even with all the additions (like FAT file system, SD card support, floating point support, raw repl, etc.) we still have 200K of space left for user code.
- debugging hooks added to the RISC-V emulator to make it easy to hook into the I/O
I've updated the first post with v13, which now includes:
- raw repl support
- automatically runs "main.py" from the SD card at startup
- better ANSI terminal support in the VGA code (so ANSI escape codes for things like underline and blinking work, as well as the standard 256 ANSI colors)
- a faster RISC-V JIT engine, which also supports compressed code. Even with all the additions (like FAT file system, SD card support, floating point support, raw repl, etc.) we still have 200K of space left for user code.
- debugging hooks added to the RISC-V emulator to make it easy to hook into the I/O
Wow, I'm sure this impressive progress will have Ken drooling
I should point out that while the current version (v13) is self hosting in theory, and even somewhat in practice (I have saved some text to the SD card and then executed it) it's not user friendly at all yet. We still need a port/write a micropython editor to run on the device; or perhaps @ozpropdev 's P2 assembly editor can be made to work with it.
I haven't used Mu yet; I had a lot of trouble getting it installed under Linux. I didn't realize that the raw repl would be useful, but it makes sense. I'm preparing a new release now, I'll add it in.
I only needed a 'git clone'. After that it was working. I added a P2 mode (basically a copy of the ESP mode) and to connect it to the P2-Eval board it needed to have the baudrate fixed in the code as mu-editor has it fixed to 115200. I opened an issue for that: https://github.com/mu-editor/mu/issues/865
I built the RISC-V toolchain myself from the official repository, but with the latest micropython github you can probably use any common rv32 toolchain, since the emulator (the p2trace one, at least) now supports compressed instructions. You'll also need fastspin to build the JIT compiler from RISC-V to P2 instructions.
I've updated the first post with v13, which now includes:
- raw repl support
- automatically runs "main.py" from the SD card at startup
- better ANSI terminal support in the VGA code (so ANSI escape codes for things like underline and blinking work, as well as the standard 256 ANSI colors)
- a faster RISC-V JIT engine, which also supports compressed code. Even with all the additions (like FAT file system, SD card support, floating point support, raw repl, etc.) we still have 200K of space left for user code.
- debugging hooks added to the RISC-V emulator to make it easy to hook into the I/O
Thanks, you're incredibly responsive and fast.
Update: success Can run code from mu. Still not working is file access but I think that is related to how mu handles the filesystem (started digging there now)
I should point out that while the current version (v13) is self hosting in theory, and even somewhat in practice (I have saved some text to the SD card and then executed it) it's not user friendly at all yet. We still need a port/write a micropython editor to run on the device; or perhaps @ozpropdev 's P2 assembly editor can be made to work with it.
There is this one here: https://github.com/robert-hh/Micropython-Editor . I can not test it is working fro some time as I need to get a VGA and USB connector board soldered up (did not get the accessory boards )
Here's some pics of my self hosted micropython effort.
I'm using one of Tubular's compact dual vga boards with usb.
The monitor on the left is displaying live hub activity.
The Parallax analog joystick controls the zoom window.
The monitor on the right is showing the colorized full editor(written in PASM2).
It's very much a work in progress but has proved the concept very nicely!
Oh, and the first post is updated to include an updated build with a few additional builtin modules (like regex parsing) which the pye.py editor use. pye.py runs now, although using it with the USB keyboard is a bit awkward because the cursor keys don't send the escape sequences that pye.py expects (instead they send the EMACS control keys that micropython uses). I suspect that it would not be hard to fix pye.py to use the EMACS sequences instead, but I'll leave that as an exercise for the reader .
Thanks Eric.
I too wasn't aware of the availability of the source.
I've been hacking your binary (v8) for all my micropython efforts.
Things were getting complex.
With regard to keycodes I ended up coming up with my own custom codes to cover cursor and editing/selection functions for my editor.
I ended up creating a table to convert scancodes (which Garry' s USB driver provides) to ANSI escape sequences, like ESC [ A for cursor up. micropython understands those just fine, as does pye, so it seems like that's probably a better option than what I was doing before (mapping the key presses directly to the EMACS keys that micropython also understands, but which pye handles differently). That hasn't made it to the binary release but it's in github.
There's still a problem with my VGA text driver not handling the ANSI sequences for scrolling regions, so screen redraws can get messed up sometimes in pye. I hope to have that fixed soon.
I've fixed the ANSI escape sequences in the VGA driver, so there's an updated binary release in the first post now. With v15 pye.py works correctly; just put pye.py (included in the zip and in the source repository) on an SD card and do:
from pye import pye
pye('myfile.py')
It works best with a USB keyboard and VGA monitor plugged in, and/or with a terminal emulator that supports ANSI escape sequences and is set up with a 100x40 character window (the same as the VGA display).
Now that it can use a full screen editor which is itself written in python I think "self-hosting" is very much enabled .
I've fixed the ANSI escape sequences in the VGA driver, so there's an updated binary release in the first post now. With v15 pye.py works correctly; just put pye.py (included in the zip and in the source repository) on an SD card and do:
from pye import pye
pye('myfile.py')
It works best with a USB keyboard and VGA monitor plugged in, and/or with a terminal emulator that supports ANSI escape sequences and is set up with a 100x40 character window (the same as the VGA display).
Now that it can use a full screen editor which is itself written in python I think "self-hosting" is very much enabled .
Sounds pretty amazing, Eric.
I imagine the editor was composed in a PC-hosted environment before being run on the P2. Does the conversion to bytecodes happen on the P2, or in the PC environment?
This is looking nice Eric. I had to take out some lines that resulted in "OS not supported" error message, then things work
For those who want to check it out this is what I found I needed to do
## turn off the P123
## plug the VGA monitor into P123 vga board attached to P48~55
## plug USB keyboard to lower connector of P123 "serial host" board attached to P16~23
## rename Eric's v15 .binary to _BOOT_P2.BIX on an SD card
## edit/strip the last function (~15 lines) off pye.py and copy the remainder to pye2.py in root directory of the SD card
## (my pye2.py with lines already stripped off attached to this post, in zip file)
## turn off P123 flash switch (in case of difficulties)
## power up and wait for micropython prompt to appear on VGA screen (few seconds)
## type the following commands into the micropython repl
import pyb
sd=pyb.SDCard()
sd.power(1)
## repl will echo 'True'
import os
os.mount(sd,'/sd')
os.chdir('/sd')
execfile("pye2.py")
from pye import pye
pye('pye2.py')
## that should bring up the editor and move around the ~970 line pye2 source.
## control-k-s to save back to SD card
## control-k-q to quit back to repl
Now that it can use a full screen editor which is itself written in python I think "self-hosting" is very much enabled .
Sounds pretty amazing, Eric.
I imagine the editor was composed in a PC-hosted environment before being run on the P2. Does the conversion to bytecodes happen on the P2, or in the PC environment?
The editor is one that someone else wrote for micropython. But you can edit and change it on the P2. The micropython binary I posted has the bytecode compiler built in, so all Python code is compiled on the P2. I think it's possible to pre-compile bytecodes, but I haven't really tried that and there's not much need (the compiler is fast enough). I guess it might be interesting for a really stripped down version of micropython, but then you'd lose the ability to type commands interactively.
I hadn't thought of booting micropython from the SD-card, thanks for trying that out!
Interesting that you had to remove that bit at the end that tests for the OS. I actually hadn't tried "execfile", I'd just been using "from pye import pye"; I guess the OS check only happens in execfile. But thanks for the fix, I'll update my version of pye.py accordingly.
It's a bummer that you have to manually mount the SD card. Any ideas of a good way to detect the presence of a card? I've been checking for pin 59 low, but it appears not all SD cards make this happen.
Comments
II'm experiencing an issue with your self hosted upython when using paste mode.
After pasting code and hitting ctrl-D it crashes and video is lost too.
I have a test cog blinking an led and its still running aftr the crash.
I can paste the same code into my self hosted upython and all is well.
BTW I have found that the 230400 baud rate is too fast for upython pasting and requires inserting breaks to avoid lost characters. (upython reporst syntax errors)
I've added a buffered serial driver that should do better at keeping up with pasted data (at least for the first 512 characters). The updated upython (v10) is in the first post, as usual.
Thanks,
Eric
Sadly it's not a double ctrl-D isuue with V9.
Repeated tests gave same outcome, screen and upython crashes.
V10 behaves differently again and crashes during download.
It appears to be stuck in a buffer loop cycling the same characters.
Here's a test sample that works on my V8 based self hosted upython and on my other micropython boards.
Cheers
Brian
#boyer-moore algorithm test bad_table = [] def setup(x): global bad_table bad_table = [len(x)]*128 for i in range(len(x)): q = len(x) - i - 1 if q < 1: q = 1 w = ord(x[i]) print(i,q,w) bad_table[int(w)] = q m="the quick brown fox jumps over the lazy dog" def find(s): setup(s) t=0 z=0 while True: print(m) print(" "*t,end="") print(s) psx = True for u in range(len(s)-1,-1,-1): z += 1 if s[u] != m[t+u]: t += bad_table[ord(m[t+u])] psx = False break if t + len(s) > len(m): print("No match found!") break if psx: print(z," comparisons made") break find("over")Eric
V11 looks good, nice work! :cool:
execfile, open, close, read, and write all seemed to do the right thing. So now with a VGA monitor, SD card, and USB keyboard you can do Python development entirely on your P2 Eval board, and even save your work.
Again, I haven't done a lot of testing so be *very* careful (don't try it with any SD Cards containing data you care about!).
One of the cards wasn't automatically detected, so I had to manually mount it with: but once that was done it seemed to work OK.
The updated source code is posted to my github micropython repository.
Eric
Trying to get mu-editor up and running and succeeded to some extend:
* can open a serial terminal (REPL)
(grr, can not upload a screenshot
what is not working is
* running a script or
* accessing the SD card on the P2-eval board.
The last requires raw_repl to be working, which does not seem to be activated in the current P2 micropython build.
Question: how can we build micropython, ie what other tools are needed?
Edit:
Let's see if I can get that working
I built the RISC-V toolchain myself from the official repository, but with the latest micropython github you can probably use any common rv32 toolchain, since the emulator (the p2trace one, at least) now supports compressed instructions. You'll also need fastspin to build the JIT compiler from RISC-V to P2 instructions.
- raw repl support
- automatically runs "main.py" from the SD card at startup
- better ANSI terminal support in the VGA code (so ANSI escape codes for things like underline and blinking work, as well as the standard 256 ANSI colors)
- a faster RISC-V JIT engine, which also supports compressed code. Even with all the additions (like FAT file system, SD card support, floating point support, raw repl, etc.) we still have 200K of space left for user code.
- debugging hooks added to the RISC-V emulator to make it easy to hook into the I/O
Wow, I'm sure this impressive progress will have Ken drooling
Self hosting is a deal breaker
I'll try later today.
Thanks, you're incredibly responsive and fast.
Update: success
There is this one here: https://github.com/robert-hh/Micropython-Editor . I can not test it is working fro some time as I need to get a VGA and USB connector board soldered up (did not get the accessory boards
I'm using one of Tubular's compact dual vga boards with usb.
The monitor on the left is displaying live hub activity.
The Parallax analog joystick controls the zoom window.
The monitor on the right is showing the colorized full editor(written in PASM2).
It's very much a work in progress but has proved the concept very nicely!
https://github.com/totalspectrum/micropython
The port is in the ports/riscv-p2 directory. Directions for obtaining the toolchain are at https://github.com/totalspectrum/riscvp2.
Sorry about that, I had a distinct memory of posting that link -- it must have been in some other thread.
I remember OzProp saying he had to do something with scancodes, but think it was function keys rather than cursor keys
I too wasn't aware of the availability of the source.
I've been hacking your binary (v8) for all my micropython efforts.
Things were getting complex.
With regard to keycodes I ended up coming up with my own custom codes to cover cursor and editing/selection functions for my editor.
There's still a problem with my VGA text driver not handling the ANSI sequences for scrolling regions, so screen redraws can get messed up sometimes in pye. I hope to have that fixed soon.
from pye import pye pye('myfile.py')It works best with a USB keyboard and VGA monitor plugged in, and/or with a terminal emulator that supports ANSI escape sequences and is set up with a 100x40 character window (the same as the VGA display).Now that it can use a full screen editor which is itself written in python I think "self-hosting" is very much enabled
Self-hosting will be a great bonus for when the real silicon arrives
Sounds pretty amazing, Eric.
I imagine the editor was composed in a PC-hosted environment before being run on the P2. Does the conversion to bytecodes happen on the P2, or in the PC environment?
For those who want to check it out this is what I found I needed to do
## turn off the P123
## plug the VGA monitor into P123 vga board attached to P48~55
## plug USB keyboard to lower connector of P123 "serial host" board attached to P16~23
## rename Eric's v15 .binary to _BOOT_P2.BIX on an SD card
## edit/strip the last function (~15 lines) off pye.py and copy the remainder to pye2.py in root directory of the SD card
## (my pye2.py with lines already stripped off attached to this post, in zip file)
## turn off P123 flash switch (in case of difficulties)
## power up and wait for micropython prompt to appear on VGA screen (few seconds)
## type the following commands into the micropython repl
import pyb
sd=pyb.SDCard()
sd.power(1)
## repl will echo 'True'
import os
os.mount(sd,'/sd')
os.chdir('/sd')
execfile("pye2.py")
from pye import pye
pye('pye2.py')
## that should bring up the editor and move around the ~970 line pye2 source.
## control-k-s to save back to SD card
## control-k-q to quit back to repl
The editor is one that someone else wrote for micropython. But you can edit and change it on the P2. The micropython binary I posted has the bytecode compiler built in, so all Python code is compiled on the P2. I think it's possible to pre-compile bytecodes, but I haven't really tried that and there's not much need (the compiler is fast enough). I guess it might be interesting for a really stripped down version of micropython, but then you'd lose the ability to type commands interactively.
I hadn't thought of booting micropython from the SD-card, thanks for trying that out!
Interesting that you had to remove that bit at the end that tests for the OS. I actually hadn't tried "execfile", I'd just been using "from pye import pye"; I guess the OS check only happens in execfile. But thanks for the fix, I'll update my version of pye.py accordingly.
It's a bummer that you have to manually mount the SD card. Any ideas of a good way to detect the presence of a card? I've been checking for pin 59 low, but it appears not all SD cards make this happen.