new loadp2, now with simple scripting

ersmith · 2019-11-30 22:05

EDIT 2019-12-05: Updated scripting language completely for v033.

Here's a version of loadp2 which adds a new option, -e, for running a simple script after the P2 has been loaded. It's mainly designed for automating interaction with interpreters like TAQOZ or microPython. It's kind of a rough and ready solution, but I think it will be useful.

The script must be specified as the argument after -e, and consists of various commands to send characters or insert pauses. The commands are separated by linespace (including newlines) and have parameters that are enclosed either in () or {} (so "send(hello)" and "send{hello}" both are valid ways to send the string "hello"). The {} form is useful when you want to send ")" in the string.

Within the sent string, a caret (^) introduces an escape. ^ + digits sends a literal ASCII character (so ^13 sends a carriage return). ^ + a letter sends a control character (so ^M also sends a carriage return). ^^ sends a "^" character.

Available commands are:

binfile(fname):    send a binary file to the P2
pauseafter(N):     insert a 1ms pause after every N characters transmitted
pausems(N):        delay for N milliseconds
recv(string):      wait until string is received
recvtimeout(N):    set a timeout in ms for the recv() command
scriptfile(fname): read script commands from file "fname"
send(string):      send a string to the P2
textfile(fname):   send contents of a text file to the P2

Example: to add two numbers using TAQOZ in ROM:

loadp2 -b230400 -xTAQOZ -e "recv(TAQOZ# ) send{2 2 + . ^M}" -t

The script waits for the P2 to send the string "TAQOZ#", then sends "2 2 + ." and a carriage return (ctrl-M).

Example: to start up micropython and have it run a script "myfile.py" you could do:

loadp2 -b230400 upython.binary -e "recv(>>> ) send(^E) textfile{myfile.py} send(^D)" -t

That script waits for the P2 to send the >>> sequence which is its prompt, then sends a ctrl-E (to enter paste mode), then sends the contents of the file "myfile.py", then sends a ctrl-D (to exit paste mode).

There's not much error checking yet and I'm sure there is much that could be improved, but please give it a try.

As you can probably guess, my intention is to use this feature in the next version of FlexGUI to make it easier to run Python, Forth, and Lisp programs on the P2.

The source code is checked into github. In the attached .zip there are loadp2.exe, loadp2.linux, and loadp2.mac executable binary files. I think you can guess which is for which platform

.

jmg · 2019-11-30 22:19

That's nifty.
Can the delay time be made variable ?
A means to toggle DTR & RTS could be useful ? (in UB3, RTS reloads the Si5351A, and currently increments the upper control reply bits)

Can the script be contained in a file ? I see a send-file, but not a run file as script ?

ersmith · 2019-11-30 22:32

jmg wrote: »

That's nifty.
Can the delay time be made variable ?

You can use more than one ^#

. But you're right, allowing an optional integer parameter to specify the delay time would probably be useful, as would a way to specify the timeout for the read options.

A means to toggle DTR & RTS could be useful ? (in UB3, RTS reloads the Si5351A, and currently increments the upper control reply bits)

That'll reset the chip on most P2 boards, so sounds dangerous.

Can the script be contained in a file ? I see a send-file, but not a run file as script ?

No, for now the script is self contained. I guess having an option to run it from a file would be feasible. I'm not trying to solve every scripting problem here, just providing a simple way to get scripts to run in TAQOZ, micropython, and proplisp.

Thanks for the suggestions!

rjo__ · 2019-11-30 23:00

This is slightly related... can micropython be invoked from a running program that isn't using cog 0?

rjo__ · 2019-11-30 23:01

or... can you load micropython bnormally without disturbing a running program that isn't using cog 0

rogloh · 2019-11-30 23:20

Scripting sounds handy Eric. I am thinking it would be great if loadp2 evolves in a way that still supports a future extension to allow network based downloads with those cheap wifi-serial adapters like esp8266 and Parallax's Wifi board. I've not looked at the source of loadp2 in a while but hopefully all these extra handy software features you have been adding won't make it too difficult for us to still do this in the future rather than need to diverge or to invent another tool altogether. Being cross platform already, loadp2 is a tool of choice for the P2. It's essential really.

ersmith · 2019-11-30 23:33

rjo__ wrote: »

or... can you load micropython bnormally without disturbing a running program that isn't using cog 0

micropython takes up most of the P2, although my version does leave the debug area at the top of HUB ram free. Invoking it from another COG is probably possible. ozpropdev did some interesting things with running other programs alongside micropython.

ersmith · 2019-11-30 23:36

rogloh wrote: »

Scripting sounds handy Eric. I am thinking it would be great if loadp2 evolves in a way that still supports a future extension to allow network based downloads with those cheap wifi-serial adapters like esp8266 and Parallax's Wifi board. I've not looked at the source of loadp2 in a while but hopefully all these extra handy software features you have been adding won't make it too difficult for us to still do this in the future rather than need to diverge or to invent another tool altogether. Being cross platform already, loadp2 is a tool of choice for the P2. It's essential really.

My changes have mostly been in the platform independent code, and don't generally care what kind of transport is used underneath. I don't think it would be difficult to put all of this over a network instead of the serial layer, but obviously there would be some work involved on both the hardware and software sides.

jmg · 2019-11-30 23:41

ersmith wrote: »

That'll reset the chip on most P2 boards, so sounds dangerous.

Of course, it needs user care, just like them toggling it from a terminal screen...

Can the script be contained in a file ? I see a send-file, but not a run file as script ?

No, for now the script is self contained. I guess having an option to run it from a file would be feasible. I'm not trying to solve every scripting problem here, just providing a simple way to get scripts to run in TAQOZ, micropython, and proplisp.

When testing using TAQOZ I found it very useful to save such script files, as easy to read names, (done for the terminal pgm I use), and those include variable delays.
The flip to TAQOZ takes a finite amount of time
Here is an example of one of those tests, shows how useful terminal scripts can be...
{// is a comment, and {250} is a delay of 250ms {5} is a delay of 5ms strings inside quotes send directly, $ is hex prefix.

{// Terminal Command File ET_Pin16_PWM_Xtal_TAQOZ.TXT}
{// RCFAST  ~ 23.0592MHz }
{// Autobaud '> ', Enter TAQOZ from reset(esc), ie $3E,$20,$1B,  then pause >195ms(115200), then Set pin16 to 10kHz RCFAST }
{//   Pin2 generate a 10% PWM with a resolution of 1,000 counts with minimum clock division. TAQOZ# 2 PIN 100 1000 1 PWM  ok}
{(// 1 give 8.610u, 3 gives 26.08u, but indicates sysslk of 11.5MHz, half expected 23MHz - RCFAST shows 680ns jitter in 867us or 783ppm just under 0.1% }
$3E,$20,$1B
{250}
'16 PIN 200 10000 1 SAW',$0D
{5}
'%10_00 HUBSET 62,000 WAITX %10_10 HUBSET',$0D

Some named files like this, would give users a quick series of board check operations than can do.

Cluso99 · 2019-12-01 02:01

Eric,
I copied the new loadp2 into the flexgui bin directory.
I also copied upython.py from \samples\upython\ to here as well, plus I added myfile.py which is just a single line "print('hello')"

I am on W10 and here is what I get from the command line...

C:\P2\flexgui 4.0.4\bin>loadp2 -b230400 upython.binary -e '^/>>>/^E^{myfile.py}^D' -t
> was unexpected at this time.

and...
c:\P2\flexgui 4.0.4\bin>loadp2 -b230400 upython.binary -e '^/>>>/' -t
> was unexpected at this time.

this works...
c:\P2\flexgui 4.0.4\bin>loadp2 -b230400 upython.binary -t
( Entering terminal mode.  Press Ctrl-] to exit. )
started USB on cog 2
MicroPython eric_v19 on 2019-11-05; P2-Eval-Board with p2-cpu
Type "help()" for more information.
>>>

What did I break?

This is brilliant

ersmith · 2019-12-01 02:17

Sorry, I did my testing with Linux, where single quotes work. Microsoft's cmd.exe apparently only understands double quotes. Try changing the single quotes in those examples to double quotes.

Also, you probably don't strictly have to wait for the >>> prompt, there is some buffering going on and I think you could leave off the "^/>>>/" part of the script (that's the script command which waits for the prompt).

Cluso99 · 2019-12-01 02:47

Thanks Eric. Should have guessed that

BTW I note when you exit from the >>> prompt in the command line, the terminal is left in an unusable state which means I cannot now do any more loadp2, etc. Is this fixable?

Currently I am trying to run this but I get caught up in what I think is loadp2 resetting the P2 and I have an SD Card in the slot causing the ROM to try and boot the P2. Have to step out so I will try this again when I get home.

import pyb
import os
def padfname(f):
    lst=f.split('.')
    if len(lst)>1:
        s=(lst[0]+'        ')[:8]+'.'+(lst[1]+'     ')[:5]
    else:
        s=(f+'              ')[:14]
    return(s)
def ls(lst):
    for i in range(0,len(lst),4):
        s=padfname(lst[i])
        if i+1<len(lst):
            s=s+padfname(lst[i+1])
        if i+2<len(lst):
            s=s+padfname(lst[i+2])
        if i+3<len(lst):
            s=s+padfname(lst[i+3])
        print(s)
sd=pyb.SDCard()
sd.present()
os.mount(sd,'/sd')
os.chdir('/sd')
lst=os.listdir()
ls(lst)

Cluso99 · 2019-12-01 07:14

Just arrived home and back to it again...

Usually the loadp2 times out with no >>> received.

When it works, I get the os.mount(sd,'/sd') failing as below (note the sd.present() works fine).

c:\P2\flexgui 4.0.4\bin>loadp2 -b230400 upython.binary -e "^/>>>/^E^{ls.py}^D" -t
( Entering terminal mode.  Press Ctrl-] to exit. )

paste mode; Ctrl-C to cancel, Ctrl-D to finish
=== import pyb
=== import os
=== sd=pyb.SDCard()
=== if sd.present():
===     print('SD present')
===     os.mount(sd,'/sd')
=== else:
===     print('SD Mount failure')
===
SD present
Traceback (most recent call last):
  File "<stdin>", line 6, in <module>
OSError: 19
>>>

ersmith · 2019-12-01 19:20

Cluso99 wrote: »

Just arrived home and back to it again...

Usually the loadp2 times out with no >>> received.

Does the script work despite that? It should still send the file contents, and if the file isn't too long it'll be buffered on the P2 and delivered to micropython.

When it works, I get the os.mount(sd,'/sd') failing as below (note the sd.present() works fine).

Is the sd already mounted? I have noticed that os.mount(sd, '/sd') causes a whole lot of grief if /sd is already mounted, but if I do os.mount(sd, '/mysd') then micropython is happy and actually creates two different views of the SD card (/sd and /mysd).

After the script times out you should still be able to type commands at micropython. Is the ls function defined?

ersmith · 2019-12-01 19:23

I've been thinking about this and it seems like I may be going down the wrong path with this scripting language idea. Not only is it not working very well, but it also is a one time only opportunity (we can send a file at the initial load time, but not at any later time).

I think I need to step back and look at this some more. Perhaps a better approach would be to (optionally) have loadp2 listen on a socket for commands of things to send to the terminal and/or things to do such as resetting the board. Then it would be easy to have GUIs send scripts to the P2 at any point (as long as loadp2 is still running).

Cluso99 · 2019-12-01 19:45

Eric,
I haven’t tried entering any python after the failure. But I don’t think it’s already mounted as I retried this a number of times with everything starting from scratch and powering up the P2. Once I power the P2 I insert the SD card and then run the loadp2 command.

As to whether it’s the right approach, I don’t know.

What I would like is to launch python from the P2 from my OS ie run the upython binary from the P2. But this requires my P2 OS and I haven’t had time to get this running by recompiling the P1 spin code to p2asm yet. I’ve been hoping Chip might release a limited version of spin2 as it would be so much easier.

ersmith · 2019-12-01 20:16

Cluso99 wrote: »

What I would like is to launch python from the P2 from my OS ie run the upython binary from the P2. But this requires my P2 OS and I haven’t had time to get this running by recompiling the P1 spin code to p2asm yet. I’ve been hoping Chip might release a limited version of spin2 as it would be so much easier.

Have you tried fastspin? It does support compiling Spin1 on the P2.

David Betz · 2019-12-01 20:22

ersmith wrote: »

Cluso99 wrote: »

What I would like is to launch python from the P2 from my OS ie run the upython binary from the P2. But this requires my P2 OS and I haven’t had time to get this running by recompiling the P1 spin code to p2asm yet. I’ve been hoping Chip might release a limited version of spin2 as it would be so much easier.

Have you tried fastspin? It does support compiling Spin1 on the P2.

Compile Spin1 "on" the P2? I didn't realize you could compile directly on the P2. Or did you mean to say "compile Spin1 *for* the P2"?

ersmith · 2019-12-02 00:29

David Betz wrote: »

ersmith wrote: »

Cluso99 wrote: »

What I would like is to launch python from the P2 from my OS ie run the upython binary from the P2. But this requires my P2 OS and I haven’t had time to get this running by recompiling the P1 spin code to p2asm yet. I’ve been hoping Chip might release a limited version of spin2 as it would be so much easier.

Have you tried fastspin? It does support compiling Spin1 on the P2.

Compile Spin1 "on" the P2? I didn't realize you could compile directly on the P2. Or did you mean to say "compile Spin1 *for* the P2"?

Sorry, yes, "for" the P2. And maybe @Cluso99 is waiting for Chip's Spin2 so he can compile "on" the P2 using his OS? But I don't think the initial Spin2 release will support that.

It'll certainly be interesting to see what Chip comes up with. But in the meantime it is still possible to program the P2 in Spin.

David Betz · 2019-12-02 00:40

ersmith wrote: »

David Betz wrote: »

ersmith wrote: »

Cluso99 wrote: »

What I would like is to launch python from the P2 from my OS ie run the upython binary from the P2. But this requires my P2 OS and I haven’t had time to get this running by recompiling the P1 spin code to p2asm yet. I’ve been hoping Chip might release a limited version of spin2 as it would be so much easier.

Have you tried fastspin? It does support compiling Spin1 on the P2.

Compile Spin1 "on" the P2? I didn't realize you could compile directly on the P2. Or did you mean to say "compile Spin1 *for* the P2"?

Sorry, yes, "for" the P2. And maybe @Cluso99 is waiting for Chip's Spin2 so he can compile "on" the P2 using his OS? But I don't think the initial Spin2 release will support that.

It'll certainly be interesting to see what Chip comes up with. But in the meantime it is still possible to program the P2 in Spin.

Yes, I will be interested to see what self-hosted options appear and how much they are actually used. I would find it hard to give up my internet connection!

Cluso99 · 2019-12-02 07:35

I just want to compile all the spin1 modules for the P2 (on a PC).

I'm not sure how the spin1 to P2ASM would work. I haven't had time to try yet. My OS stays partially resident so there are a few gotchas that I need to rework, and I need to get Kye's spin FAT32 code converted. I've done but not tested the SD pasm driver. That was weeks ago

There is always something better to do when I get a chance - such as trying uPython.

kamilion · 2019-12-02 15:05

Would you be so kind as to make a github release with the new loadp2 binary? The last binary release was july 26th and it appears much has changed since then.

ersmith · 2019-12-03 11:35

kamilion wrote: »

Would you be so kind as to make a github release with the new loadp2 binary? The last binary release was july 26th and it appears much has changed since then.

Huh, I had forgotten that I ever did binary releases of loadp2. I guess I can do a new one, but loadp2 is still in flux. For now I suggest you use the loadp2.exe from the .zip file earlier in this thread -- it's based on the newest sources.

Rayman · 2019-12-03 16:35

I'm testing out the new loadp2 for loading flash and having trouble...

So, I downloaded latest FlexGUI and can't get it to load flash or RAM.
This is with my own board, so maybe that's an issue, but I don't think so.

Strange thing about this is that with SpinEdit, I can load RAM just fine using your Fastspin and loadp2.
Also, the Taqoz stuff seems to work in FlexGUI.
But, when trying to load RAM or Flash, I get:

ERROR: timeout waiting for initial checksum: got -1
Press enter to continue...

Any idea what I might be doing wrong?

ersmith · 2019-12-03 16:50

Rayman wrote: »

I'm testing out the new loadp2 for loading flash and having trouble...

So, I downloaded latest FlexGUI and can't get it to load flash or RAM.
This is with my own board, so maybe that's an issue, but I don't think so.

Strange thing about this is that with SpinEdit, I can load RAM just fine using your Fastspin and loadp2.

If it's working in SpinEdit, what command line are you using for loadp2? How does it compare with flexgui's command line? You aren't using the -SINGLE command line option by any chance are you? That uses a very different download path to the fast download used by default (and required for flash loading).

Does your board use the standard protocol for reset, using DTR/RTS? loadp2 resets the board then downloads a small helper program, which then changes the clock mode and frequency to do a fast download.

Hmmm, maybe that's the trouble; how is the clock set up on your board?

Rayman · 2019-12-03 17:00

I just figured out that adding -SINGLE makes your command line work

I think my clock and FTDI is exactly the same as Eval board...

Rayman · 2019-12-03 17:02

Was there some issue with using a USB hub?
I'll try a direct connection...
That didn't help...

ersmith · 2019-12-03 17:10

USB hubs shouldn't be a problem. If -SINGLE works it's odd that the fast downloader is timing out, if the clock settings are the same. You could try adding -v (verbose) to the loadp2 command line and see if that gives any additional information.

Rayman · 2019-12-03 17:21

Now that I think about it, I do have 10k resistors between FTDI Tx/Rx and P2 pins...
That might slow it down a bit...
What baud is it trying to do?

Also, I have the FTDI I/O at 5V instead of 3.3 V...

This is what I get in verbose mode:

trying \\\\.\\COM9...
P2 version G found on serial port \\\\.\\COM9
Setting load mode to CHIP
Setting clock_mode to 12427f8
Loading fast loader for chip...
ERROR: timeout waiting for initial checksum: got -1
Press enter to continue...

ersmith · 2019-12-03 17:24

It defaults to 2Mbaud. You could slow it down with to 230400 baud by passing -l230400.

Rayman · 2019-12-03 17:26

Ok, yes that works.

Works at 921600 too...

new loadp2, now with simple scripting

Comments