Lets take a look at the planned (?) setup for a new hardware library. Let's assume a new attractive device like WS2812 comes to the market, which needs its own fast interface.
Step; Task; who; tool; where to find the result (Don't know, how to make a table here.)
1. Write device driver; Jon; pnut-pasm2
2. Write mapper code; Jon; pnut-spin2
3. Write docu+example; Jon; pnut-spin2; ==> Github
Now we need that precompiled code, which shall be put into an array of micropyton during runtime.
4. Compile; Mr. X, a clever man, who has to be fit in pasm2, spin2 (and probably C) and micropython does this; flexgui is used (hopefully Jon has not used tricks, the compiler can't do);
At this point it is unclear for me, how parameters are passed to and results from the driver and how the different routines of the driver are accessed. At least at this point all information is lost about these function names, parameters for python, so:
5. Write new wrapper in python; Mr. X; he will use flexgui and micropython;
6. Write new docu+example; Mr. X; micropython ==> Github
7. User tries it and wants to tweak it a little bit; unfortunately he is a little bit less fit, so he has some problems to understand pasm2+spin2+this parameter and name conversion + micropython.
Is this the plan?
We have not included the task, that at some point, such library has to be integrated better into micropython, like a screen driver or a storage driver, because the standard routines shall use them. In this case it gets even more funny:
Some really clever person has to find "this other C-compiler", that is capable to compile the micropython source, which is very large and not easy to understand. Hopefully this other C-Compiler still works. Then he has to (re)write the device driver in assembler and C, to recompile micropython (and perhaps a new actal version of it) with it, document his changes and put this once again enlarged new micropython into Github.
"This other C-Compiler" is some secret weapon, it is not looked after and not documented.
The strange thing is, that micropython is written in C, and made to use C-routines but the "plan" is to use 3 different compilers.
IF the device drivers would be written in C directly, which is fast enough for >>95% of the problems and if really needed with some inline assembler. And IF there was a C-compiler, which provides Compact Memory Model (because the thumb instruction set is not there) along the other models cog and hub. And IF some "ParallaxPython" was stripped down to bare minimum of the really necessary things for eduction.
- Then everything could be done with ONE C-compiler, significant steps for programming and for documentation would not be necessary. A lot of more ram would be available, because only the needed libraries would be linked. I don't know it, but I think, that several instances of a very much simplified "ParallelPython" could run on >1 cogs. Only ONE tool has to be maintained. (I don't see any reason, that mueditor cannot be used with a stripped down ParallaxPython)
Ken, don't you want to write down, which bare minimum of a python you really need for education? PICAXE-chips, made for education, cannot even handle negative integer numbers. This trial will cost you a few hours and perhaps save you manyears.
Just write some answer, that you have considered all this deeply and I will give up and shut up.
I am very sorry, I too want to be polite and nice and say, that everything is emerging so good.
Best regards Christof
Step; Task; who; tool; where to find the result (Don't know, how to make a table here.)
1. Write device driver; Jon; pnut-pasm2
2. Write mapper code; Jon; pnut-spin2
3. Write docu+example; Jon; pnut-spin2; ==> Github
Now we need that precompiled code, which shall be put into an array of micropyton during runtime.
4. Compile; Mr. X, a clever man, who has to be fit in pasm2, spin2 (and probably C) and micropython does this; flexgui is used (hopefully Jon has not used tricks, the compiler can't do);
Step 4 doesn't require any particular knowledge. The precompiled array of bytes needed for micropython is literally just the DAT section of the spin2 driver, so it can be produced automatically by e.g. "fastspin -c". Possibly PNut could also do this, or if it can't that feature could trivially be added (openspin and bstc certainly have the ability to compile just the DAT).
At this point it is unclear for me, how parameters are passed to and results from the driver and how the different routines of the driver are accessed. At least at this point all information is lost about these function names, parameters for python, so:
That's all part of the documentation for the Spin2 driver, presumably. Most drivers take their parameters in the form of a pointer to a mailbox. That pointer, along with a pointer to the code, is passed to the micropython cpu.start() method.
That's assuming that the Team Oz micropython has the Cpu() method that I wrote for my version of micropython. Their version is based on my version, so probably it does. If not, I'm sure they'll provide something similar.
I don't want to make it sound like everything is solved and it will all be smooth. There are some gotchas with any device driver to be used in micropython:
(1) No inline assembly in methods. All the assembly code will have to be in the DAT section.
(2) Any addresses of DAT section variables will have to be computed at run time, or else passed as part of the mailbox. This isn't too unreasonable, since Chip's Spin2 compiler has a similar restriction.
(3) Not all Spin2 builtin methods have corresponding micropython methods, at least not in my micropython. If Team Oz hasn't added these yet, the micropython interpreter will have to be extended to have them. We can get started now though, the basic pin access methods are there (again, in my version of micropython; we haven't seen Team Oz's yet, but I'm sure they'll have them as well).
In principle the conversion from Spin2 to micropython could be automated, just as spin2cpp already automates the conversion from Spin2 to C++. There would be some real work to add Python output to spin2cpp, but it could be done.
Another approach would be to have a "unified" driver with a common DAT section, Spin2 methods for accessing this, and Python methods too, all in the same file. This would be easiest if Spin2 were officially extended to include #ifdef; then the Python part would be under #ifdef PYTHON. But it could also be done with some kind of special comments in the Spin2 file, and a preprocessor that could extract these. Then e.g. a blinking light driver could be written with something like:
' code to blink a PIN
' mailbox just has the pin number (extend as desired)
' and number of cycles to delay between blinks
DAT
org 0
entry
rdlong pinnum, ptra[0]
rdlong delay, ptra[1]
loop
drvnot pinnum ' toggle pin
waitx delay ' wait
jmp #loop
pinnum long 0
delay long 0
' spin version of control functions
VAR long mailbox[2]
PUB start(pin, delay) : cognum
mailbox[0] := pin
mailbox[1] := delay
cognum := coginit(16, @entry, @mailbox)
' Python version of control functions
{{!python
def start(pin, delay):
cog = pyb.Cpu()
mailbox = array.array('i', [pin, delay])
cog.start(dat, mailbox)
}}
That's assuming that the Team Oz micropython has the Cpu() method that I wrote for my version of micropython. Their version is based on my version, so probably it does. If not, I'm sure they'll provide something similar.
Yep this Cpu() method is present, I just ported the same API to be compatible.
(3) Not all Spin2 builtin methods have corresponding micropython methods, at least not in my micropython. If Team Oz hasn't added these yet, the micropython interpreter will have to be extended to have them. We can get started now though, the basic pin access methods are there (again, in my version of micropython; we haven't seen Team Oz's yet, but I'm sure they'll have them as well).
Yes the same IO pin control should be available in pyb module. It should be very similar/compatible to yours but I think I might have needed to interface to slightly different low level HAL controls for p2gcc/p2asm for timing stuff IIRC. It's been a while since that work was done so I can't recall offhand.
Another approach would be to have a "unified" driver with a common DAT section, Spin2 methods for accessing this, and Python methods too, all in the same file. This would be easiest if Spin2 were officially extended to include #ifdef; then the Python part would be under #ifdef PYTHON. But it could also be done with some kind of special comments in the Spin2 file, and a preprocessor that could extract these. Then e.g. a blinking light driver could be written with something like:
' code to blink a PIN
' mailbox just has the pin number (extend as desired)
' and number of cycles to delay between blinks
DAT
org 0
entry
rdlong pinnum, ptra[0]
rdlong delay, ptra[1]
loop
drvnot pinnum ' toggle pin
waitx delay ' wait
jmp #loop
pinnum long 0
delay long 0
' spin version of control functions
VAR long mailbox[2]
PUB start(pin, delay) : cognum
mailbox[0] := pin
mailbox[1] := delay
cognum := coginit(16, @entry, @mailbox)
' Python version of control functions
{{!python
def start(pin, delay):
cog = pyb.Cpu()
mailbox = array.array('i', [pin, delay])
cog.start(dat, mailbox)
}}
After reading what Christof Eb. talked about I think it certainly makes sense to not have to recompile everyone's MicroPython environment each time someone wants to create a new module to be used in MicroPython. They just would want to bring it into their existing system and use it. So there really needs to be some agreed connection to be able to access PASM/SPIN2 stuff from Python, and the mailbox way is a reasonable approach to consider. A lot of people would probably want to have some "standard" MicroPython image/environment that already exists with a basic set of functionality and then other great tools like FastSpin could be used to build other driver extensions which could be shared and accessed via MicroPython module wrappers. The discussion about using frozen byte code might be problematic this way if it does need recompilation and inclusion in the actual binary environment. That being said other people might need to customize Python further and have more control over the build. It's definitely not all sorted.
Yes, that is about how I figured. All information about names or number or types of parameters are lost in the mailbox and have to be declared newly.
OK, if the driver consists of more than one function. Do you have a jump table then or do you need some sort of command interpreter?
(I use forth as command interpreter between a P1 and a python script running on a raspi.)
I did not say, that all this cannot be done.
Do you recommend to keep this multicompiler track then?
OK, let me say that I agree with you @"Christof Eb." that the decision to make the primary language for the P2 a new, custom language was a poor one. Parallax would have gotten much more traction with developers if the chip could be "officially" programmed in C or even Spin1. But that horse has been dead for a while now and beating it won't make it get up again .
All information about names or number or types of parameters are lost in the mailbox and have to be declared newly.
OK, if the driver consists of more than one function. Do you have a jump table then or do you need some sort of command interpreter?
??? The mailbox is an internal feature of the driver, it's not something the user has to see. To use the driver from python one would do something like:
import serial
ser = Serial.new()
ser.start(txpin, rxpin, baud)
ser.string("hello, world!\n")
ser.dec(100)
The individual methods (start, string, dec, and so on) are all written in Python.
To write a new driver in Python the only nasty part is that the PASM part (if any) has to be converted to binary by some external tool. This could be overcome if we had an assembler written in Python, and/or if an assembler was built in to micropython. In fact micropython does have an assembler for some architectures, but it's syntax is non-standard and porting it to the P2 is highly non-trivial (but do-able).
Other paths that could be done to make driver development for micropython easier are:
(1) An automated tool like spin2cpp that not only compiles the PASM to binary, but also converts the Spin2 parts into Python;
(2) OR: A way to load compiled C files as objects. Micropython supports this in Linux with .so file loading, and I think there's a rudimentary way to load compiled files on other platforms in the latest micropython. But this will need work to port to P2!
(3) OR: A way to run compiled Spin2 bytecodes directly from micropython, together with a way to invoke the Spin2 methods, and some kind of automated way to create the Python wrappers for Spin2
Thanks, Eric,
the problem I see is, that there is nobody, who writes the driver in python or does this job of converting it. This guy, named X, has to be a real and well gifted person, even if it is made more easy. The people, who write the driver for spin2 or for flexc will not do that job. This user is the one who wants to have the driver, but it is unlikely, that he is capable to do the conversion.
No good idea to strip down python radically and use flexC as linker?
This Tinypython stopped at some point with a missing header file and needed at this point 100k of ram using Propeller_C for P1 with cmm. So this is still rather large with P1.
What I don't understand is, why can a Risc-V Compiler with an emulated Risc-V in P2 be so fast? Does this mean, that the oftenly used instructions are very similar and the others are used very seldom? If this compiler was used, you would write the drivers with risc-V-assembler?
At this moment only a few drivers exist, the docu for spin2 is not yet written, and no examples, so it is the very last moment to reanimate the horse. I think, this debug() can possibly be written in C using some sort of macro, which tends this global database with some text and formatting for the debug-printer-spooler-cog and lets the active cog put the data into a buffer?
OK, now it is discussed to write a Spin2 compiler in Spin2 in the pnut thread.
???
I give up.
Sorry to have disturbed.
The main reason I think that is an interesting effort is that it gives hope of a Parallax-supported compiler that could be made to be cross platform. We, of course, already have some nice cross-platform compiles like fastspin, catalina, and p2gcc but they are third party software and some customers may want to use something that is supported directly by Parallax. Spin2 in Spin2 seems like the only way that will happen short of convincing Parallax to adopt one of the third party solutions as their own.
Comments
Lets take a look at the planned (?) setup for a new hardware library. Let's assume a new attractive device like WS2812 comes to the market, which needs its own fast interface.
Step; Task; who; tool; where to find the result (Don't know, how to make a table here.)
1. Write device driver; Jon; pnut-pasm2
2. Write mapper code; Jon; pnut-spin2
3. Write docu+example; Jon; pnut-spin2; ==> Github
Now we need that precompiled code, which shall be put into an array of micropyton during runtime.
4. Compile; Mr. X, a clever man, who has to be fit in pasm2, spin2 (and probably C) and micropython does this; flexgui is used (hopefully Jon has not used tricks, the compiler can't do);
At this point it is unclear for me, how parameters are passed to and results from the driver and how the different routines of the driver are accessed. At least at this point all information is lost about these function names, parameters for python, so:
5. Write new wrapper in python; Mr. X; he will use flexgui and micropython;
6. Write new docu+example; Mr. X; micropython ==> Github
7. User tries it and wants to tweak it a little bit; unfortunately he is a little bit less fit, so he has some problems to understand pasm2+spin2+this parameter and name conversion + micropython.
Is this the plan?
We have not included the task, that at some point, such library has to be integrated better into micropython, like a screen driver or a storage driver, because the standard routines shall use them. In this case it gets even more funny:
Some really clever person has to find "this other C-compiler", that is capable to compile the micropython source, which is very large and not easy to understand. Hopefully this other C-Compiler still works. Then he has to (re)write the device driver in assembler and C, to recompile micropython (and perhaps a new actal version of it) with it, document his changes and put this once again enlarged new micropython into Github.
"This other C-Compiler" is some secret weapon, it is not looked after and not documented.
The strange thing is, that micropython is written in C, and made to use C-routines but the "plan" is to use 3 different compilers.
IF the device drivers would be written in C directly, which is fast enough for >>95% of the problems and if really needed with some inline assembler. And IF there was a C-compiler, which provides Compact Memory Model (because the thumb instruction set is not there) along the other models cog and hub. And IF some "ParallaxPython" was stripped down to bare minimum of the really necessary things for eduction.
- Then everything could be done with ONE C-compiler, significant steps for programming and for documentation would not be necessary. A lot of more ram would be available, because only the needed libraries would be linked. I don't know it, but I think, that several instances of a very much simplified "ParallelPython" could run on >1 cogs. Only ONE tool has to be maintained. (I don't see any reason, that mueditor cannot be used with a stripped down ParallaxPython)
Ken, don't you want to write down, which bare minimum of a python you really need for education? PICAXE-chips, made for education, cannot even handle negative integer numbers. This trial will cost you a few hours and perhaps save you manyears.
Just write some answer, that you have considered all this deeply and I will give up and shut up.
I am very sorry, I too want to be polite and nice and say, that everything is emerging so good.
Best regards Christof
That's all part of the documentation for the Spin2 driver, presumably. Most drivers take their parameters in the form of a pointer to a mailbox. That pointer, along with a pointer to the code, is passed to the micropython cpu.start() method.
That's assuming that the Team Oz micropython has the Cpu() method that I wrote for my version of micropython. Their version is based on my version, so probably it does. If not, I'm sure they'll provide something similar.
I don't want to make it sound like everything is solved and it will all be smooth. There are some gotchas with any device driver to be used in micropython:
(1) No inline assembly in methods. All the assembly code will have to be in the DAT section.
(2) Any addresses of DAT section variables will have to be computed at run time, or else passed as part of the mailbox. This isn't too unreasonable, since Chip's Spin2 compiler has a similar restriction.
(3) Not all Spin2 builtin methods have corresponding micropython methods, at least not in my micropython. If Team Oz hasn't added these yet, the micropython interpreter will have to be extended to have them. We can get started now though, the basic pin access methods are there (again, in my version of micropython; we haven't seen Team Oz's yet, but I'm sure they'll have them as well).
In principle the conversion from Spin2 to micropython could be automated, just as spin2cpp already automates the conversion from Spin2 to C++. There would be some real work to add Python output to spin2cpp, but it could be done.
Another approach would be to have a "unified" driver with a common DAT section, Spin2 methods for accessing this, and Python methods too, all in the same file. This would be easiest if Spin2 were officially extended to include #ifdef; then the Python part would be under #ifdef PYTHON. But it could also be done with some kind of special comments in the Spin2 file, and a preprocessor that could extract these. Then e.g. a blinking light driver could be written with something like:
' code to blink a PIN ' mailbox just has the pin number (extend as desired) ' and number of cycles to delay between blinks DAT org 0 entry rdlong pinnum, ptra[0] rdlong delay, ptra[1] loop drvnot pinnum ' toggle pin waitx delay ' wait jmp #loop pinnum long 0 delay long 0 ' spin version of control functions VAR long mailbox[2] PUB start(pin, delay) : cognum mailbox[0] := pin mailbox[1] := delay cognum := coginit(16, @entry, @mailbox) ' Python version of control functions {{!python def start(pin, delay): cog = pyb.Cpu() mailbox = array.array('i', [pin, delay]) cog.start(dat, mailbox) }}Yes the same IO pin control should be available in pyb module. It should be very similar/compatible to yours but I think I might have needed to interface to slightly different low level HAL controls for p2gcc/p2asm for timing stuff IIRC. It's been a while since that work was done so I can't recall offhand.
After reading what Christof Eb. talked about I think it certainly makes sense to not have to recompile everyone's MicroPython environment each time someone wants to create a new module to be used in MicroPython. They just would want to bring it into their existing system and use it. So there really needs to be some agreed connection to be able to access PASM/SPIN2 stuff from Python, and the mailbox way is a reasonable approach to consider. A lot of people would probably want to have some "standard" MicroPython image/environment that already exists with a basic set of functionality and then other great tools like FastSpin could be used to build other driver extensions which could be shared and accessed via MicroPython module wrappers. The discussion about using frozen byte code might be problematic this way if it does need recompilation and inclusion in the actual binary environment. That being said other people might need to customize Python further and have more control over the build. It's definitely not all sorted.
OK, if the driver consists of more than one function. Do you have a jump table then or do you need some sort of command interpreter?
(I use forth as command interpreter between a P1 and a python script running on a raspi.)
I did not say, that all this cannot be done.
Do you recommend to keep this multicompiler track then?
??? The mailbox is an internal feature of the driver, it's not something the user has to see. To use the driver from python one would do something like:
import serial ser = Serial.new() ser.start(txpin, rxpin, baud) ser.string("hello, world!\n") ser.dec(100)The individual methods (start, string, dec, and so on) are all written in Python.To write a new driver in Python the only nasty part is that the PASM part (if any) has to be converted to binary by some external tool. This could be overcome if we had an assembler written in Python, and/or if an assembler was built in to micropython. In fact micropython does have an assembler for some architectures, but it's syntax is non-standard and porting it to the P2 is highly non-trivial (but do-able).
Other paths that could be done to make driver development for micropython easier are:
(1) An automated tool like spin2cpp that not only compiles the PASM to binary, but also converts the Spin2 parts into Python;
(2) OR: A way to load compiled C files as objects. Micropython supports this in Linux with .so file loading, and I think there's a rudimentary way to load compiled files on other platforms in the latest micropython. But this will need work to port to P2!
(3) OR: A way to run compiled Spin2 bytecodes directly from micropython, together with a way to invoke the Spin2 methods, and some kind of automated way to create the Python wrappers for Spin2
the problem I see is, that there is nobody, who writes the driver in python or does this job of converting it. This guy, named X, has to be a real and well gifted person, even if it is made more easy. The people, who write the driver for spin2 or for flexc will not do that job. This user is the one who wants to have the driver, but it is unlikely, that he is capable to do the conversion.
No good idea to strip down python radically and use flexC as linker?
This Tinypython stopped at some point with a missing header file and needed at this point 100k of ram using Propeller_C for P1 with cmm. So this is still rather large with P1.
What I don't understand is, why can a Risc-V Compiler with an emulated Risc-V in P2 be so fast? Does this mean, that the oftenly used instructions are very similar and the others are used very seldom? If this compiler was used, you would write the drivers with risc-V-assembler?
At this moment only a few drivers exist, the docu for spin2 is not yet written, and no examples, so it is the very last moment to reanimate the horse. I think, this debug() can possibly be written in C using some sort of macro, which tends this global database with some text and formatting for the debug-printer-spooler-cog and lets the active cog put the data into a buffer?
We will see, if there is a reaction.
???
I give up.
Sorry to have disturbed.