@rogloh said:
@Tubular if you are meeting the MP guys tomorrow you might want to ask if there is a way to configure MP to keep the heap in an external memory and read/write data from there indirectly via a port-specific API instead of from addresses directly memory mapped into the micro's address space. If that's possible then we might be able to use external PSRAM for that purpose. If not, you might like to ask if they would consider that capability down the line. It's likely a matter of just using some optional API to read/write the heap where it is required, although if that touches a lot of places I can imagine they'd not be too keen. If instead all the heap data structure accesses go through some common function(s) then it might be feasible to tap into that function to redirect to read/write from external RAM (perhaps).
For the RISC-V version, at least, we could do this in the RISC-V emulation -- instead of compiling loads and stores to rdlong/wrlong we could compile them to subroutine calls that do the real load/store from external memory.
Yeah that could be handy, then we can detect specific regions of address space that GCC sees that could be indirectly mapped via mailbox calls. Although caching is likely also required to improve performance as there is quite a bit of latency for multi-COG applications.
I wonder if it might be better to use the external storage for code (and perhaps a framebuffer) rather than the MP heap, though? I think most MicroPython ports use execute in place from flash, so they expect a relatively large text segment but not so much room for data.
It's another possibility though I'd imagine we already have room for the MP executable in HUB so we'd would want the ability to enable larger programs, and perhaps multi-COGs each with their own heap.
For example, something like this would be kinda cool with multiple MP Cogs down the track, and could be useful for standalone:
2-4 MP COGs each with their own stack (HUB) and heap (PSRAM), cached
1 multi-UART driver COG or for other IO needs per MP COG like shared SD/flash access perhaps
1 video COG - for local console use or other graphics output
1 USB COG - for local console/ HID device control
1 PSRAM COG
@Tubular said:
Micropython is a good option for what you describe, and about to get better.
Version 1.23 will include OpenAMP support for spinning up additional cores, which can be running forth or spin2 or pasm2 or whatever, a bit cleaner and more standard than the CPU method we use now in both streams of MicroPython. MP 1.23 might also feature tinyusb support if we get our act together
I think the Micropython docs are pretty good. They have benefitted from a Google "season of docs" (or two)
(From Tread: Thoughts about a tiny operating system)
Yes, I agree, Micropython has good docs and seems to have enough clear standardisation to make sharing of code between processors attractive and possible.
My experience using Micropython and Python is, that it is very nice, if you can mainly use precompiled libraries, written in C, like Numpy and Scipy or the libraries included in Micropython like SSD1306. - On the other side, if you have to write and use significant Python code, than the lamentable efficiency of Micropython becomes a problem. (I had to give up one project, a complete desaster. :-) ) The Micropython people seem to be well ware of the bad efficiency, they have included inline assembler and additionally the Viper language in the Pico distribution.
So, in my experience everything depends on the selection of libraries, compiled from C, which can be included into the distribution. - So one question would be, what you plan to include?
The possibility to dynamically load/link compiled C functions would solve the limitation given by RAM. Is this also in the pipeline?
The next question is, if inline assembler and Viper are planned too?
Of course everything will only make sense for P2, if multiple cores can be used directly with Python code, which seems to be the effect of OpenAmp.
So, in my experience everything depends on the selection of libraries, compiled from C, which can be included into the distribution. - So one question would be, what you plan to include?
The possibility to dynamically load/link compiled C functions would solve the limitation given by RAM. Is this also in the pipeline?
The next question is, if inline assembler and Viper are planned too?
Of course everything will only make sense for P2, if multiple cores can be used directly with Python code, which seems to be the effect of OpenAmp.
Christof I think the current challenge is just to get synced up with the main MP line (which Ray has largely done), and then see if we can get a basic Propeller-2 port included in the MP distribution. Once we get that far, its possible to extend MP with nice libraries, and of course its possible to build MP with your own custom needs.
Thats probably not what you want to hear at this instant, but having missed one opportunity to do this back on 1.13, keen to see if we can achieve this now
The 'multiple cores can be used directly with Python code' is a bigger challenge, unlikely to be solved right now. For many things a single MP core + access to shared memory + other supporting cores (running forth spin pasm2 etc) would still be awesome
It appears that the RiscvP2 version could in principle already run another python instance if it used LUT for cache and therefore was slow.
But, so many things to do and that's way on the back of my list anyway...
First for me is getting i2c and spi to work. If can't get the machine one going, then my own version appears to be easy substitute until issue resolved.
There are a lot of device drivers around that use i2c and spi and want to try them...
@Rayman said:
It appears that the RiscvP2 version could in principle already run another python instance if it used LUT for cache and therefore was slow.
But, so many things to do and that's way on the back of my list anyway...
First for me is getting i2c and spi to work. If can't get the machine one going, then my own version appears to be easy substitute until issue resolved.
There are a lot of device drivers around that use i2c and spi and want to try them...
IMO it's best to try to do things the proper MP way if you can and use the machine approach instead of a home grown solution. It may take a little more work initially but would be more suitable for any port agreed to be maintained by the official MP team and will fit existing MP code written by others to do I2C transfers. I know it's certainly doable as I've already done it for native P2 MP as well as machine SPI.
@Rayman said:
@rogloh would love to use official version of i2c, just can’t figure out how to make it work yet..
Ok no problem. Hopefully you'll get there in the end. I know it takes time to figure it out. In general (and this is not exhaustive) you'll probably need to:
grab machine_pin.c from my native build to be used instead of what is in modpybpin.c. Basically modpyb stuff needs to be replaced by functionality in modmachine.c, machine_pin.c and machine_cpu.c etc
update modmachine.c so I2C and SPI object are included in the import lib hierarchy with changes to its machine_module_globals_table:
Had to tweak the makefile just a bit to work with the shiny new version of riscvp2, now based on xpack-riscv-none-elf-gcc-13.2.0-2-linux-x64.
Things seem to be otherwise the same...
But, I can't figure out where... Don't think in any of the port files...
I do see that "machine" started showing up in the modules at some point, not sure exactly how or where this happens, but guess it's the same thing.
Anyway, maybe doesn't matter if it compiles?
Now on to trying to enable soft i2c and then spi...
MP SPI for P2 could be either. It can be soft (bit bang) or somewhat accelerated by using MISO/MOSI smart pins operating in TX/RX SYNC modes and a clock pin in some clock transition mode at some nominal frequency, so really you could go either way. Obviously the simplest is just pure bit bang in order to get it working but the performance will be slightly restricted.
Ideally hardware assisted would be the better solution for highest speeds, but doing that probably sort of needs some integration with MP's pin "modes", treating the smart pin mode as an AF (alternative function) rather than general input or output. That's how I see it anyway. Bit bang can still go a long way for simple sensor transfers though.
Got it to compile with i2c stuff in it, probably won't work yet though.
Issue had to get past is that /extmod/modmachine.c and extmod/machine_i2c.c are included in extmod/extmod.mk
Seems they need to be removed from the makefile in order to get it to compile.
But, this doesn't seem right... Or is it?
Ok, yeah not right. This removes machine module completely...
Have to figure out how to get that version and this version of modmachine to coexist...
Looks like they include the port specific modmachine.c file from the extmod folder file according to this comment I found in the STM32 port in Github's top of tree of MP so it's not built independently in the Makefile directly. Rather ugly to include source C files from other C files via #defines and easy to miss. Would be better to name it modmachine.inc or something obvious so you know it's special and not intended for a Makefile.
Thanks @rogloh . Think have a clue as to what's going on here now.
Got it to compile and now I2C shows up as a machine method. Might actually work once there is I2C code there...
So, there is something going on here that I think is superfluous fluff, but not sure about...
It seems some of the ports like to define an i2c type and assign it to self and then pass self as the first parameter in a variable parameter function and then they process variable parameter length functions in a strange way like this:
The technique of passing self as the first argument is typically one way to emulate OOP with just pure C code. Although in this case that first argument is the first element of the pos_args array which itself is the second function argument. Whatever. Am sure there are reasons probably related to the fact that you can send different arguments in different cases of the function invoked - i.e. function/method overloading.
Is the self thing needed to have multiple instances of the i2c method?
Not really sure how this works…
If i2c pins are defined as simple ints does that mean there is only ever one version of them?
Maybe this is what the make new function is for? To create a new instance of the self structure so the multiple instances of i2c method can have unique setttings?
@Rayman said:
This took some time to figure out...
But, there's some magic in this "self" stuff.
In particular, one needs:
self->base.type = &machine_i2c_type;
for it to work right...
There's nothing magic about it. It's just a pointer to a struct containing all the context for that particular I2C device; a new instance gets created for every device. All object-oriented programming works this way, whether it handles self for you (where it is magic) or you have to do it by hand.
I'm new to the micropython source code and barely know any actual Python to boot, so started out way over my head.
But, think starting to get a handle on it now...
Btw also think seeing that what is taken over here is the hardware version of i2c.
But one can also activate the soft i2c. That relies on a Pins module, but looks like @ersmith has implemented that in his pyb modules. So, if didn’t need pull-ups, looks like that is almost out of the box ready.
Comments
Yeah that could be handy, then we can detect specific regions of address space that GCC sees that could be indirectly mapped via mailbox calls. Although caching is likely also required to improve performance as there is quite a bit of latency for multi-COG applications.
It's another possibility though I'd imagine we already have room for the MP executable in HUB so we'd would want the ability to enable larger programs, and perhaps multi-COGs each with their own heap.
For example, something like this would be kinda cool with multiple MP Cogs down the track, and could be useful for standalone:
2-4 MP COGs each with their own stack (HUB) and heap (PSRAM), cached
1 multi-UART driver COG or for other IO needs per MP COG like shared SD/flash access perhaps
1 video COG - for local console use or other graphics output
1 USB COG - for local console/ HID device control
1 PSRAM COG
(From Tread: Thoughts about a tiny operating system)
Yes, I agree, Micropython has good docs and seems to have enough clear standardisation to make sharing of code between processors attractive and possible.
My experience using Micropython and Python is, that it is very nice, if you can mainly use precompiled libraries, written in C, like Numpy and Scipy or the libraries included in Micropython like SSD1306. - On the other side, if you have to write and use significant Python code, than the lamentable efficiency of Micropython becomes a problem. (I had to give up one project, a complete desaster. :-) ) The Micropython people seem to be well ware of the bad efficiency, they have included inline assembler and additionally the Viper language in the Pico distribution.
So, in my experience everything depends on the selection of libraries, compiled from C, which can be included into the distribution. - So one question would be, what you plan to include?
The possibility to dynamically load/link compiled C functions would solve the limitation given by RAM. Is this also in the pipeline?
The next question is, if inline assembler and Viper are planned too?
Of course everything will only make sense for P2, if multiple cores can be used directly with Python code, which seems to be the effect of OpenAmp.
Christof I think the current challenge is just to get synced up with the main MP line (which Ray has largely done), and then see if we can get a basic Propeller-2 port included in the MP distribution. Once we get that far, its possible to extend MP with nice libraries, and of course its possible to build MP with your own custom needs.
Thats probably not what you want to hear at this instant, but having missed one opportunity to do this back on 1.13, keen to see if we can achieve this now
The 'multiple cores can be used directly with Python code' is a bigger challenge, unlikely to be solved right now. For many things a single MP core + access to shared memory + other supporting cores (running forth spin pasm2 etc) would still be awesome
It appears that the RiscvP2 version could in principle already run another python instance if it used LUT for cache and therefore was slow.
But, so many things to do and that's way on the back of my list anyway...
First for me is getting i2c and spi to work. If can't get the machine one going, then my own version appears to be easy substitute until issue resolved.
There are a lot of device drivers around that use i2c and spi and want to try them...
IMO it's best to try to do things the proper MP way if you can and use the machine approach instead of a home grown solution. It may take a little more work initially but would be more suitable for any port agreed to be maintained by the official MP team and will fit existing MP code written by others to do I2C transfers. I know it's certainly doable as I've already done it for native P2 MP as well as machine SPI.
Ray is the soft SPI and soft I2C also broken/missing? Or just the hardware ones
Tried soft for i2c
Haven’t tried spi yet, that might be somewhere in between for P2?
@rogloh would love to use official version of i2c, just can’t figure out how to make it work yet..
Ok no problem. Hopefully you'll get there in the end. I know it takes time to figure it out. In general (and this is not exhaustive) you'll probably need to:
Note that this was with 1.13 so the later builds may well have additional requirements or things may have changed in other areas I am unfamiliar with.
Had to tweak the makefile just a bit to work with the shiny new version of riscvp2, now based on xpack-riscv-none-elf-gcc-13.2.0-2-linux-x64.
Things seem to be otherwise the same...
Thanks @rogloh, I'll give it a shot...
Hmm... First thing I see is that all these headers are gone in 1.22.2:
This is probably going to take some work...
Looks like they rolled up the HW related header files into extmod/modmachine.h
They are just keeping you on your toes Ray.
We had the monthly MP meetup last night so I provided a P2 update on your work. Damien couldn't make it this time unfortunately
Got it to compile with modmachine.c included in the makefile.
But, had to comment out this part at the bottom because already defined somewere:
//RJA Seems already defined somewhere else? //const mp_obj_module_t mp_module_machine = { // .base = { &mp_type_module }, // .globals = (mp_obj_dict_t *)&machine_module_globals, //};But, I can't figure out where... Don't think in any of the port files...
I do see that "machine" started showing up in the modules at some point, not sure exactly how or where this happens, but guess it's the same thing.
Anyway, maybe doesn't matter if it compiles?
Now on to trying to enable soft i2c and then spi...
Btw think our spi is hard or soft?
More like hardware assisted?
MP SPI for P2 could be either. It can be soft (bit bang) or somewhat accelerated by using MISO/MOSI smart pins operating in TX/RX SYNC modes and a clock pin in some clock transition mode at some nominal frequency, so really you could go either way. Obviously the simplest is just pure bit bang in order to get it working but the performance will be slightly restricted.
Ideally hardware assisted would be the better solution for highest speeds, but doing that probably sort of needs some integration with MP's pin "modes", treating the smart pin mode as an AF (alternative function) rather than general input or output. That's how I see it anyway. Bit bang can still go a long way for simple sensor transfers though.
Appears they moved all the actual I2C implementation files into the ports folders...
Hopefully, can just copy and paste from ozpropdev version...
Got it to compile with i2c stuff in it, probably won't work yet though.
Issue had to get past is that /extmod/modmachine.c and extmod/machine_i2c.c are included in extmod/extmod.mk
Seems they need to be removed from the makefile in order to get it to compile.
But, this doesn't seem right... Or is it?
Ok, yeah not right. This removes machine module completely...
Have to figure out how to get that version and this version of modmachine to coexist...
Looks like they include the port specific modmachine.c file from the extmod folder file according to this comment I found in the STM32 port in Github's top of tree of MP so it's not built independently in the Makefile directly. Rather ugly to include source C files from other C files via #defines and easy to miss. Would be better to name it modmachine.inc or something obvious so you know it's special and not intended for a Makefile.
https://github.com/micropython/micropython/blob/master/ports/stm32/modmachine.c
line 27:
Actually they seem to use this technique for a lot of the machine things now...
Same STM32 port, see its mpconfigport.h file:
Thanks @rogloh . Think have a clue as to what's going on here now.
Got it to compile and now I2C shows up as a machine method. Might actually work once there is I2C code there...
Ok, think getting close now. Can init() and scan().
BTW: forum software doesn't seem to like ">", even when inside a code block... Have to delete them all...
So, there is something going on here that I think is superfluous fluff, but not sure about...
It seems some of the ports like to define an i2c type and assign it to self and then pass self as the first parameter in a variable parameter function and then they process variable parameter length functions in a strange way like this:
STATIC mp_obj_t pyb_i2c_send(size_t n_args, const mp_obj_t *pos_args, mp_map_t *kw_args) { static const mp_arg_t allowed_args[] = { { MP_QSTR_send, MP_ARG_REQUIRED | MP_ARG_OBJ, {.u_obj = MP_OBJ_NULL} }, { MP_QSTR_addr, MP_ARG_INT, {.u_int = PYB_I2C_MASTER_ADDRESS} }, { MP_QSTR_timeout, MP_ARG_KW_ONLY | MP_ARG_INT, {.u_int = 5000} }, }; // parse args pyb_i2c_obj_t *self = MP_OBJ_TO_PTR(pos_args[0]); mp_arg_val_t args[MP_ARRAY_SIZE(allowed_args)]; mp_arg_parse_all(n_args - 1, pos_args + 1, kw_args, MP_ARRAY_SIZE(allowed_args), allowed_args, args);Thinking can just do all this the old way. Seems to be working so far, but we'll see...
The technique of passing self as the first argument is typically one way to emulate OOP with just pure C code. Although in this case that first argument is the first element of the pos_args array which itself is the second function argument. Whatever. Am sure there are reasons probably related to the fact that you can send different arguments in different cases of the function invoked - i.e. function/method overloading.
Is the self thing needed to have multiple instances of the i2c method?
Not really sure how this works…
If i2c pins are defined as simple ints does that mean there is only ever one version of them?
Maybe this is what the make new function is for? To create a new instance of the self structure so the multiple instances of i2c method can have unique setttings?
This took some time to figure out...
But, there's some magic in this "self" stuff.
In particular, one needs:
self->base.type = &machine_i2c_type;for it to work right...
There's nothing magic about it. It's just a pointer to a struct containing all the context for that particular I2C device; a new instance gets created for every device. All object-oriented programming works this way, whether it handles
selffor you (where it is magic) or you have to do it by hand.I'm new to the micropython source code and barely know any actual Python to boot, so started out way over my head.
But, think starting to get a handle on it now...
Btw also think seeing that what is taken over here is the hardware version of i2c.
But one can also activate the soft i2c. That relies on a Pins module, but looks like @ersmith has implemented that in his pyb modules. So, if didn’t need pull-ups, looks like that is almost out of the box ready.