I don't know how exactly it works, but the compiler somehow knows to convert FFCONF_H into "lib/oofatfs/ffconf.h" with 1.19.
But, won't do it in 1.20.0 and gives the wrong configuration file error. So, I just manually stuck it in there.
That got me past one hurdle and into mess of more difficult looking things...
@Rayman
I think by adding ffconf.h manually you probably broke the linking stage as you appear to have multiple definitions related to VFS APIs now. If you look at the makefile output for 1.19 you'll see these defines everywhere passed into the individual compiler commands. For some reason they are not in the 1.20 makefile and this might be a problem.
-DFFCONF_H=\"lib/oofatfs/ffconf.h\"
This ffconf.h include file should already be setup to only be included once through its own means (using #ifndef etc) and not generate multiple definitions which your forced change may have introduced.
Your other 1.20 linker errors at the end of the log seem to also be related to the disk and vfs type functions are also likely to be related to ffconf.h IMO or maybe they changed the way file IO stuff is built in the Makefile and it needs to be customized to suit that - check the differences with how the other platforms do it perhaps.
I also noticed a bunch of files stored the extmod folder from line 265 to 349 in the 1.19 makefile output do not appear in the 1.20 version. Maybe the MP guys rearranged this area but it seems strange to not build these and they also relate to disk/vfs stuff. So that might be something missing in the build config. Similarly this make step in 1.19 was not done in 1.20:
mkdir -p build/extmod/
Hope that helps a bit but you'll just have to mess about a bit more to fix the linker errors. It should work once you've figured it out.
These were those missing files not getting built in 1.20, you can see that the vfs stuff is also missing which was one of the problems with the undefined references in the linker output of 1.20.
CC ../../extmod/moduasyncio.c
CC ../../extmod/moductypes.c
CC ../../extmod/modujson.c
CC ../../extmod/moduos.c
CC ../../extmod/modure.c
CC ../../extmod/moduzlib.c
CC ../../extmod/moduheapq.c
CC ../../extmod/modutimeq.c
CC ../../extmod/moduhashlib.c
CC ../../extmod/moducryptolib.c
CC ../../extmod/modubinascii.c
CC ../../extmod/virtpin.c
CC ../../extmod/machine_bitstream.c
CC ../../extmod/machine_mem.c
CC ../../extmod/machine_pinbase.c
CC ../../extmod/machine_signal.c
CC ../../extmod/machine_pulse.c
CC ../../extmod/machine_pwm.c
CC ../../extmod/machine_i2c.c
CC ../../extmod/machine_spi.c
CC ../../extmod/modbluetooth.c
CC ../../extmod/modussl_axtls.c
CC ../../extmod/modussl_mbedtls.c
CC ../../extmod/moduplatform.c
CC ../../extmod/modurandom.c
CC ../../extmod/moduselect.c
CC ../../extmod/moduwebsocket.c
CC ../../extmod/modwebrepl.c
CC ../../extmod/modframebuf.c
CC ../../extmod/vfs.c
CC ../../extmod/vfs_blockdev.c
CC ../../extmod/vfs_reader.c
CC ../../extmod/vfs_posix.c
CC ../../extmod/vfs_posix_file.c
CC ../../extmod/vfs_fat.c
CC ../../extmod/vfs_fat_diskio.c
CC ../../extmod/vfs_fat_file.c
CC ../../extmod/vfs_lfs.c
CC ../../extmod/utime_mphal.c
CC ../../extmod/uos_dupterm.c
CC ../../shared/libc/abort_.c
CC ../../shared/libc/printf.c
No I am just saying it doesn't appear that the Makefile even made these files, which it did in 1.19 - it's likely a problem for the linker if a bunch of files are now missing.
Unless it's documented that something changed I'd expect you to only need to do a single make operation but why is this step not happening for 1.20? I don't know either as I've only built up to 1.13. Perhaps it's a new config file setting in mpconfigport.h etc that needs to be altered/included or some other thing they changed that's now missing in the build files for the RISC-V variant?
Perhaps your Makefile might need some updating to suit the new scheme. All the other port's Makefiles seemed to have been updated from 1.19 to 1.20 so if you were still using something based on your 1.19 RISC-V variant it probably won't work. Take a look at the pattern of how each ports Makefile changed (e.g. for stm32/esp8266/samd etc) in the diffs above and see if you can do the same or similar changes for your RISC-V one.
A pertinent comment from the checkin above is at the top of the page and includes this comment:
"This makes it so that all a port needs to do is set the relevant variables
and "include extmod.mk" and doesn't need to worry about adding anything to
OBJ, CFLAGS, SRC_QSTR, etc."
Without really digging in, I still don't see how/where the extmod parts get included, might just be via settings in the port's mpconfigport.h file.
Unfortunately you need to be a bit of a Makefile guru to figure all this stuff out quickly and work out how it all goes together (and I am not).
Got 1.20 to compile! Thanks @rogloh for pointing me in the right direction.
After I copied all the .c files that define SRC_EXTMOD_C in extmod.mk to the source file list in the makefile, it works.
Obviously, this is probably not the right way...
Maybe I can see the right way from somebody some other makefile?
@Rayman said:
After I copied all the .c files that define SRC_EXTMOD_C in extmod.mk to the source file list in the makefile, it works.
Obviously, this is probably not the right way...
Maybe I can see the right way from somebody some other makefile?
Yes that's the general idea. You want to examine the other port's Makefiles in the various micropython/ports folders. Ideally you see how they did it for their CPU and also if they changed anything from 1.19 to 1.20 using Github commit history to see the modifications for this file. If you have not done that before, just click on the file in the tree at the left to navigate to the Makefile of interest in Github which views the file, then choose the "History" link/button on the check in comment line next to the little timer icon to browse each commit that recently changed it. You'll then see how it changed. You may also want to select the 1.20 tag first instead of Master in the file tree.
Confused myself for a second cause was showing ver. 1.19. But, that was just because it's baked into mpconfig.h that was just copied from 1.19.
Then, having an issue while testing flashing... It would not work once flashed. But, that seems to have magically fixed itself, no idea what is going on there...
Anyway, what is strange though is once flashed and connected by USB cable get a flash of strange characters at start up.
If powered by barrel connector only, don't get it. Not sure seen this before...
Ok, so 1.21 compiles, and sorta works by pye is broke and says "no module named 'ure'"
Just great. For this version seems they decided to rename all the built in modules to remove the 'u'.
This seems to have been controversial and it's not making me happy either...
Congrats Ray! Great reward for your tenacious effort
Possibly great timing because the monthly MicroPython meetup is this coming Wednesday, its done both live and on zoom, but I think its around 5am your time unfortunately. If you're ok with me doing a quick P2 update on where you've gotten to, then I'm happy to do that, or if you're an early riser and want to, you could do it yourself. I'm planning to be there and I think Brian might be too.
Happy to do some testing/verification if you post or email the binaries
Comments
Here are the makefiles and makelogs from 1.19 and 1.20.
Getting to 1.19 was just a matter of changing a few folder names in the makefile, as I recall.
1.20 has all kind of problems.
Here's one strange compiler thing that I don't understand in the 1.20.0 version that works in 1.19
I don't know how exactly it works, but the compiler somehow knows to convert FFCONF_H into "lib/oofatfs/ffconf.h" with 1.19.
But, won't do it in 1.20.0 and gives the wrong configuration file error. So, I just manually stuck it in there.
That got me past one hurdle and into mess of more difficult looking things...
Took the liberty of sprucing up the CPU name and moving board name to platform.h, so can be easily changed to match a particular P2 board
Oh cool thanks for posting those Ray
@Rayman
I think by adding ffconf.h manually you probably broke the linking stage as you appear to have multiple definitions related to VFS APIs now. If you look at the makefile output for 1.19 you'll see these defines everywhere passed into the individual compiler commands. For some reason they are not in the 1.20 makefile and this might be a problem.
-DFFCONF_H=\"lib/oofatfs/ffconf.h\"
This ffconf.h include file should already be setup to only be included once through its own means (using #ifndef etc) and not generate multiple definitions which your forced change may have introduced.
Your other 1.20 linker errors at the end of the log seem to also be related to the disk and vfs type functions are also likely to be related to ffconf.h IMO or maybe they changed the way file IO stuff is built in the Makefile and it needs to be customized to suit that - check the differences with how the other platforms do it perhaps.
I also noticed a bunch of files stored the extmod folder from line 265 to 349 in the 1.19 makefile output do not appear in the 1.20 version. Maybe the MP guys rearranged this area but it seems strange to not build these and they also relate to disk/vfs stuff. So that might be something missing in the build config. Similarly this make step in 1.19 was not done in 1.20:
mkdir -p build/extmod/
Hope that helps a bit but you'll just have to mess about a bit more to fix the linker errors. It should work once you've figured it out.
These were those missing files not getting built in 1.20, you can see that the vfs stuff is also missing which was one of the problems with the undefined references in the linker output of 1.20.
CC ../../extmod/moduasyncio.c
CC ../../extmod/moductypes.c
CC ../../extmod/modujson.c
CC ../../extmod/moduos.c
CC ../../extmod/modure.c
CC ../../extmod/moduzlib.c
CC ../../extmod/moduheapq.c
CC ../../extmod/modutimeq.c
CC ../../extmod/moduhashlib.c
CC ../../extmod/moducryptolib.c
CC ../../extmod/modubinascii.c
CC ../../extmod/virtpin.c
CC ../../extmod/machine_bitstream.c
CC ../../extmod/machine_mem.c
CC ../../extmod/machine_pinbase.c
CC ../../extmod/machine_signal.c
CC ../../extmod/machine_pulse.c
CC ../../extmod/machine_pwm.c
CC ../../extmod/machine_i2c.c
CC ../../extmod/machine_spi.c
CC ../../extmod/modbluetooth.c
CC ../../extmod/modussl_axtls.c
CC ../../extmod/modussl_mbedtls.c
CC ../../extmod/moduplatform.c
CC ../../extmod/modurandom.c
CC ../../extmod/moduselect.c
CC ../../extmod/moduwebsocket.c
CC ../../extmod/modwebrepl.c
CC ../../extmod/modframebuf.c
CC ../../extmod/vfs.c
CC ../../extmod/vfs_blockdev.c
CC ../../extmod/vfs_reader.c
CC ../../extmod/vfs_posix.c
CC ../../extmod/vfs_posix_file.c
CC ../../extmod/vfs_fat.c
CC ../../extmod/vfs_fat_diskio.c
CC ../../extmod/vfs_fat_file.c
CC ../../extmod/vfs_lfs.c
CC ../../extmod/utime_mphal.c
CC ../../extmod/uos_dupterm.c
CC ../../shared/libc/abort_.c
CC ../../shared/libc/printf.c
@rogloh I barely have any idea what is going on here for sure.. think u r saying that I need to do make in exmod first?
No I am just saying it doesn't appear that the Makefile even made these files, which it did in 1.19 - it's likely a problem for the linker if a bunch of files are now missing.
Unless it's documented that something changed I'd expect you to only need to do a single make operation but why is this step not happening for 1.20? I don't know either as I've only built up to 1.13. Perhaps it's a new config file setting in mpconfigport.h etc that needs to be altered/included or some other thing they changed that's now missing in the build files for the RISC-V variant?
Hmm. Wonder if it’s something to do with “make submodules”
I wonder if these are the things that got converted to .mpy
I think you gave me a big clue here though @rogloh
Looks like it was py.mk where that used to happen
Ok I give up again…
Looking through the changes from 1.19 to 1.20 it looks like they refactored the whole EXTMOD thing.
https://github.com/micropython/micropython/commit/d6d87225585a9494093d791c807bce652e4c82d8#diff-30cef9b231b7db5b93a399e2a6cd41af81f46f2fbf5c428559219667779ec152
Perhaps your Makefile might need some updating to suit the new scheme. All the other port's Makefiles seemed to have been updated from 1.19 to 1.20 so if you were still using something based on your 1.19 RISC-V variant it probably won't work. Take a look at the pattern of how each ports Makefile changed (e.g. for stm32/esp8266/samd etc) in the diffs above and see if you can do the same or similar changes for your RISC-V one.
A pertinent comment from the checkin above is at the top of the page and includes this comment:
"This makes it so that all a port needs to do is set the relevant variables
and "include extmod.mk" and doesn't need to worry about adding anything to
OBJ, CFLAGS, SRC_QSTR, etc."
Without really digging in, I still don't see how/where the extmod parts get included, might just be via settings in the port's mpconfigport.h file.
Unfortunately you need to be a bit of a Makefile guru to figure all this stuff out quickly and work out how it all goes together (and I am not).
@Rayman It looks like you might need to add the line
include $(TOP)/extmod/extmod.mk
into your Makefile portion as below because yours was missing unlike the other ports which changed in this checkin: https://github.com/micropython/micropython/commit/c118b5d0e428094bd64a003d97b078c2d7c7500d
So your Makefile should have this:
include $(TOP)/py/py.mk
include $(TOP)/extmod/extmod.mk
ifeq ($(CROSS), 1)
CROSS_COMPILE = $(RV32_PREFIX)
endif
INC += -I.
INC += -I$(TOP)
INC += -I$(BUILD)
There could still be other missing parts that have changed in other Makefile checkins for 1.20 from 1.19, I don't know.
@rogloh I think this helps and I also see need to add third party flags on this like to make it work:
CFLAGS = $(INC) -Wall -std=c99 $(CFLAGS_RISCV) $(COPT) $(CFLAGS_MOD) $(CFLAGS_THIRDPARTY)
Got 1.20 to compile! Thanks @rogloh for pointing me in the right direction.
After I copied all the .c files that define SRC_EXTMOD_C in extmod.mk to the source file list in the makefile, it works.
Obviously, this is probably not the right way...
Maybe I can see the right way from somebody some other makefile?
Spoke a little too soon... It compiles and runs but the special modules like os and ones I made are missing...
Good news about getting it to compile. You probably just need to find the right variable to update to include your custom files.
Yes that's the general idea. You want to examine the other port's Makefiles in the various micropython/ports folders. Ideally you see how they did it for their CPU and also if they changed anything from 1.19 to 1.20 using Github commit history to see the modifications for this file. If you have not done that before, just click on the file in the tree at the left to navigate to the Makefile of interest in Github which views the file, then choose the "History" link/button on the check in comment line next to the little timer icon to browse each commit that recently changed it. You'll then see how it changed. You may also want to select the 1.20 tag first instead of Master in the file tree.
@rogloh Figured that part out.
Also, think now need to add this line to bottom of user modules to have them work:
MP_REGISTER_MODULE(MP_QSTR_p2, p2_module);
It says you have to do this in the docs, but wasn't doing it until now...
Wow, os and sdcard access is broke badly...
phew, got sd working again
Was looking at a dead end for a second...
Copied over mpconfig.h from 1.19 and that seems to have fixed it.
Had to add a couple new things from the 1.20 version though to get it to compile.
Congrats Ray and Roger, excellent news
pye reports 213632 bytes memory available. down about 20k seems...
213 kiB is great. We need Brian's memory displayer up and running again
Confused myself for a second cause was showing ver. 1.19. But, that was just because it's baked into mpconfig.h that was just copied from 1.19.
Then, having an issue while testing flashing... It would not work once flashed. But, that seems to have magically fixed itself, no idea what is going on there...
Anyway, what is strange though is once flashed and connected by USB cable get a flash of strange characters at start up.
If powered by barrel connector only, don't get it. Not sure seen this before...
Ok, so 1.21 compiles, and sorta works by pye is broke and says "no module named 'ure'"
Just great. For this version seems they decided to rename all the built in modules to remove the 'u'.
This seems to have been controversial and it's not making me happy either...
Took the 'u' out of some things in mpconfigport.h and pye seems OK again:
I have a feeling it's not 100% the way it is supposed to be because there is still a lot of 'u' in stuff there, but seems to be working AFAIK
Think have 1.22.0 working, just needed to comment out something in modmachine.c
// Power related functions. //RJA { MP_ROM_QSTR(MP_QSTR_idle), MP_ROM_PTR(&machine_idle_obj) },Latest 1.22.2 now working! Guess can declare victory for now...
Maybe I'll try to post some binaries somewhere...
Congrats Ray! Great reward for your tenacious effort
Possibly great timing because the monthly MicroPython meetup is this coming Wednesday, its done both live and on zoom, but I think its around 5am your time unfortunately. If you're ok with me doing a quick P2 update on where you've gotten to, then I'm happy to do that, or if you're an early riser and want to, you could do it yourself. I'm planning to be there and I think Brian might be too.
Happy to do some testing/verification if you post or email the binaries