The difference in fread(buf, 1000, 1, stream) and fread(buf, 1, 1000, stream) is, that in the first case you get only one chunk of 1000 bytes or nothing, if the file is smaller and in the second case you get everything in the file less than and up to 1000 bytes.
That is not actually the case. fread is allowed to read a partial elements (and not tell you about it). It's just broken.
The difference in fread(buf, 1000, 1, stream) and fread(buf, 1, 1000, stream) is, that in the first case you get only one chunk of 1000 bytes or nothing, if the file is smaller and in the second case you get everything in the file less than and up to 1000 bytes.
That is not actually the case. fread is allowed to read a partial elements (and not tell you about it). It's just broken.
It is not "broken". What an implementation does in such error situations is "indeterminate". But that's entirely normal and to be expected.
But what the function returns even in such cases is perfectly well defined.
But that's not an error, that's just what the function does when it reaches the end of the file. It reads the partial record into your buffer and then forgets about it. That's rarely useful behaviour that unnecessarily adds a divide instruction where it isn't needed.
C library has lots of stuff that's just dumb or broken (gets, anyone?), it's fine, they designed it in the stone age when no one really knew how to do it better.
C library has lots of stuff that's just dumb or broken (gets, anyone?), it's fine, they designed it in the stone age when no one really knew how to do it better.
There's good reasons why C is still wildly popular, despite its flaws (which few would deny). Even on the Propeller - which is a bleeding edge microcontroller - C is still the most functional language on offer.
However, this is now getting off-topic. If you want to continue the discussion I'm happy to do so, but I'd suggest starting a separate thread.
Is this to say that the -c option compiles a binary image that is identical to what would be found in the Flash after it was written by the loader? In the past, the binary image portion had holes that were filled in by the flash loader. I need a complete binary that I can send to a customer who will put in on an SD card for self-updating.
@JonnyMac said:
Is this to say that the -c option compiles a binary image that is identical to what would be found in the Flash after it was written by the loader? In the past, the binary image portion had holes that were filled in by the flash loader. I need a complete binary that I can send to a customer who will put in on an SD card for self-updating.
The flash loader that is prepended to the compiled image programs the flash and prepends a loader that always pulls the main app code in on boot-up.
To compile with flash:
PNut_v47 filename.spin2 -cf
That makes a binary filename.bin file.
Then, to download the binary and have it program the flash:
PNut_v47 filename.bin -b
So, you give the customer the .bin file and PNut app, plus maybe a .bat file to execute the command above. They plug the PropPlug into your product and then run the .bat file to do the firmware update.
So, you give the customer the .bin file and PNut app,
No. I want to give my customer a clean BIN file that they can put onto the SD card in their product. One boot-up the app looks for the update and copies it to flash if its there (the app then removes it from the SD).
Please, just give us a clean BIN like Propeller Tool does for the P1.
In the P1 -- and what we want to do with the P2 -- is download the BIN file via XBee to the product (which has an SD). For that reason I want dead code removal and an image that will just run (like the P1 BIN file)
Agree that it's a far nicer experience for an end user customer if they can avoid needing to install vendor or other special software tools for doing updates. Especially if there are lots of them and they are not necessarily especially tech savvy. I'd expect more of them would be able to put a downloaded/emailed file attachment onto an SD card but not all would necessarily have Prop-Plugs or know anything about what PNut/Loadp2 is etc.
Agree with @JonnyMac
Finding the flashing very confusing/difficult, especially with non-Parallax boards/software.
Seems like was so much easier with P1.
@Rayman said:
Finding the flashing very confusing/difficult, especially with non-Parallax boards/software.
Recently loadp2 gained a -FLASH flag that JustWorksTM. It was slightly silly before that.
The confusion here stems from the fact that the P2 ROM does not just load the whole flash image automatically. It only reads the first block and then that has to contain a bootloader stub to read the rest.
So the binary that comes out of the compiler is useful for serial download or _BOOT_P2.BIX SD boot, but can't just be dumped into a flash chip as-is.
So @JonnyMac is actually asking for the opposite of a "clean binary that just runs", one that has extra guff specific to Chip's flash boot stub prepended onto it, so he can dump it into flash without the (trivial) extra work of figuring out the file size and checksum.
For the particular usecase of self-updating the flash in the field, I'd really urge for modifying the boot stub so after a failed update it can fall back to a second copy of the software instead of just failing to boot.
@Rayman said:
Thanks @Wuerfel_21
Remembering that now…
Anybody know why the P2 rom doesn’t load like the P1 rom?
IIRC Making it more tolerant to different flash chip types + more flexibility in how the data is loaded.
The way that the P1 does it is kinda weird, too. It always launches into the Spin interpreter, so if you want to run your own ASM you need at least the tiniest bit of spin bytecode to make it happen.
@Rayman said:
Ok, I’m still with @JonnyMac though
Just give me a binary that I can copy to flash myself.
That part is not hard.
I proved it isn't by writing a small Python program that removes the flash-writing header and plugs the holes left in the code (these must be plugged so the P2 will boot the code from flash).
#!/usr/bin/env python
# -*- coding: utf-8 -*-
#
# p2img.py
# -- started 01 SEP 2024
# -- updated 01 SEP 2024
#
# By Jon McPhalen with big assists from
# - w3schools.com
# - stackoverflow.com
# - ChatGPT
#
import sys
import os
import struct
import time
def process_file(infile):
outfile = os.path.splitext(infile)[0]+'.p2img'
print(f'Converting {infile} to {outfile}')
filesize = os.path.getsize(infile)
print(f'- Input file size is {filesize} bytes')
imgsize = filesize - 0x160
print(f'- Output file size is {imgsize} bytes')
sectors = (imgsize + 4095) // 4096
print(f'- P2 image requires {sectors} flash sectors in P2')
pages = (imgsize + 255) // 256
print(f'- P2 image occupies {pages} flash pages')
print()
print('Loader Segment Touch-up')
applongs = (imgsize - 0x90) // 4
print(f'- ${applongs:08X} longs in application segment')
# open file
# skip past flash programmer segment
# convert to list of longs
with open(infile, 'rb') as file:
file.seek(0x160)
ibytes = file.read()
n = len(ibytes) // 4
ilongs = list(struct.unpack(f'{n}I', ibytes))
# calculate & neg sum of longs in application
appsum = 0
for x in range(0x24, len(ilongs)):
appsum += ilongs[x]
appsum = 0x100000000 - (appsum & 0xFFFFFFFF) # -appsum (32-bit)
print(f'- ${appsum:08X} is the sum of application longs')
# modify loader image to calculate final loader sum
ilongs[0x20] = applongs
ilongs[0x21] = applongs
ilongs[0x22] = appsum
loadersum = 0
for x in range(0, 256):
loadersum += ilongs[x]
loadersum = 0x100000000 - (loadersum & 0xFFFFFFFF) # -loadersum (32-bit)
print(f'- ${loadersum:08X} is the loader sum')
# update loader sum and validate checksum
ilongs[0x23] = loadersum
checksum = 0
for x in range(0, 256):
checksum += ilongs[x]
checksum &= 0xFFFFFFFF
print(f'- ${checksum:08X} is the checksum')
if checksum == 0x706F7250:
print('- Pass!')
else:
print('- Fail... Aborting.')
return -1
# convert modified longs to bytes and write to output file
obytes = struct.pack(f'{len(ilongs)}I', *ilongs)
with open(outfile, 'wb') as file:
file.write(obytes)
print('- Conversion complete!')
time.sleep(0.2) # let terminal finish
return 0
def check_file(fname):
if os.path.exists(fname):
return 0
else:
print(f'Error: {fname} not found')
return -1
def show_help():
print('P2IMG\nConvert PNut Binary to P2IMG file')
print('Error: Invalid number of arguments')
print('-- Use: python p2img.py IN_FILE')
def main(args):
if len(args) == 2:
if check_file(args[1]) != 0:
return -2
if process_file(args[1]) == 0:
return 0
else:
return -3
else:
show_help()
return -1
if __name__ == '__main__':
sys.exit(main(sys.argv))
The Flash Image, not a Binary Image. I use a Binary image named _BOOT_P2.BIX to boot my light controllers. He needs an image that can be copied directly to flash.
@JonnyMac said:
For that reason I want dead code removal and an image that will just run.
Dead code removal +100. Then I can finally start to write a sample graphics lib that won't have to bloat the P2.
Rogloh, are you interested in Spin2 method removal, or PASM code removal?
Well primarily SPIN2 method removal, but automatically based on what is called in the object hierarchy, not done by finely grained control with a conditional #ifdef for every method for example, that's too tedious at that level once the size of the library gets very large. Hopefully any inline PASM defined in these methods could also be culled.
For example if we wanted to make a graphics library that contains multiple methods for drawing items such as points, circles, lines, and rectangles, and a bunch of other drawing capabilities, and the client application code just so happens to only ever draw rectangles, we don't want every single library method to be included in the build.
Flexspin already removes methods that are never referenced, but PNut does not do that and just you get everything defined in the object, creating code bloat for library style objects. Bloat like that would turn people off using these larger libraries, especially when memory constrained so no one can really provide these libraries with the expectation that people will use them. The only solution right now is to only release these to operate under flexspin (which I'd prefer not to do).
The Flash Image, not a Binary Image. I use a Binary image named _BOOT_P2.BIX to boot my light controllers. He needs an image that can be copied directly to flash.
Understood. PNut is currently saving a raw binary image file if you just use "filename -c".
I didn't write the uSD loader that is in the P2 ROM, so I am not sure how it works, but my understanding is that if you put a binary file on a formatted SD card called "_BOOT_P2.BIX", it will boot from that.
Perhaps this does not answer your question, though.
Just to remember that Spin Tools IDE has most of the features you are looking for, unused method removal, P2 flash image export (thanks to @JonnyMac advice), preprocessor, and many others.
I know that PNut is the reference implementation and officially supported, and that Spin Tools IDE may be a bit behind the latest implementations, may show some bugs, certainly is not perfect, but I'm doing my best to make it work realiably and better than anything else. So, give it a try.
@JonnyMac said:
For that reason I want dead code removal and an image that will just run.
Dead code removal +100. Then I can finally start to write a sample graphics lib that won't have to bloat the P2.
Rogloh, are you interested in Spin2 method removal, or PASM code removal?
Well primarily SPIN2 method removal, but automatically based on what is called in the object hierarchy, not done by finely grained control with a conditional #ifdef for every method for example, that's too tedious at that level once the size of the library gets very large. Hopefully any inline PASM defined in these methods could also be culled.
For example if we wanted to make a graphics library that contains multiple methods for drawing items such as points, circles, lines, and rectangles, and a bunch of other drawing capabilities, and the client application code just so happens to only ever draw rectangles, we don't want every single library method to be included in the build.
Flexspin already removes methods that are never referenced, but PNut does not do that and just you get everything defined in the object, creating code bloat for library style objects. Bloat like that would turn people off using these larger libraries, especially when memory constrained so no one can really provide these libraries with the expectation that people will use them. The only solution right now is to only release these to operate under flexspin (which I'd prefer not to do).
Method removal may be reasonably simple to achieve. I think it would look like this:
Method dependencies are any of the following occurrences within PUB/PRI methods:
1) Identify all inter-method dependencies within each file, starting from each PUB method.
2) Identify all inter-object dependencies within each file by noting which PUB/PRI methods referenced which child OBJects' PUB methods.
3) After top-level compilation (compilation started at the bottom levels), a complete list of all dependencies can be assembled, revealing dead objects and methods within each file.
4) Recompile, but this time skip over all dead OBJ's, PUB's, and PRI's in each file.
Maybe it would be good to also allow for local indented CON/VAR/DAT sub-blocks to be defined within PUB/PRI blocks, so that they could be skipped, too, along with their associated unused methods.
Not sure if this is a complete approach, but it might be.
It's a starting approach, which is good. Would the inline PASM blocks also get removed if the method defining them goes away? I take it that code is stored in the bytecode for the method, not tacked onto some DAT section, although I don't know the internals.
Perhaps the conditional pre-processor work can also be part of it somehow if the file is going to need to be parsed multiple times anyway, assuming it can be leveraged. If not, keep it separate.
@rogloh said:
It's a starting approach, which is good. Would the inline PASM blocks also get removed if the method defining them goes away? I take it that code is stored in the bytecode for the method, not tacked onto some DAT section, although I don't know the internals.
Perhaps the conditional pre-processor work can also be part of it somehow if the file is going to need to be parsed multiple times anyway, assuming it can be leveraged. If not, keep it separate.
Yes, inline PASM code is inserted right into the method code, not a DAT section.
Comments
That is not actually the case. fread is allowed to read a partial elements (and not tell you about it). It's just broken.
It is not "broken". What an implementation does in such error situations is "indeterminate". But that's entirely normal and to be expected.
But what the function returns even in such cases is perfectly well defined.
But that's not an error, that's just what the function does when it reaches the end of the file. It reads the partial record into your buffer and then forgets about it. That's rarely useful behaviour that unnecessarily adds a divide instruction where it isn't needed.
C library has lots of stuff that's just dumb or broken (
gets, anyone?), it's fine, they designed it in the stone age when no one really knew how to do it better.There's good reasons why C is still wildly popular, despite its flaws (which few would deny). Even on the Propeller - which is a bleeding edge microcontroller - C is still the most functional language on offer.
However, this is now getting off-topic. If you want to continue the discussion I'm happy to do so, but I'd suggest starting a separate thread.
Jon, I got this fixed. It was an oversight that it didn't work, in the first place. I will get a new v47 out soon with this fix.
I uploaded a new v47 into the OBEX (see top of thread).
PNut_v47.exe can now load and run binary files from the command line.
I also added the #register syntax, so #pr0 returns $1D8 as expected, instead of an error.
Is this to say that the -c option compiles a binary image that is identical to what would be found in the Flash after it was written by the loader? In the past, the binary image portion had holes that were filled in by the flash loader. I need a complete binary that I can send to a customer who will put in on an SD card for self-updating.
The flash loader that is prepended to the compiled image programs the flash and prepends a loader that always pulls the main app code in on boot-up.
To compile with flash:
PNut_v47 filename.spin2 -cf
That makes a binary filename.bin file.
Then, to download the binary and have it program the flash:
PNut_v47 filename.bin -b
So, you give the customer the .bin file and PNut app, plus maybe a .bat file to execute the command above. They plug the PropPlug into your product and then run the .bat file to do the firmware update.
No. I want to give my customer a clean BIN file that they can put onto the SD card in their product. One boot-up the app looks for the update and copies it to flash if its there (the app then removes it from the SD).
Please, just give us a clean BIN like Propeller Tool does for the P1.
In the P1 -- and what we want to do with the P2 -- is download the BIN file via XBee to the product (which has an SD). For that reason I want dead code removal and an image that will just run (like the P1 BIN file)
Agree that it's a far nicer experience for an end user customer if they can avoid needing to install vendor or other special software tools for doing updates. Especially if there are lots of them and they are not necessarily especially tech savvy. I'd expect more of them would be able to put a downloaded/emailed file attachment onto an SD card but not all would necessarily have Prop-Plugs or know anything about what PNut/Loadp2 is etc.
Dead code removal +100. Then I can finally start to write a sample graphics lib that won't have to bloat the P2.
Agree with @JonnyMac
Finding the flashing very confusing/difficult, especially with non-Parallax boards/software.
Seems like was so much easier with P1.
Recently loadp2 gained a
-FLASHflag that JustWorksTM. It was slightly silly before that.The confusion here stems from the fact that the P2 ROM does not just load the whole flash image automatically. It only reads the first block and then that has to contain a bootloader stub to read the rest.
So the binary that comes out of the compiler is useful for serial download or
_BOOT_P2.BIXSD boot, but can't just be dumped into a flash chip as-is.So @JonnyMac is actually asking for the opposite of a "clean binary that just runs", one that has extra guff specific to Chip's flash boot stub prepended onto it, so he can dump it into flash without the (trivial) extra work of figuring out the file size and checksum.
For the particular usecase of self-updating the flash in the field, I'd really urge for modifying the boot stub so after a failed update it can fall back to a second copy of the software instead of just failing to boot.
Hopefully it now allows image sizes up to the full 512kB.
Thanks @Wuerfel_21
Remembering that now…
Anybody know why the P2 rom doesn’t load like the P1 rom?
Oh, maybe for speed?
Perhaps starting in RC clock mode would make boot too slow?
That must be it.
IIRC Making it more tolerant to different flash chip types + more flexibility in how the data is loaded.
The way that the P1 does it is kinda weird, too. It always launches into the Spin interpreter, so if you want to run your own ASM you need at least the tiniest bit of spin bytecode to make it happen.
Ok, I’m still with @JonnyMac though
Just give me a binary that I can copy to flash myself.
That part is not hard.
It allows bigger image sizes, in fact (riscvp2 can run code directly from flash, which was one impetus for the
-FLASHflag).Rogloh, are you interested in Spin2 method removal, or PASM code removal?
Any chance it’s possible to have one binary that can be loaded by serial or directly to flash?
I proved it isn't by writing a small Python program that removes the flash-writing header and plugs the holes left in the code (these must be plugged so the P2 will boot the code from flash).
#!/usr/bin/env python # -*- coding: utf-8 -*- # # p2img.py # -- started 01 SEP 2024 # -- updated 01 SEP 2024 # # By Jon McPhalen with big assists from # - w3schools.com # - stackoverflow.com # - ChatGPT # import sys import os import struct import time def process_file(infile): outfile = os.path.splitext(infile)[0]+'.p2img' print(f'Converting {infile} to {outfile}') filesize = os.path.getsize(infile) print(f'- Input file size is {filesize} bytes') imgsize = filesize - 0x160 print(f'- Output file size is {imgsize} bytes') sectors = (imgsize + 4095) // 4096 print(f'- P2 image requires {sectors} flash sectors in P2') pages = (imgsize + 255) // 256 print(f'- P2 image occupies {pages} flash pages') print() print('Loader Segment Touch-up') applongs = (imgsize - 0x90) // 4 print(f'- ${applongs:08X} longs in application segment') # open file # skip past flash programmer segment # convert to list of longs with open(infile, 'rb') as file: file.seek(0x160) ibytes = file.read() n = len(ibytes) // 4 ilongs = list(struct.unpack(f'{n}I', ibytes)) # calculate & neg sum of longs in application appsum = 0 for x in range(0x24, len(ilongs)): appsum += ilongs[x] appsum = 0x100000000 - (appsum & 0xFFFFFFFF) # -appsum (32-bit) print(f'- ${appsum:08X} is the sum of application longs') # modify loader image to calculate final loader sum ilongs[0x20] = applongs ilongs[0x21] = applongs ilongs[0x22] = appsum loadersum = 0 for x in range(0, 256): loadersum += ilongs[x] loadersum = 0x100000000 - (loadersum & 0xFFFFFFFF) # -loadersum (32-bit) print(f'- ${loadersum:08X} is the loader sum') # update loader sum and validate checksum ilongs[0x23] = loadersum checksum = 0 for x in range(0, 256): checksum += ilongs[x] checksum &= 0xFFFFFFFF print(f'- ${checksum:08X} is the checksum') if checksum == 0x706F7250: print('- Pass!') else: print('- Fail... Aborting.') return -1 # convert modified longs to bytes and write to output file obytes = struct.pack(f'{len(ilongs)}I', *ilongs) with open(outfile, 'wb') as file: file.write(obytes) print('- Conversion complete!') time.sleep(0.2) # let terminal finish return 0 def check_file(fname): if os.path.exists(fname): return 0 else: print(f'Error: {fname} not found') return -1 def show_help(): print('P2IMG\nConvert PNut Binary to P2IMG file') print('Error: Invalid number of arguments') print('-- Use: python p2img.py IN_FILE') def main(args): if len(args) == 2: if check_file(args[1]) != 0: return -2 if process_file(args[1]) == 0: return 0 else: return -3 else: show_help() return -1 if __name__ == '__main__': sys.exit(main(sys.argv))I think what Jon wants is this:
The Flash Image, not a Binary Image. I use a Binary image named _BOOT_P2.BIX to boot my light controllers. He needs an image that can be copied directly to flash.
Well primarily SPIN2 method removal, but automatically based on what is called in the object hierarchy, not done by finely grained control with a conditional #ifdef for every method for example, that's too tedious at that level once the size of the library gets very large. Hopefully any inline PASM defined in these methods could also be culled.
For example if we wanted to make a graphics library that contains multiple methods for drawing items such as points, circles, lines, and rectangles, and a bunch of other drawing capabilities, and the client application code just so happens to only ever draw rectangles, we don't want every single library method to be included in the build.
Flexspin already removes methods that are never referenced, but PNut does not do that and just you get everything defined in the object, creating code bloat for library style objects. Bloat like that would turn people off using these larger libraries, especially when memory constrained so no one can really provide these libraries with the expectation that people will use them. The only solution right now is to only release these to operate under flexspin (which I'd prefer not to do).
Understood. PNut is currently saving a raw binary image file if you just use "filename -c".
I didn't write the uSD loader that is in the P2 ROM, so I am not sure how it works, but my understanding is that if you put a binary file on a formatted SD card called "_BOOT_P2.BIX", it will boot from that.
Perhaps this does not answer your question, though.
Self Promotion ON
Just to remember that Spin Tools IDE has most of the features you are looking for, unused method removal, P2 flash image export (thanks to @JonnyMac advice), preprocessor, and many others.
I know that PNut is the reference implementation and officially supported, and that Spin Tools IDE may be a bit behind the latest implementations, may show some bugs, certainly is not perfect, but I'm doing my best to make it work realiably and better than anything else. So, give it a try.
Self Promotion OFF
Method removal may be reasonably simple to achieve. I think it would look like this:
method()
object{[]}.method()
@method
@object{[]}.method
1) Identify all inter-method dependencies within each file, starting from each PUB method.
2) Identify all inter-object dependencies within each file by noting which PUB/PRI methods referenced which child OBJects' PUB methods.
3) After top-level compilation (compilation started at the bottom levels), a complete list of all dependencies can be assembled, revealing dead objects and methods within each file.
4) Recompile, but this time skip over all dead OBJ's, PUB's, and PRI's in each file.
Maybe it would be good to also allow for local indented CON/VAR/DAT sub-blocks to be defined within PUB/PRI blocks, so that they could be skipped, too, along with their associated unused methods.
Not sure if this is a complete approach, but it might be.
It's a starting approach, which is good. Would the inline PASM blocks also get removed if the method defining them goes away? I take it that code is stored in the bytecode for the method, not tacked onto some DAT section, although I don't know the internals.
Perhaps the conditional pre-processor work can also be part of it somehow if the file is going to need to be parsed multiple times anyway, assuming it can be leveraged. If not, keep it separate.
Yes, inline PASM code is inserted right into the method code, not a DAT section.
Is this true? Does the "Save Flash File" create a file that can be directly copied into the flash boot chip without modification?