Binary header and checksum?

Peter Jakacki · 2012-09-20 22:43

I want to dump an Intel hex dump of my Tachyon image with all the extensions loaded but when I convert the hex file to binary on the PC the Spin tool rejects it with a bad checksum message. I thought I calculated the Spin tool's checksum correctly but here's my Forth code if someone can look over it or see where I went wrong. Thanks

[B][COLOR=#000000][FONT=Ubuntu Mono]BYTE cksum[/FONT][/COLOR][/B]
[B][COLOR=#000000][FONT=Ubuntu Mono]pub [/FONT][/COLOR][COLOR=#000000][FONT=Ubuntu Mono][B]FIXCKSUM[/B][/FONT][/COLOR][COLOR=#000000][FONT=Ubuntu Mono] ( src cnt -- )[/FONT][/COLOR]
[COLOR=#000000][FONT=Ubuntu Mono]    2DUP + DUP 8 W! 8 + $0A W!        \ vbase = end; dbase = end+8[/FONT][/COLOR]
[COLOR=#000000][FONT=Ubuntu Mono]    $0A W@ 4 + $0E W!                 \ dcurr = dbase+4[/FONT][/COLOR]
[COLOR=#000000][FONT=Ubuntu Mono]     cksum C~ 5 C~                    \ zero our cksum and the header's checksum [/FONT][/COLOR]
[COLOR=#000000][FONT=Ubuntu Mono]      ADO I C@ cksum C+! LOOP          \ add up all the bytes in memory[/FONT][/COLOR]
[COLOR=#000000][FONT=Ubuntu Mono]     $14 cksum C@ - 5 C!              \ adjust and update header's checksum    [/FONT][/COLOR]
[COLOR=#000000][FONT=Ubuntu Mono]    ;[/FONT][/COLOR]

[COLOR=#000000][FONT=Ubuntu Mono]\ Dump memory in Intel hex format so that a binary can be created on a PC [/FONT][/COLOR]
[COLOR=#000000][FONT=Ubuntu Mono]\ but calculate the checksum of a Propeller binary image first[/FONT][/COLOR]
[COLOR=#000000][FONT=Ubuntu Mono]: [/FONT][/COLOR][COLOR=#000000][FONT=Ubuntu Mono][B]IDUMP[/B][/FONT][/COLOR][COLOR=#000000][FONT=Ubuntu Mono] ( src cnt --- )[/FONT][/COLOR]
[COLOR=#000000][FONT=Ubuntu Mono]     2DUP FIXCKSUM[/FONT][/COLOR]
[COLOR=#000000][FONT=Ubuntu Mono]      ADO cksum C~             \ Proceed with Intel hex dump[/FONT][/COLOR]
[COLOR=#000000][FONT=Ubuntu Mono]       CR ":" EMIT[/FONT][/COLOR]
[COLOR=#000000][FONT=Ubuntu Mono]       #16 DUP cksum C+! .BYTE[/FONT][/COLOR]
[COLOR=#000000][FONT=Ubuntu Mono]       I $FF AND cksum C+! [/FONT][/COLOR]
[COLOR=#000000][FONT=Ubuntu Mono]       I 8 SHR $FF AND cksum C+![/FONT][/COLOR]
[COLOR=#000000][FONT=Ubuntu Mono]       I $FFFF AND .WORD[/FONT][/COLOR]
[COLOR=#000000][FONT=Ubuntu Mono]       0 .BYTE I #16 [/FONT][/COLOR]
[COLOR=#000000][FONT=Ubuntu Mono]         ADO I C@ DUP cksum C+! .BYTE LOOP[/FONT][/COLOR]
[COLOR=#000000][FONT=Ubuntu Mono]        cksum C@ NEGATE .BYTE[/FONT][/COLOR]
[COLOR=#000000][FONT=Ubuntu Mono]         #16 +LOOP[/FONT][/COLOR]
[COLOR=#000000][FONT=Ubuntu Mono]     CR ." :00000001FF" CR[/FONT][/COLOR]
[COLOR=#000000][FONT=Ubuntu Mono]     ;[/FONT][/COLOR][/B]

Dave Hein · 2012-09-21 06:04

Can you post a Spin or C version of your routine? It would help to understand what your doing.

Peter Jakacki · 2012-09-21 06:12

I coded from a far more verbose C example from David Betz which used file methods but I don't know where the information came from, I haven't been able to locate any other references. It's not code I'm having a problem with, it's just knowing about any special conditions on what is really just a very primitive checksum. The C code looked like it added up all the bytes in the binary including the checksum byte which I assume was initialized to zero and then it subtracted this from $14 and stored the byte in location 5. What am I missing?

Anyway, for reference this is the code I worked from:

/*
 * program to calculate and set up the checksum in a propeller binary
 * written by David Betz
 */
#include <stdio.h>
#include <stdlib.h>
#include <stdint.h>

/* spin object file header */
typedef struct {
    uint32_t clkfreq;
    uint8_t clkmode;
    uint8_t chksum;
    uint16_t pbase;
    uint16_t vbase;
    uint16_t dbase;
    uint16_t pcurr;
    uint16_t dcurr;
} SpinHdr;

/* target checksum for a binary file */
#define SPIN_TARGET_CHECKSUM    0x14

int main(int argc, char *argv[])
{
    size_t fileSize;
    int chksum, byte;
    SpinHdr hdr;
    FILE *fp;

    /* check the arguments */
    if (argc != 2) {
        fprintf(stderr, "usage: fixchecksum <file>\n");
        return 1;
    }
    
    /* open the file to fix */
    if (!(fp = fopen(argv[1], "r+b"))) {
        fprintf(stderr, "error: can't open '%s'\n", argv[1]);
        return 1;
    }
    
    /* read the existing file header */
    if (fread(&hdr, 1, sizeof(hdr), fp) != sizeof(hdr)) {
        fprintf(stderr, "error: can't read file header\n");
        return 1;
    }
    
    /* if the checksum isn't zero it's already been fixed */
    if (hdr.chksum != 0) {
        fprintf(stderr, "error: file has already been fixed\n");
        return 1;
    }
    
    /* get the file size minus the trailer we will add later */
    fseek(fp, 0, SEEK_END);
    fileSize = ftell(fp);
    
    /* fixup the header to point past the spin bytecodes and generated PASM code */
    hdr.vbase = fileSize;
    hdr.dbase = fileSize + 2*sizeof(uint32_t); // stack markers
    hdr.dcurr = hdr.dbase + sizeof(uint32_t);
    
    /* rewrite the header so the checksum will come out right */
    fseek(fp, 0, SEEK_SET);
    if (fwrite(&hdr, 1, sizeof(hdr), fp) != sizeof(hdr)) {
        fprintf(stderr, "error: can't update file header\n");
        return 1;
    }
    
    /* compute the checksum */
    fseek(fp, 0, SEEK_SET);
    for (chksum = 0; (byte = getc(fp)) != EOF; chksum += byte)
        ;
        
    /* store the checksum in the header */
    hdr.chksum = SPIN_TARGET_CHECKSUM - chksum;
    
    /* write the updated header */
    fseek(fp, 0, SEEK_SET);
    if (fwrite(&hdr, 1, sizeof(hdr), fp) != sizeof(hdr)) {
        fprintf(stderr, "error: can't update file header\n");
        return 1;
    }
        
    /* close the file we're updating */
    fclose(fp);

#if 0
    /* show what we've done */
    printf("clkfreq: %08x\n", hdr.clkfreq);
    printf("clkmode: %02x\n", hdr.clkmode);
    printf("chksum:  %02x\n", hdr.chksum);
    printf("pbase:   %08x\n", hdr.pbase);
    printf("vbase:   %08x\n", hdr.vbase);
    printf("dbase:   %08x\n", hdr.dbase);
    printf("pcurr:   %08x\n", hdr.pcurr);
    printf("dcurr:   %08x\n", hdr.dcurr);
#endif   
    return 0;
}

Dave Hein · 2012-09-21 06:29

The byte sum is computed from 0 to the address before VBASE plus eight extra bytes, which are $FFF9FFFF, $FFF9FFFF. You then subtract this from whatever was at location $5. The two $FFF9FFFF values make up the initial Spin stack frame that is added by the loader. The sum of the extra bytes is $EC, which is -$14.

Peter Jakacki · 2012-09-21 06:51

Dave Hein wrote: »

The byte sum is computed from 0 to the address before VBASE plus eight extra bytes, which are $FFF9FFFF, $FFF9FFFF. You then subtract this from whatever was at location $5. The two $FFF9FFFF values make up the initial Spin stack frame that is added by the loader. The sum of the extra bytes is $EC, which is -$14.

Is this information located somewhere? The binary files that are compiled by the Spin tool don't include the extra longs so why would they need to be there etc etc Subtracting from whatever was at location 5 doesn't seem to make any sense plus the C code seems to indicate that it just overwrites it with --> hdr.chksum = SPIN_TARGET_CHECKSUM - chksum

All the compiled binaries I have looked at just end with the Spin bytecodes and do not include the $FFF9FFFFs

Anyway the FIXCKSUM Forth code is short and concise and also commented.

Clear the cksum byte and also clear the byte at location 5
cksum C~ 5 C~ \ zero our cksum and the header's checksum

This line just sits in the loop indexing from the "src" or 0 in this case for "cnt" bytes or the total length of the binary.
ADO I C@ cksum C+! LOOP \ add up all the bytes in memory to cksum

and here we subtracted from the "magic" $14 and stored the byte at location 5
$14 cksum C@ - 5 C! \ adjust and update header's checksum

Dave Hein · 2012-09-21 07:53

I have included a C program below that will compute the checksum of a Spin binary file. I've run it on my PC, and I always get zero for any binary produced by BST or the Prop tool. The magic $14 is needed because the $FFF9FFFF longs are not included in the binary file. I believe the loader program in ROM adds these values after receiving the binary data, and it also must clear the remaining hub RAM starting at VBASE. For some reason, the checksum includes the $FFF9FFFF longs, probably because it is compute after the hub RAM is initialized.

You have to subtract your computed sum plus $14 from location 5 so that the checksum will be zero. If you had already zeroed out location 5 then you just need to put -(sum+$14) at that location.

#include <stdio.h>

int main(int argc, char **argv)
{
    unsigned char buffer[32768];
    FILE *infile = fopen(argv[1], "r");
    int i, num = fread(buffer, 1, 32768, infile);
    unsigned char checksum = -0x14;

    for (i = 0; i < num; i++) checksum += buffer[i];

    printf("Checksum = %02x\n", checksum);
    return 0;
}

Peter Jakacki · 2012-09-21 08:16

Dave Hein wrote: »
I have included a C program below that will compute the checksum of a Spin binary file. I've run it on my PC, and I always get zero for any binary produced by BST or the Prop tool. The magic $14 is needed because the $FFF9FFFF longs are not included in the binary file. I believe the loader program in ROM adds these values after receiving the binary data, and it also must clear the remaining hub RAM starting at VBASE. For some reason, the checksum includes the $FFF9FFFF longs, probably because it is compute after the hub RAM is initialized.

You have to subtract your computed sum plus $14 from location 5 so that the checksum will be zero. If you had already zeroed out location 5 then you just need to put -(sum+$14) at that location.
#include <stdio.h>

int main(int argc, char **argv)
{
    unsigned char buffer[32768];
    FILE *infile = fopen(argv[1], "r");
    int i, num = fread(buffer, 1, 32768, infile);
    unsigned char checksum = -0x14;

    for (i = 0; i < num; i++) checksum += buffer[i];

    printf("Checksum = x\n", checksum);
    return 0;
}

To keep it simple this is the Spin tool generated binary for a null program with a single pub declaration and no code.
00 1B B7 00 00 6A 10 00 1C 00 24 00 18 00 28 00 0C 00 02 00 08 00 00 00 32 00 00 00
Notice the checksum of $6A and the code ends at $001B so vbase is $001C. This checksum is checked by the Spin tool if you try to load a binary file and bombs out there if it's incorrect so it doesn't even get to be loaded.

Dave Hein · 2012-09-21 08:18

The problem may be with "$14 cksum C@ - 5 C!". In Spin it would be byte[5] := $14 - cksum. Is that what the forth code is doing?

EDIT: So 6A is correct since the sum is 14. So what does your forth code generate? Is it 96, which would be cksum - $14?

Peter Jakacki · 2012-09-21 08:24

Dave Hein wrote: »

The problem may be with "$14 cksum C@ - 5 C!". In Spin it would be byte[5] := $14 - cksum. Is that what the forth code is doing?

Yes, and isn't that what David's C code is doing too?
hdr.chksum = SPIN_TARGET_CHECKSUM - chksum
Where SPIN_TARGET_CHECKSUM is equal to $14.

Dave Hein · 2012-09-21 08:31

Try cksum $14 C@ - 5 C!

Peter Jakacki · 2012-09-21 08:38

That wouldn't work as it would subtract the contents of location $14 from the address of the cksum variable and then store it to location 5. But this isn't an issue of coding as the code I have is correct for the method that I think is correct. Iit's an issue of needing to know the finer details about the checksum and where that information may be found. It may be buried deep within the bowels of the forum. Was it ever published?

Dave Hein · 2012-09-21 08:55

This isn't rocket science. The sum of all the bytes in a binary file should be $14. It that's not what you get then there is a bug in your program. What do you get for your example of
00 1B B7 00 00 6A 10 00 1C 00 24 00 18 00 28 00 0C 00 02 00 08 00 00 00 32 00 00 00
If I add it on the handy dandy Windows calculator I get $14. If you zeroed out byte 5 first you would get $14-$6A = $AA. $14 - $AA = $6A. It's pretty basic math. There must be a bug in your forth code.

Dave Hein · 2012-09-21 11:33

I spent my lunch hour learning a little bit more forth, and I'm guessing your problem isn't the Prop binary checksum, but maybe the checksums that are included on each line of the hex file. Your forth code contains many words that are defined elsewhere. This makes it virtually impossible to determine what it's actually doing without seeing the words it references.

The bytes in each line of the hex file must sum up to zero, so the checksum byte at the end of the line is the negative of the sum of all the previous bytes in the row. As an example, the last byte in the final record is FF, or -1, since the previous bytes add up to 1.

Peter Jakacki · 2012-09-21 19:47

Dave, I keep telling you that I don't have a problem with the code, that is the easiest part, nor do I have a problem with checksums per se. But checksums are calculated differently depending upon the implementer's goals. The Intel hex dump utility's checksum has nothing to do with the Propeller's binary checksum but it's just a method of conveying the binary file to the terminal and besides the Intel hex checksum is clearly documented.

All I have been asking is WHERE is this information about the Prop's binary format and checksum method?

jazzed · 2012-09-21 20:53

Peter Jakacki wrote: »

All I have been asking is WHERE is this information about the Prop's binary format and checksum method?

Parallax has never officially published this information as far as I know. We just had to figure it out. We've certainly asked for it more than once.

Phil Pilgrim (PhiPi) · 2012-09-21 21:22

It's easy enough to find the info through unofficial channels:

http://forums.parallax.com/showthread.php?117526-eeprom-file-format

Parallax can't be expected to think of everything people want to know!

-Phil

Peter Jakacki · 2012-09-22 00:40

Phil Pilgrim (PhiPi) wrote: »

It's easy enough to find the info through unofficial channels:
http://forums.parallax.com/showthread.php?117526-eeprom-file-format

Parallax can't be expected to think of everything people want to know!

-Phil

Turn up your audio and you may hear me breathe a sigh of relief! Thanks for that link Phil and why can't we EXPECT Parallax to think of everything? Chip has just about done that already with P2

Now I know what to look out for so therefore I can adjust my code accordingly.

Peter Jakacki · 2012-09-22 01:11

So I found out there is nothing really wrong with my checksum code then because it is doing what it is doing but I was asking it to compute the checksum on the whole binary image which unfortunately contains volatile DAT areas with registers and buffers etc. So the only way to make a binary file with the correct checksum is do this with the EEPROM image and then do an Intel hex dump of the EEPROM instead. Now I can create "fat" images of Tachyon that I others can download onto their Prop and have it up and fully running with all the extra modules. Of course I also have the option of saving the fat image onto an SD card as well but I wanted to have the basic method available too.

Dave Hein · 2012-09-22 05:15

You could compute the checksum as you write the data out, and then put it at the end. I don't know how your code is structured, but this would work for Spin. You would just have to add a long at the end and add 4 to VBASE in the header.

Cluso99 · 2012-09-22 15:50

Peter, have not had time to follow ths thread. Would the source of the prop rom help? If so, Chip published it and its in the booter.spin section, along with the peanut interpreter.

Peter Jakacki · 2012-10-10 16:48

This post is not about beating a dead horse, instead I have found a live one in the stable! (after reviving the dead one). Going over my Tera-Quads of hard-drives I came across some webpages I saved on this very subject back in 2006. So I processed the page into PDF and I have hereby submitted this evidence to the forum. Then I got sidetracked because......the author of this document is Cliff L. Biffle who wrote the first Forth for the Propeller. Following the links there are a few interesting things on his website such as how to coax an FT232R chip into sending a packet to the PC without having to wait for it's 16ms timeout or a full buffer. He has also been assimilated by the MakerBots, replicators from planet 3D

So where are all these past Prop champs? Whatever happened to deSilva and many others? (I do remember he was copping it a bit hard for his direct manner).

Parallax™ Propeller Binary Format.pdf

Binary header and checksum?

Comments