Large read-only data tables

macca

Hi,

I'm exploring what possibilities there are to easily manage large read-only data tables in eeprom using the existing tools (openspin, propeller-load, etc.).
My ideal solution would be to have everything, code and data, on a single binary file written to the eeprom. I see that openspin allows eeprom sizes greater than 32k however I don't see a way to write this large binary file to the eeprom using propeller-load (looks limited to 32k). With openspin there is also the problem of the dat files order that I was able to partially fix by loading the data tables file as the last object, however any spin code in that file will be located outside the ram range so can't be used.

I looked at the gcc option, but I can't see a way to set a data block as eeprom data. xmm may be a possibility, however the default is to have everything in eeprom, I see there are directives to store code and data to hub ram, but I would like to do the opposite, have everthing in hub ram except some data in eeprom.

Any suggestion on how to do something like that using the existing tools ?

Speaking of openspin, it throws a segmentation fault when setting the -u option to remove unused methods along with the eeprom size parameter. It also doesn't seems to have a listing option like bstc -ls so it is a nightmare trying to understand where each object is located in the binary file.

DavidZemon

For PropGCC, i think you might be able to declare your table in a file named "ecogc". Typically this is used for writing drivers that only need to be loaded once (so it makes sense to store the data in EEPROM) but it should work nicely for you too. You'll then need to use the same pointer magic for getting your array address as is used to start an ecogc driver.

https://github.com/parallaxinc/propgcc/tree/master/demos/cog_loader

LoopyByteloose

Much depends of how fast you need the data.

Awhile back, I tried to set up a big lookup table of delays in the EEPROM for a Forth project and the serial interface was too slow to do a proper job. So I ended up point the whole table in Hubram where the timing wasn't thrown off by the added conversion from 8bit serial to 32bit parallel. It wasted a heck of a lot of the HubRam, but the performance with spot on.

It is different story if the tables are sitting in EEPROM to be loaded into HubRAM at start-up. IOW, you can't get a stepper motor to do fasting timing loops directly from a table in EEPROM

macca

The ecog way seems a good start, I did some quick tests and it seems to be limited to 1984 bytes for each source / data table (2048 bytes cog ram, minus the predefined registers), I think I can work around this as long as the sections are contiguous. Is there a way to make the compiler ignore the size ?

Speed is not an issue for now so eeprom read would be fine.

LoopyByteloose

I am not really a fan of Gcc and prefer Forth on the Propeller.

I managed to have a table of 4096 32bit Longs (16Kbytes or 50% of it) in HubRam using Forth. The actual Lookup call was cost about 25useconds that was deducted from each entry in the table. The table was created in Excel and then merely inserted into the Forth dictionary that was included in a .spin file

Forth is very, very fast and direct. The only thing faster is PASM itself. To maintain speed, 32bit Longs are better than 8bit bytes. The CPU doesn't have to unpackage the value into 4 items. For a stepper motor to just 'get, use,and do again' this is ideal. There were two stepper motors driven from the same table on different cogs and triggered independently.

macca

Just an update after some more experiments.

The .ecog section works well for my needs, as I said before each sections is limited to 1984 bytes and requires to split larger data arrays so each chunk is smaller than the maximum allowed. This raises the problem of storing the chunks in contiguous locations so the code can read all data using a single start address. I found that the linker places the sections in reverse order as they appear in a source file, so had to do a bit more mangling to get all data orderer correctly.

For example, for a source with:

int32_t table_01[7][15] __attribute__((section("table_01.ecog"))) = {
...
};

int32_t table_02[7][15] __attribute__((section("table_02.ecog"))) = {
...
};

int32_t table_03[7][15] __attribute__((section("table_03.ecog"))) = {
...
};

It produces the following map:

table_03.ecog   0x0000000000000000      0x1a4 load address 0x00000000c0000000
 table_03.ecog  0x0000000000000000      0x1a4 /tmp/ccu3pWne.o
                0x0000000000000000                _table_03

table_02.ecog   0x0000000000000000      0x1a4 load address 0x00000000c00001a4
 table_02.ecog  0x0000000000000000      0x1a4 /tmp/ccu3pWne.o
                0x0000000000000000                _table_02

table_01.ecog   0x0000000000000000      0x1a4 load address 0x00000000c0000348
 table_01.ecog  0x0000000000000000      0x1a4 /tmp/ccu3pWne.o
                0x0000000000000000                _table_01

To have the data contiguous, in the order they should be read, you need to place them in the reverse order in the source.
I only hope that this isn't a side effect of the current gcc release or the environment or whatever...

It would be nice to have a .eeprom section for data with arbitrary size. I have analyzed the binutils source a bit, I found where the .ecog sections are defined (in the propeller.em file), I'm not very familiar with it but looks simple to add another section, I think I'll experiment a bit.

Heater.

macca,

I don't think you should ever rely on the linker putting data consecutively into memory. I have stumbled on this before when I found that building with optimizations switched on changed the order of arrays in memory and optimization off did not. Which made debugging interesting

And, as you say, different compilers may do things differently. You are in the world of C's undefined behaviour.

It might be better to tell the language what you want. If you want all that data laid out consecutively in memory then define it that way, one huge array. If you want to then deal with parts of that data in separate chunks define them.

Something like this:

// The big table containing all data consecutively
int32_t table[21][15] = { /* stuff .... */ };

// A type that lets us access chunks of data within the big table
typedef int32_t  (*table_t)[7][15] ;

// Intialize the chunks
table_t table_00 = (table_t)table[0];
table_t table_01 = (table_t)table[7];
table_t table_02 = (table_t)table[14];

// Use the chunks   
int32_t a = (*table_00)[0][0];
int32_t b = (*table_01)[2][3];

macca

Heater. wrote: »

If you want all that data laid out consecutively in memory then define it that way, one huge array.

Yes, I would do that, if the .ecog sections were not limited to 1984 bytes each!

But, I think I found the solution: the .drivers section.
This is where the linker puts the .ecog sections but is not limited (well, if I read the linker script correctly it is limited to 1 megabyte...)

This code:

int32_t table_01[35][15] __attribute__((section(".drivers"))) = {
...
}

Produces, this map:

.drivers        0x00000000c0000000      0x834
 *(.drivers)
 .drivers       0x00000000c0000000      0x834 /tmp/ccjVL8Vj.o
                0x00000000c0000000                _table_01
                0x0000000000000318                __load_start_kernel = LOADADDR (.lmmkernel)
                0x0000000000000310                ___CTOR_LIST__ = ADDR (.ctors)
                0x0000000000000314                ___DTOR_LIST__ = ADDR (.dtors)

I'm a bit puzzled that __load_start_kernel seems located after the drivers section but its address is correct (in hub ram) and everything works correctly.

BTW, the propeller-load code from https://github.com/dbetz/propeller-gcc is missing the eeprom_cache.dat requires to load data to the upper 32k of the eeprom. I have used the file that comes with the Parallax distribution and works.