Simple Method for Storing and Retrieving Data in Flash

Juliet Oscar Echo

I'm working on a project where I need to save sensor data to flash. I'll probably use the 64 mbit SST26VF064B by Microchip. I've used it in projects for over two years now and have always liked it. In the past, when logging data, I would simply write the data one line at a time in comma delimited ASCII format, complete with separate bytes for each comma, carriage, return, etc. For instance, let's say I needed to save temperature, pressure, and voltage one entry/line at a time:

Temp = 13 deg C
Pressure = 5.2 psi
Voltage = 3.2 V

I would write the following data to the Flash one ASCII character at a time:

13,5.2,3.2

That's a total of 10 bytes (not including Carriage Return and/or checksum). It's not very efficient, but it's simple since I can read the Flash directly into a .CSV file. Unfortunately in my latest application I am needing to save large amounts of data per entry for long periods of time. I need a vastly more efficient way of saving data.

What if, instead, I saved the data as follows: (Note: to keep this example simple, just assume I can keep track of my decimal places and none of my data has negative values.)

$0D$34$20

Now instead of 10 bytes, it's only 3 bytes. The issue here is there's no way to mark the end of a data entry/line with Carriage Return. If I'm reading the Flash and I get off by one byte in the three byte data sequence, none of the data will be correct. I can't use the carriage return byte since this is $0D. Every time there is a 13 in my data, things will get thrown off. Is there a simple trick to use in this type of scenario that I'm just not aware of? If anyone knows the solution to this problem, please share or link to it. One idea I had was to use a unique combination of bytes to mark the end of a line of data. This seems cumbersome, and in theory, the data could in rare instances match my unique combination of bytes. Another idea was to check each byte as I write it to the buffer. If it's a 13, change it to a 14. Is it really that big a deal to log 14 deg C when it's 13 deg C outside? I could even add an extra byte to my data entry that logs any of these ghost "13's" so that the applicable 14's gets changed back to a 13 when the data is read back. Let's just hope this extra byte is never a 13.

Thoughts? Ideas?

For those who are interested, here's a good read on integrating the Propeller with Flash

https://www.parallax.com/sites/default/files/downloads/AN012-SRAM-v1.0.pdf

rbehm

You can use something that is often done for sending binary data over serial lines.
You define two special bytes, say DLE and ETX. An ETX is regarded as a delimiter (your CR), but only if it directly follows a DLE, otherwise it is a data byte.
DLE data bytes are written twice.
So assuming DLE=$16, ETX=$03

Data record1 = $11$12$14 is written as $11$12$14$16$03
Data record21 = $11$16$14 is written as $11$16$16$14$16$03

Or look at what PPP/HDLC are using
<https://tools.ietf.org/html/rfc1662&gt;

jmg

Juliet Oscar Echo wrote: »

....

Temp = 13 deg C
Pressure = 5.2 psi
Voltage = 3.2 V

I would write the following data to the Flash one ASCII character at a time:

13,5.2,3.2

That's a total of 10 bytes (not including Carriage Return and/or checksum). It's not very efficient, but it's simple since I can read the Flash directly into a .CSV file. Unfortunately in my latest application I am needing to save large amounts of data per entry for long periods of time. I need a vastly more efficient way of saving data.

What if, instead, I saved the data as follows: (Note: to keep this example simple, just assume I can keep track of my decimal places and none of my data has negative values.)

$0D$34$20

Now instead of 10 bytes, it's only 3 bytes. The issue here is there's no way to mark the end of a data entry/line with Carriage Return. If I'm reading the Flash and I get off by one byte in the three byte data sequence, none of the data will be correct.

What would make you 'get off by 1 byte' ? You can issue new read or write address any time you like, and even read things twice, if you want to make it more noise immune.

Juliet Oscar Echo wrote: »

.. Is there a simple trick to use in this type of scenario that I'm just not aware of?

You can zoom into the bit-level, and define valid ranges of values. - then either unique values, or whole bits, can be used as flags.

You seem to be working to 1 degree C LSB, but what range do you need ? is (eg) -28°C to 99°C enough ? that frees the MSB of one byte, or you can just reserve
Likewise what PSI range will you capture and what voltage range ? 0..12.8V also gives another free-bit.

Peter Jakacki

You have used SD before but failed because of the software which did not flush until it was closed. Well, that is just poor software. SD is Flash and you can use it just like Flash by writing to sectors and if those sectors line up with a FAT32 file then hey presto, you have a CSV file and no practical memory limitations. You can even flush after each line if need be without any worries although I normally use timeouts.

In fact in Tachyon/TAQOZ I open a file as virtual memory of up to 4GB. But for text files normally I preallocate a few megabytes per file but there is nothing stopping you from having a 128MB file or more.

Cluso99

Or you could use $FF as the delimiter, presuming that 255 is not valid.

kwinn

Another option would be to use a byte count at the beginning of each line. To be honest, I don't see the need for a byte count or a delimiter if you have a fixed number of entries of fixed sizes. That would only be necessary if the number entries or the size of the entries varied as they may do when stored as ascii strings.

Rayman

Been a while since I used that chip, but I think the basic unit you can write to there is a "page" of 256 bytes.

I guess I would decide how many bytes I needed for each dataset. For you, sounds like 8 bytes might be good choice.

When erased, all bits of flash are 1s. I'd make it so that any dataset that reads as all $FF is a sign of not being programmed. You might want to start a dataset with a byte like $5E to show it's been written.

Since it's easiest to just program a whole page for each dataset, but a dataset is smaller than a page, I'd keep a page buffer in HUB RAM. Initialize it to all $FF, so that only new data is actually written (1s don't do anything when writing to flash).

With 8 byte dataset, you can do 32 dataset writes to the same page before reinitializing the buffer and starting again at the next page.

Juliet Oscar Echo

Rayman wrote: »

Been a while since I used that chip, but I think the basic unit you can write to there is a "page" of 256 bytes.

I guess I would decide how many bytes I needed for each dataset. For you, sounds like 8 bytes might be good choice.

When erased, all bits of flash are 1s. I'd make it so that any dataset that reads as all $FF is a sign of not being programmed. You might want to start a dataset with a byte like $5E to show it's been written.

Since it's easiest to just program a whole page for each dataset, but a dataset is smaller than a page, I'd keep a page buffer in HUB RAM. Initialize it to all $FF, so that only new data is actually written (1s don't do anything when writing to flash).

With 8 byte dataset, you can do 32 dataset writes to the same page before reinitializing the buffer and starting again at the next page.

Rayman, I actually use the buffer technique as well. I realize my example was very simple with just a few bytes of data. That was just so I could get this discussion going. My actual application has me attempting to save 5 kilobytes/second for up to 10 minutes.

Juliet Oscar Echo

kwinn wrote: »

Another option would be to use a byte count at the beginning of each line. To be honest, I don't see the need for a byte count or a delimiter if you have a fixed number of entries of fixed sizes. That would only be necessary if the number entries or the size of the entries varied as they may do when stored as ascii strings.

Kwinn, I completely agree with you. I'm just trying to write something that is a bit more robust. If the system experiences a power failure during a write, I want it to be very apparent when looking at the data. The same would go for an error in my code that causes the 256 byte buffer (one page of memory on this particular chip) to overflow.

Rayman

You could also think about using the upper 32 kB of the usual 64 kB of EEPROM to help document what has been written to flash...

For example, you could save the packet # as one long value in EEPROM location #1 before writing and then save the same packet # to location #2 after writing.

Then, you'd have a quick way to see what has happened...

Tracy Allen

I too recommend a fixed length binary record, with each field as a preordained number of bytes. That way you can always locate a record and a field by its absolute offset from the start of the file. Following up on Rayman's post, you can store a format string somewhere, to apply to retrieved data.

My own experience is with the Adesto (formerly ATMEL) AT45DB081. It too is a 64Mbit flash in an 8-pin package, addressed by SPI. The AT45... series differs from the SST26 series in that it incorporates dual RAM buffers that shadow the flash array, so it takes a burden off the main processor, which no longer has to give up its own RAM for that purpose. Once a page is full, the processor issues the command to burn to flash and then starts streaming its data into the second RAM buffer in a ping-pong fashion that is well suited to audio or high speed recording.

My original use however was for data logging with the BASIC Stamp, which is highly limited in the RAM department. I wasn't going for anywhere even in the ballpark audio speed, but the extra RAM and Mbits in an 8-pin soic were indispensable. The AT45 RAM can also be used for other purposes simply as extra RAM.