Circular buffer for flight recording
Basil
Posts: 380
Hi All,
Wondering if I could run this past you.
This psuedo code is meant to do the following.
-Record sensor data into a ~1 second circular buffer until launch is detected.
-When launch detected, copy entire circular buffer into dataflash memory page by page (via DF buffer) until complete
-Continue recording directly into DF memory page by page (via DF buffer)
-The data is: 12 bits from acceleromter (stored in long), 12 bits from pressure sensor (stored in long), time in ms from launch (stored in long)
Does anyone see any glaring errors in my thinking?
Thanks for taking the time to look this over [noparse]:)[/noparse]
EDIT: I added a small amount of code to switch between the 2 dataflash buffers. Write time from DF buffer to DF memory is ~3ms but I can write to the 2nd buffer while this is happening in the background.
EDIT 2: Fixed a few big problems and added some documentation about the dataflash.
▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔
-Alec
My our page
Post Edited (Basil) : 9/1/2007 11:23:11 PM GMT
Wondering if I could run this past you.
This psuedo code is meant to do the following.
-Record sensor data into a ~1 second circular buffer until launch is detected.
-When launch detected, copy entire circular buffer into dataflash memory page by page (via DF buffer) until complete
-Continue recording directly into DF memory page by page (via DF buffer)
-The data is: 12 bits from acceleromter (stored in long), 12 bits from pressure sensor (stored in long), time in ms from launch (stored in long)
Does anyone see any glaring errors in my thinking?
Code removed as updated version below.
Thanks for taking the time to look this over [noparse]:)[/noparse]
EDIT: I added a small amount of code to switch between the 2 dataflash buffers. Write time from DF buffer to DF memory is ~3ms but I can write to the 2nd buffer while this is happening in the background.
EDIT 2: Fixed a few big problems and added some documentation about the dataflash.
▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔
-Alec
My our page
Post Edited (Basil) : 9/1/2007 11:23:11 PM GMT
Comments
The first two are used one at a time, just rewriting the same 2k. On launch, write a marker sequence meaning "launched" to the current
second file and then start the launched file going.
On recovery your files will have one partial second of prelaunch data -- marker sequence is present, an older full second of prelaunch data, and the in-air file.
Do you mean I keep 3 files in the dataflash and write directly to the DF rather than using a buffer?
The reason I chose to use a buffer is because the dataflash only has 100000 write cycles per page.
A rocket could be sitting on the pad for minutes before a launch which would chew through 100000 writes in no time at the sample rate of 200sps.
Please let me know if I misunderstood [noparse]:)[/noparse]
EDIT: Oh, my time overhead is <5ms i.e. the time between samples
▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔
-Alec
My our page
Post Edited (Basil) : 8/30/2007 11:08:47 PM GMT
If so, with two one_second files, it seems you'd hit that barrier in 55 hours on the ground. (A·location overwritten 30x a minute. 100000/30 =3333 minutes)
Once launched, you will be keeping these files and never writing them again, so you should be safe.
Why so high a sample rate on the ground? It seems you would acres of the·same data except for the time mark.
·
Hi Fred,
The high samples rate on the ground is to make launch detection as accurate as possible.
Some of the faster rockets pull +40G's off the pad. Launch detection occurs when the accelerometer detects 2.5G's for 0.25seconds or 5G's for 0.125 seconds.
I am interested in your idea, could you perhaps post a short bit of psuedo code to clarify? My brains is mush right now
▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔
-Alec
My our page
Sorry, forgot to answer this bit properly.
Only 1 second of pre-launch data is actually being saved to the dataflash
The buffer in the cog is continuously being overwritten prior to launch, to keep only 1 second of pre-launch measurements.
It is not until launch is detected that the pre-launch buffer is copied to the dataflash.
Only after this happens, does writing directly to the DF start (there is no buffer needed during flight).
Well i hope thats what my code is doing!
▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔
-Alec
My our page
Looking at your on-pad data stream, only the time value really changes. You might be better off gathering an initial barometer value,
and then averaging the next five to check against the first. Write a new value only when the average of five readings changes from
your saved value. Acceleration won't be an interesting value until you actually start a launch. Acceleration change is how you tell you
have launched -- and drops you out the on-pad loop behavior.
If you define your file as various timestamped entries (a barometer record, an acceleration record, a landing record, a launch record,
an abort launch record, an abort flight record) then writing it as it happens won't be a big deal. And if you don't write to the file until a value
changes (besides time), then you have lost most of the reason to have a pre-launch buffer.
Then the only reason for buffering then is to fulfill a write requirement for x number of bytes. You might not even have this limitation at all.
Writing a file might devolve down to just a series of byte writes.
Fred
The way I understand what your saying, is you are suggesting I use the memory purely to save data on the various flight event's, each event
having a time stamp and appropriate sensor readings.
My intent is to produce a fully LOGGING altimeter (on top of just detecting flight events). That way the user can upload the data and view thrust curves,
altitude vs time, velocity vs time etc and effectivly 'playback' the flight in real time.
The 1 second of data prior to launch ensures no acceleration information is missed due to slow launch detection.
You are quite right about the barometric data. It will never change until the rocket is actually in flight....I will consider leaving pressure data out.
Thanks for the discussion [noparse]:)[/noparse] Please correct me if I mis-understood.
Alec
PS. I have made a decision not to time stamp each entry. The sample period is a constant 5ms so there is really no point recording this as is can be caluculated
using the record number (E.g. record #50 is 50x5ms into the flight = 250ms)
▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔
-Alec
My our page
The preflight data coud in that case be shoved directly into the two DF buffers (1056 bytes) and then transferred to flash with single DF commands.
No having to transfer the Prop buffer to the DF buffer. That would require that a pointer be stored in DF for the zero point.
I guess the question is, why logging at 0.4 millisecond better than at 1 ms? Could 4 samples
be taken in 1 milisecond and averaged for the log file?
▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔
Tracy Allen
www.emesystems.com
I have just made changes to do exactly what you said [noparse]:)[/noparse]
Originally I was storing the 2x ADC readings as words (not longs as my original code said...), but I am now storing them as 12bits each.
Also, I have done away with the timestamp as per my previous post. This cuts one 'set' of data down to 6 bytes rather than 12.
This means I can get 0.88seconds of data into 2 buffers @ 5ms between samples, which is enough to capture all the data I need prior to launch detection.
Also, it takes 3ms to transfer each buffer to memory. I can't do this if I use any more than 2 buffer-fulls of data!
(I should mention, each 'sample' is actually an average of 10 readings from the ADC.)
The reason I am using the prop as a pre-buffer... buffer, is due to the way I interpret the 'Buffer to Main Memory Page Program' command.
It looks as though it copies the DF buffer starting at the start. If I was implement a circular buffer directly into the DF this would mean the start
of the DF buffer might not be the start of my 1second (well 0.88 seconds really...) when I go to trasnfer it to the main memory.
Using the pre-buffer in the prop means I can 'sort' the data when I trasnfer it to the DF buffer prior to writing it to memory.
So many buffers! Anyway, and suggestions are welcome [noparse]:)[/noparse]
EDIT: Forgot to ask, where is the 0.4ms from? I hope I got my calcs rights, it should be 5ms
▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔
-Alec
My our page
Post Edited (Basil) : 8/31/2007 11:16:13 PM GMT
▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔
-Alec
My our page
It looks fine, 3 bytes per sample, 175 samples takes 525 bytes, fits in one DF buffer, and takes 0.875 second at 5 milliseconds per sample. That leaves 3 bytes "free" in buffer 1.
It is true that the data written directly in the DF buffer in a circular manner would not end up with its start point at physical location 0. But I was thinking that
one of those "spare" 3 bytes at the end of the buffer could be maintained as a pointer to the logical start of file. The data readout routine would retrieve that pointer.
It would save the step of having to transfer Prop buffer to DF buffer.
▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔
Tracy Allen
www.emesystems.com
Don't have much time right now but will give it a go in the morning [noparse]:)[/noparse]
Any other suggestions appreciated!
Alec
▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔
-Alec
My our page
Good morning [noparse]:)[/noparse]
Tracy, How does this look? If there is a better way to do the pointer please let me know [noparse]:)[/noparse]
Fred, is this along the lines of what you were thinking?
▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔
-Alec
My our page
Post Edited (Basil) : 9/3/2007 7:15:03 AM GMT
One DF page then holds 175 records and the preflight pointer would then increment mod 175. One byte now is enough to hold the pointer. The pseudo code for sending data for each record would 1) open the DF buffer, 2) send the address location (which is now pointer*3) and the 3 byte record, and 3) close the buffer, and 4) increment the record pointer, {pointer := (pointer + 1) // 175}. When launch is detected, it would 1) open the RAM buffer, 2) send the address 527, followed by the pointer value to that reserved location, and 3) close the buffer. So long as the prelaunch period is greater than 0.875 second, the buffer will be full of samples, and the pointer retrieved from location 527 will reference the first record followed mod 175 by the others.
It is just my personal preference, but I'd keep on tracking the main pointer after launch in terms of records, instead of bytes. There is an issue, of whether you allow records to span across memory pages, or you only allow complete records on pages and leave a few unused bytes at the end of each page (3 bytes, or whatever it is after you add the additional sensors and time stamp after launch). The calculation and other things are easier IMHO if there is always a whole number of records per page. The logging could stop at the end of physical memory, or, it could wrap around until the head catches the tail.
▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔
Tracy Allen
www.emesystems.com
I am glad that Tracy has weighed in, as I my comments regarding dataflash were based upon a complete misunderstanding.
(Realization: You're not using an SD card. Doh! What sort of DF chip are you using?)
Backwoods rule: don't lose altitude unwillingly.
This bit seems to risk that: "if DF_page_location => 4096 then memory full. Abort flight and upload/clear memory
Erase Dataflash pages from DF_page_location to end of memory"
I would not want to automatically throw everything recorded away, but rather to manage erasing separately and distinctly.
How do you know that you are close to filling memory? You don't want to launch with insufficient space available, particularly
if you include an erase when full routine.
Fred
The DF chip is an Amtel Dataflash (16Mbit AT45DB161D)
That bit you quoted is just a note for myself. In reality, prior to each flight, the memory remaining can be checked via LED's or LCD, I haven't decided yet.
If the memory is full (or within, say, 60seconds of being full) the altimeter will produce an error and will not operate until the user does something about it.
The program will not automatically throw away data [noparse]:)[/noparse] The user can upload the data to a PC and clear the dataflash at any time. Hi Tracy,
What do you mean by 'increment mod 175' ?
Does this mean I would be writting to the buffer byte by byte rather than leaving the buffer 'open' and continuously writing until buffer is full?
I dont really understand the pre-flight bit sorry lol. Its probably my understanding, but could you clarify a little?
During flight, I might keep the buffer open and write byte by byte as I am already doing, but I change the code to only use 526 bytes of the buffer,·saving the last byte to save the 'next DF page of data' pointer before closing the buffer and writing it to memory.
That way I have a record of when each flight data ends. (Ie 'next page' = NULL)
▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔
-Alec
My our page
Mod is a nifty way of implementing circular values.
For example:
If you were using "MOD 3" and kept adding 1 to that variable, it would give the following results: 0,1,2,0,1,2,0,1,2,...
This is not the same as base-3 which would give you: 0,1,2,10,11,12,20,21,22,100,...
Your program could continually add 1 to the pointer. When the pointer got to the top, it would wrap around back to the beginning.
The old fashion way of doing this without MOD is the following:
pointer := pointer + 1
if pointer > 175 then pointer = 0
But mod has its advantage: If you had a 175 byte circular buffer and was wondering where the 43,285th byte was written, it would be 43285 mod 175
Yes, you could leave the buffer open and write continuously during the preflight interval, even after the buffer is full. The buffer pointer inside the dataflash chip will roll back to zero (MOD 528) again and again on its own, head eating tail. 528 is exactly divisible by 3, 528/3=176, so the circular buffer holds 176 complete records. At 0.005 second per record, it ends up holding the most recent 0.88 second of data, as you already pointed out. The thing is, to read back the data the data offload program has to know where the head and tail are located in the circular buffer.
Do you have a real time clock on the system with backup RAM? If so, that is a great place to store pointers that won't get lost in a reset. Or that pointer could be stored somewhere else, like in the Prop eeprom along with DF_page_location. Or, pointer could be stored right on the flash page. But to do that, the pointer has to be stored at a known fixed address where the offload program can retrieve it. One way to do that is to leave three extra bytes at a fixed location (525 to 527) on the flash page. But to do that, your program has to manage the buffer mod 175 instead of allowing the dataflash to manage it automatically mod 176. The buffer has to close and reopen from time to time, maybe not for each byte or even for each 3 byte record, but at least once every 525 bytes or 175 records. Again, it's very convenient to have battery backed RAM for pointers.
Also for DF_page_location, it could be circuilar mod 4096, so the head could chase the tail around the entire array from flight to flight. Even if the flight starts out pointing to page 4095, it would wrap around to 0 within one flight. I'm also wondering if it is necessary to preerase the entire array. (<Erase Dataflash pages from DF_page_location to end of memory>)
▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔
Tracy Allen
www.emesystems.com
That clears things up alot! MOD sounds like one of those wonderful code saving things everyone should know!
There is no real time clock, but for pointers and such, I will be storing them in the EEPROM (as with DF_page_location).
It sounds easier than storing them on the DF page...
Ill take another look at my code soon [noparse]:)[/noparse] (My birthday today so won't get much time to play around)
The reason I chose to erase the entire array is because the erase time for a single page is like 10ms or something. Its to long to do on the fly during flight, and I won't know how long the flight is (ie how many pages to erase if I do it page by page) before the flight has happened.
If I didn't have to erase I wouldn't, but it seems I have to before I can write to a page.
▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔
-Alec
My our page
▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔
Tracy Allen
www.emesystems.com
▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔
-Alec
My our page
This one uses mod (hopefully correctly! and saves the start and end page to EEPROM, and also the start byte for the pre-flight data.
When reading flight data back post flight, I will:
Go to the chosen DF_data_START_page and load 528bytes from DF_data_START_byte MOD 528 (or however its written)
Then once that first page is read, I will load until page DF_data_END_page as per normal.
What do you think...
▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔
-Alec
My our page
When the flight ends, the DF_data_END_page will point to an empty page, so there is no need to increment it again at the initialization.
Figure that at 0.005 second per 3-byte sample, each page fills up in 0.88 second. The erase-with-write takes less than 0.01 second. I am still questioning the need to erase the whole array initially. There is plenty of time to accomplish erase-with-write in situ, given that you are ping-ponging between the two buffers. Remember, this memory chip is meant for real time continuous audio recording!
▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔
Tracy Allen
www.emesystems.com
Good point! I was still thinking of the old 2 buffer pre-flight buffer thing where I had 5ms to transfer 2 buffers to memory after launch was detected [noparse]:)[/noparse]
I will change the routine to the one which includes erase.
FYI, plans are to include Kalman filtering in the flight routine. Doing this makes it more practical to record at 128sps, so I now have 1.35 seconds to save to memory [noparse]:D[/noparse]
Thanks for the help guys!
PS, I assume by lots of details, you mean get rid of the psuedo code and replace it with spin :P
▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔
-Alec
My our page
I just see this in Basil's code
It's the right time to recode it:
Good point about the buffer number. It is either 0 or 1 as sent to the chip so the math could be reduced to the simple //2.
▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔
Tracy Allen
www.emesystems.com
But 0 based sequences have always advantages!
▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔
-Alec
My our page