Data Manipulation - SD Card

g3cwi

Hi all

I have some data stored in a CSV file on an SD card. There are around 50,000 lines of data each with about 50 ASCII characters e.g.

G/SP-004,559,-2.0105,53.2603,-20105,532603,1779895,1432603,0,2,2
G/SP-013,385,-2.0463,53.1506,-20463,531506,1779537,1431506,358,1099,1457
G/SP-015,343,-2.1451,53.1703,-21451,531703,1778549,1431703,1346,902,2248
G/SP-001,636,-1.8722,53.3843,-18722,533843,1781278,1433843,1383,1238,2621
G/SP-002,582,-1.8838,53.5389,-18838,535389,1781162,1435389,1267,2784,4051
G/SP-011,454,-2.1438,53.6926,-21438,536926,1778562,1436926,1333,4321,5654

I am thinking that I perhaps need to replace the commas with nulls (The prop book suggests this is a good move). Do I need to read the entire file into a huge array, do the changes and then resave or can I do it more directly by changing the file on the SD card? Using FSRW currently.

I am also concerned about running out of memory and execution speed...

Thanks

Richard

Mike Green

Don't do either.

First of all, why do you want to replace commas with nulls?

I would never update this in place. If I wanted to modify this file, I would read this file line by line, make the changes, and write out the line to a different file so the original doesn't get changed. You could do this with two instances of FSRW, one for reading only and one for writing.

Execution speed is going to be mostly limited by the transfer rate to and from the SD card.

The suggestion for replacing commas by nulls might be in order to make each comma delimited value into a null-terminated string. You could do that, but I'm not sure you'd get any advantages from doing that. The SD card file gets read a byte at a time copying the data from the FSRW internal buffer to wherever you specify. It's easy enough to do the comma to null translation then.

g3cwi

Thanks Mike. The Prop book suggest that null delimited data is more amenable to handling but happy to have another view on this!

The application will work with four comma delimited fields for each record (my sample above includes some redundant data). The recors are effectively POIs (Points of Interest) The fields are a reference e.g. G/SP-004, an altitude in metres e.g. 559, a latutude that has been changed into an integer and a longitude that has been changed into an integer. I intend doing a simple calculation to find the nearest lat/long to my location (from the GPS). The calculation will simply be to compare the current lat/long to the file and by subtraction and addition work out a single figure for "difference". The record with the smallest difference will be the closest one to me. Thus as I move about the spacial file will have to be recalculated. This could be quite time consuming I guess - especially with the realatively slow data transfer speed of an SD card.

AN-003 looks helpful.



G/SP-004
559
1779895
1432603


G/SP-013
385
1779537
1431506


G/SP-015
343
1778549
1431703


G/SP-001
636
1781278
1433843


G/SP-002
582
1781162
1435389


G/SP-011
454
1778562
1436926


G/SP-009
477
1777488
1436699


G/SP-008
517
1778907
1438166


G/NP-032
508
1779878
1440252


G/NP-029
357
1779360
1439928


G/NP-025
506
1780107
1440333


G/NP-028
402
1781720
1439028


G/SP-010
456
1774829
1436295
















































Cheers

Richard

denominator

If your 50,000 points of data is pre-existing, you could always try to arrange (i.e. index) the data in a way to make it quickly searchable. Something along the lines of http://janmatuschek.de/LatitudeLongitudeBoundingCoordinates, perhaps?