SD Card help/advice please?
Rforbes
Posts: 281
Hey all,
I started goofing around with the sd card on my prop boe and now I'm trying to figure out how to do something I thought would be easy, but.... it's stumping me.
I'm trying to figure out how to perform the following:
Write several strings to a file.
Close the file and forget about it for awhile.
Open the file when needed and read the last string in the file to a buffer.
Remove the last string from the file, adjust the file size, and close the file.
Repeat later on.
How can I adjust the file size/get rid of the last string ?
I have managed to get the result I'm looking for by writing the rest of the file to a different file, then renaming it to the original ... it works, but with large files it's slow and it seems there's got to be an easier way to do this.
I've attached an archive to show what I thought might work without too much fuss, to no avail.
Suggestions or help please?
Thanks in advance!
I started goofing around with the sd card on my prop boe and now I'm trying to figure out how to do something I thought would be easy, but.... it's stumping me.
I'm trying to figure out how to perform the following:
Write several strings to a file.
Close the file and forget about it for awhile.
Open the file when needed and read the last string in the file to a buffer.
Remove the last string from the file, adjust the file size, and close the file.
Repeat later on.
How can I adjust the file size/get rid of the last string ?
I have managed to get the result I'm looking for by writing the rest of the file to a different file, then renaming it to the original ... it works, but with large files it's slow and it seems there's got to be an easier way to do this.
I've attached an archive to show what I thought might work without too much fuss, to no avail.
Suggestions or help please?
Thanks in advance!
Comments
Maybe not very helpful for you but if you did it in Tachyon Forth then it would be a cinch and it even includes FTP which I think you were going to post at one time (dig dig).
I think it's a little premature to try and tackle Tachyon Forth for me. I've been reading posts and such about it, but for now I've got all my old mind can handle. But seriously, I *will* give it a shot soon. And you'd better be prepared for an onslaught of questions.
And dang, you're right. I have a working ftp client- I think I was supposed to post it for someone. I'll dust it off and clean up my comments and slap it on here somewhere for anyone who's interested to peak at. (You wouldn't believe how many curse words are in my comments. It's the Navy in me coming out.)
I've used up all my curse words on the WIZnet chip, yeah, you can get it to work but the manufacturer's saving face policy does not endear you to them. US companies tend to be more upfront and helpful, if only Parallax made a similar chip! Pity Microchip didn't do more with theirs and had hardwired TCP/IP as well.
When you are ready to draw up anchor and start sailing out into Tachyon waters you'll find the fast and interactive nature of the environment a big help so you will only need to annoy me just because you can
For housekeeping of the size of the file there are several possibilities which depend on what else you want to do with the file. Does it have to make sense for a PC (because you move the SD card from propeller to PC to read that file directly)? Then I'd suggest to fill all the not used bytes with spaces or linebreaks and do the housekeeping with a variable that you store elsewhere (other file or EEPROM). There are nice tips on how to store a variable directly on EEPROM.
If the file is for propeller use only you can pre-fill it with $a5 (for example) and you can search for the last entry. With a binary search you can easily find the end of the file with a few sector-reads.
When adding a string you just add it, when deleting you just overwrite with $a5 again.
With this sector approach you can also have another file opened the regular way.
I guess my virtual memory code would help here if you have at least 512 byte of free memory to keep a whole sector in HUB-RAM.
Please find vMem here:
http://forums.parallax.com/showthread.php/117573-Virtual-Memory/page2
Post #10 gives you the basis ... full FSRW and a test program and post #24 gives you the latest version of fsrw.spin including some bugfixes.
That sounds really great! And it just confused the Smile outta me.
There will actually be between 48 and 480 files on the sdcard. The maximum size will be limited to around 1 megabyte per file. The contents of each file is an series of ascii strings and would normally not be used except by the propeller itself. But on occasion I pull the card and view the contents with a text editor to double check data.
At certain times, new strings will be added to the card (1 at a time) and then at other times the strings will be read to a buffer (one at a time0. So the file will be growing and shrinking throughout the day. It will even be "0" in size at times.
In the past, I would open the file and read the FIRST string out, then read 8192 byte chunks of the file into a temporary file via a buffer until I reached the end. Then delete the original file and rename the temporary file to what the original file was called. This works well, but when the file gets large it's very time consuming. My goal is to minimize that time usage. So was hoping to figure out how to simply strip off the LAST string in the file, modify the file size and close it. Then the next time I open it I would go to the end of file, strip off the LAST string again, modify the size, close file. Rinse and repeat as needed.
Basically I am using the file as a FIFO buffer of strings. I would like to use it as a FILO buffer of strings in order to save time in modifying/re-writing the file.
This sounds almost doable for me but seems like it won't save any time. I'd have to open the file, and start searching back through it until I find the last string. So when the file has very little actual data in it, I'll spend a lot of time searching from the end through all the $a5.
In my testing code, I wrote null instead of $a5. I 've been using the SD-MMC_FATEngine driver and haven't tried to use the fsrw.spin object yet.
All you have to know about sectors and clusters is, that in FAT (FAT is a software layer) a cluster is the smallest amount of bytes which can be allocated. The number of sectors which are in a cluster is variable and defined when formatting the filesystem - then it stays fixed of course. A sector on an SD card and on other devices is 512 bytes, but this is currently going to be extended by latest harddisc-drives for example (it's 4096 bytes there). Access to the data of an SD-card itself is organized in sectors. This is a hardware layer. So, on a 2GB SD-card you have sectors 0-4194303.
If a SD is formatted freshly, all clusters and sectors (except the ones storing the FAT information) are considered as being empty. If you copy files to such a card the clusters/sectors are simply in a row. Things start to get tricky when you delete files. New files will take the first free clusters, but if there is no more free cluster in a row, the next free cluster has to be searched. Having non continuous files is called fragmentation, as the file is no longer occupying sectors one after another. The more deletions and rewrites of files with different sizes you have, the more fragmented the file system will be and the more work there is to read/write a file.
That's why I prefer to use fixed size files which are pre-installed so to say. These files ARE using sectors which are in line. So, to access the file you only need to know the first sector (and maybe it's size if you have to avoid overwriting the next file). And you never need FAT access. No directory entries change, no cluster table change. Pre-installing the files gives you the benefit that those files are still readable by PC.
So, what would I do in your case?
1. Have one file for storing the used size of your 480 files - initialized to 0 - and the address of the first sector (let's call the file dir.txt)
2. Have 480 empty files with a fixed size.
3. The program uses slot 1 of the virtual memory for the dir.txt
4. So, when you want to open a file you only have to lookup the sector address and then you can attach it to a second slot of vMem.
* By knowing the used size you can directly append another string and update the used size in slot 1. -> no sequential search needed
* By overwriting the string and updating the used size you can remove an entry.
vMem simply maps the files to virtual addresses in the range of 0 to (filesize-1).
It would make things much easier if you have a max. length for the strings. Because then you have direct access to each string by calculating it's address instead of searching string end characters.
Say you know that the longest string that ever will be written is 93 bytes, then you could say that each string will be stored in a block of 128 (this divides by 512 and has some advantages over using 93 directly).
Then you know if you want to access string #23, you have to read/write from/to virtual address 23*128 = 2944. This is of course a waste of bytes, but it speeds things up and keeps the code easier.
Maybe I can prepare a little demo for you this evening. (currently it is 3pm local time, so maybe in 7 hours)
As I only have this one setup and micro SD, I don't want to take the risk destroying the operation system files on this SD card. Hope it works for you.
The demo requires connection to the PropellerTerminal. And the files file_000.txt to file_003.txt being copied to the SD card.
Please be aware of:
1. the doInit is there, but it does not flush the changes to SD - that's why on the second run it would init again. Just add an sdfat.flush at the end of init and the changes will be written
2. the doInit would only work properly for NUM_OF_FILES in the range of 2 to 9, as the filename generation needs to be changed to support 000-999.
3. the doValidate is only there as a comment-skeleton. You have to fill it with flesh by yourself if you want this to happen.
4. as I found out you can only have around 500 files on the SD cards root folder
I don't know the state of FSRW I added to the lib (did not use this version for years), so if it does not work for you either maybe that's broken. In this case you can take the version from the thread mentioned in the earlier post which worked for CasLan.
I stumbled over this while writing my ISAM driver.
I need to look up my source, but what I did was to make a public function to overwrite the file size before closing the file.
Then everything went smooth.
will post changes shortly, need to find files...
Enjoy!
Mike
Just add this to Kye's Fat_Engine
PUB setCurrentSizeToPosition {{ '' //////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////// '' // sets current filesize to file position. This can be used to truncate files before closing them '' // Returns the filesize set '' //////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////// }} result := currentSize := currentPositionYou need to open your file and move the filepointer to the position of the last string you want to remove,
and then call setCurrentSizeToPosition before closing the file.
Enjoy!
Mike
@msrobots- Now that's what I need, perfect. Thank you so very much.
Unfortunately I was away from the internet much of the day. I just spent 8 hours tinkering with stuff and finally got it to do what I want- but it's incredibly clunky and while it's effective, it takes up a couple methods and more code than I'd like to include.
Tomorrow I'm going to graft my hotspot to my face so that I never have to go without internet again.