Back to Work On File System. Last chance for suggestions.

Kye · 2010-02-25 06:06

Hey guys,

Spring break is coming up after the next week of classes at my school. During·the break·I plan to finish my full file system driver and post it online.

So, if you haven't read and this thread here: http://forums.parallax.com/showthread.php?p=871049, do so. Or at least please read the first post by me to see what I'm talking about.

---

I'll be putting 40 hours in on the driver and I'll try to have it completely finished by the end of spring break. Fun stuff may come up however that will waste my time so I'm posting here now to get any and all last minute·suggestions before starting to go to work.

---

So, lets have at it then. I won't be updating the code after its posted for a while unless·I get a large chunk of free time.

---

Notes:

I have decided to drop support for file system paths. While they would be cool to support they require dynamic memory to do *properly* and I'm not interested in finding a hacked solution to support them. Additionally I would need to rewrite 50% of the code to do so.

I have decided to delay adding in multiblock support for SD/SDHC and MMC cards. I have yet to get my code fully working to support those modes and I have not worked out all the error case problems which exist by using those modes. I plan to support these modes later as I want more speed too.

---

Attached is the current working code. ITS! really ALPHA! right now as I did alot of stuff under the hood. While the functions are still the same I changed the way alot of things worked and I have not sufficently tested them so use the code at your own risk.

I'm posting the code however so that you can look over it and suggest things to add or change.

▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔
Nyamekye,

pgbpsu · 2010-02-25 16:49

Kye-

Thanks for the response. I've seen all your hard work on the file system driver stuff, so I'll take your word that there's no easy way to do it. Your example does (in a cleaner fashion) what I currently doing. I think I have a work around for my problem. I can simply scan the disk at startup and find out how many files and how much space is used. If I keep an accurate count of how many bytes I've written to the disk, I'll have a pretty good idea of how much space I still have free.

Thanks,
Peter

Kye · 2010-02-27 02:19

Pretty crazy how no one is interested.

Amazing.

I'll continue to keep this thread alive for a week.

▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔
Nyamekye,

Timothy D. Swieter · 2010-02-27 02:21

I'm interested in what you create Kye, it is just that I don't know what to say regarding features or functions. I imagine a driver would need the ability to write strings, decimals, hex, binary to a file easily. Maybe that is in your code or maybe it is functions on top of your code.

▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔
Timothy D. Swieter, E.I.
www.brilldea.com - Prop Blade, LED Painter, RGB LEDs, 3.0" LCD Composite video display, eProto for SunSPOT
www.tdswieter.com

WBA Consulting · 2010-02-27 03:36

Kye, not sure if this is a suggestion as your file system may currently do what I need if I learn to use it properly, but I have a project on my back burner that will need a database type file stored as a text file on an SD card. For example, my text file may contain:

415-0223, 3, D, 0945, 04122010
DR-435-00, 4, C, 1632, 04142010
14-039445-10, 3, C, 1310, 04132010
----

And I need a quick method of pulling the data after the first comma when I look for the data before the comma. In other words, "DR-435-00" is entered in via a keyboard and the rest of the data string would be retrieved and utilized by the code appropriately. This list if data in the "database file" could be several data fields wide and have about 200 lines.

thanks!

▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔
Andrew Williams
WBA Consulting
WBA-TH1M Sensirion SHT11 Module
Special Olympics Polar Bear Plunge, Mar 20, 2010

Kye · 2010-02-27 04:36

@WBA Consulting

If you know where the tag line you're looking for starts then you could just seek there. Other wise you'll have to search through the file character by character looking for a string match.

A slightly fast approach would be to transfer a block of text into memory and do string comparison on it in there. This would avoid the overhead of calling the getCharacter function over and over again.

... Given the right setup and implementation my file system has no problem doing what you need. But neither does the current FSRW.

@Everyone - I'm just looking for more input on usefull random file system features that people may want. Recent I incorporated a boot function allowing you to boot from a file on the SD card for example.

Thanks,

▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔
Nyamekye,

Dave Hein · 2010-02-27 14:10

Kye said...
...
I have decided to drop support for file system paths. While they would be cool to support they require dynamic memory to do *properly* and I'm not interested in finding a hacked solution to support them. Additionally I would need to rewrite 50% of the code to do so.
...

Kye,
Can you elaborate on what you mean by file system paths?· Are you talking about directory paths?· Also, are you referring to malloc and free for dynamic memory, or do you mean something else?· If you need a malloc routine there are a couple of them in the OBEX.· I have also recently written a malloc/free object for myself if your interested.
Dave

Kye · 2010-02-27 16:27

By directory paths I mean:

C:/windows/system32/... etc like commands. The idea is that you would be able to type things like that in and maninpulate files.

The problem with supporting paths is that I need to have some way of copying the entire user input string and saving it for string processing.

Addtitionally I have to also save the current working directory path.

Both of these things require an extra 512 bytes of buffer memory space and would add more size to the whole object which is quite large already. Additionally because paths could be on infinite length I would quickly run into problems on disks with more than 256 characters of path length. Eg. A disk with 23 folders each insider each other would be the limit.

Basically the whole "path thing" screams that I should use dynamic memory.

By having the system work as it does now I do not suffer from any of these problems. But my functionality is limited.

▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔
Nyamekye,

Dave Hein · 2010-02-27 17:50

OK.· If you would like to use my malloc later on, just let me know.· It only uses 94 longs of program space.· At some point I'll post it to the OBEX as part of a C function library object that is·written in Spin.

Dave

VIRAND · 2010-02-28 04:43

Someone recently asked which filesystem object is the fastest. Do you know? I have no speed greed though.

I am very interested in what you are doing but you are the expert on it relative to myself, so what more can I say?

I don't need the maximum speed and I assume that your work towards a "tape recorder emulator"
if I understood it correctly, is probably sufficient speed for me, I think. If you did not make a tape emulator
then I guess I will not cross it off my own list of project ideas. I have yet to try the object you mentioned that I
am referring to as such, due to many distractions.

▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔
I should be typing in Spin now.
Coming soon. My open Propeller Project Pages and favorite links index.

Kye · 2010-02-28 05:12

Mine is way slower due to a bunch of protection based code I have implemented. All my functions execute as fast as possible but my block driver does not have unrolled loops/operate in multiblock mode/etc, so I only get 88KBs read and about 64KBs write.

My feature list however is massive. I pretty much have everything needed ever and more.

▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔
Nyamekye,

Duane Degn · 2010-02-28 21:17

Kye,

I’m very interested in what you’re doing. I presently use version 1.6 of fsrw to record information (weights, barcode information and data from a spectrometer) I use in my work. Since I started using a Propeller to log the data from the instruments’ serial ports, I’ve been able to work much faster and more accurately (I used to write it all down with pen and paper). I’ve been too busy to try your driver but after looking through the files you just posted, I’m sure I’ll want to use many of your new features. I just added a RTC to my board yesterday to start time stamping data and I’m very pleased to see the time and date features in your object.

One problem I’ve had in using SD card for data collection is corruption of the file system. If the Propeller is turned off while a file is open, the open file is corrupted. The corrupt file can cause subsequent files not record properly (or not at all). As I’ve learned on this forum, reformatting the card restores it to usability again. Is there a way to check the file system, using the Propeller, for corruption? I see in the readme file you have a "test" method. Could this be used to check for corruption of the file system?

One method I planned on using to check the file system integrity is to create a new file at the beginning of a data collecting session. I’d write some data to this new file (I’d also keep the information in RAM), close the file, and check the data was present and accurate. (If I understand your "test" method (I’m not sure I do), it does about the same thing.) Writing and then reading a file would give me some confidence subsequent files will be retrievable. Is there a better way to check the integrity of the SD card?

I don’t know if I understand what is meant by a partition on a SD card. Is it the same as a partition on a hard drive? Would using the "formatPartition" method fix a corrupt file system as well as formatting the card with a PC? I hope so.

I’ve read elsewhere on this forum one shouldn’t delete files from a SD card because of possible corruption problems (which is a major concern for me). Is there a safe way to delete files, other than reformatting the card?

I’m very excited to add increased functionality to my data collection. Thank you for such a useful tool.

Duane

Timothy D. Swieter · 2010-03-01 00:26

Duane - your application of the Propeller as a data logger sounds interesting. When you record your samples to the SD card I found that using the "sdfat.pflush" command in FSRW helps to alleviate any corrupt files from occuring. I also added a read/write activity LED so I knew to not pull the card during the middle of a read/write.

▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔
Timothy D. Swieter, E.I.
www.brilldea.com - Prop Blade, LED Painter, RGB LEDs, 3.0" LCD Composite video display, eProto for SunSPOT
www.tdswieter.com

Kye · 2010-03-01 04:47

For my file system all you need to do is call the "flushCharacters" method which writes any buffered data to disk periodically.

And the format partition function does exactly what you think it does.

Corruption isn't too much of a problem for my driver. However, loosing power during a write will still corrupt the SD card. There is no way to protect against this since there are not multiple copies of everything on FAT partitions.

My file system driver allows you to also set the Check Disk Flag upon mount. So if the SD card is removed improperly you can then plug it into windows and the check disk utility will be called by windows to repair the disk.

▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔
Nyamekye,

Duane Degn · 2010-03-01 18:20

Timothy - thanks for the information about "sdfat.pflush" and the LED.

I test samples of Leonardite for humic acid. I noticed a lot of the equipment I used had a serial port. I finally learned enough about microcontrollers (i.e. Propeller chips) to record the data (from the serial ports) onto a SD card. It’s been great. Right now the program I use opens a file when the Propeller is first turned on and it stays open, writing data as it is sent to it, until I press a button to close the file and end the program. I’m starting to think there is a better way of doing this. Maybe I should be saving the data in a buffer and then periodically open a file and write the data.

Kye - thanks for the "flushCharacters" idea (I’ll hold my questions about it until I try it out myself) and for the "set the Check Disk Flag." As I mentioned above, I think the way I’m writing to the SD card now leaves me vulnerable to corruption problems. I’m almost caught up with testing Leonardite samples and should have time to try your file system today.

Duane

PS Timothy, I use one of your 3" composite LCDs with one of my recorders. It’s great. I hope you find more.

rokicki · 2010-03-01 21:53

To my knowledge, if the file system is written properly, losing power during a write should *not* corrupt an SD card. You may lose recently written data, but I don't think you should ever see a corrupt card.

The worst that should happen is some FAT blocks could be marked "in use" while they are actually free.

Kye · 2010-03-01 21:58

@rokicki - Actually what I'm talking about is when the SD card is programing a new sector. If the power fails during that time the data is gone.

Nothing you can do since the SD card erases sectors before programing them.

What can be really bad is for this to happen when a new FAT block is being updated. Or when you are writing to the DIR structures.

▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔
Nyamekye,

rokicki · 2010-03-01 22:05

Check out the SD spec. I think you are wrong. I think SD cards are more complicated than that. Give it a shot if you want; it's
easy enough.

rokicki · 2010-03-01 22:21

Let me make sure I'm clear here. You still need to close all your files if you want to maintain the data written to them. All I am saying is that
the filesystem itself should not be corrupted because of power loss. It is *possible* but it should be exceedingly unlikely. But it's always best
to err on the side of caution.

In any case, you *must* close your files (or open another file for read, which closes the "current" file if it's opened for writing) if you want the
data to still be there. (Or call pflush() to flush the data to the card.)

Kye · 2010-03-01 22:27

HAha! I would hope I am wrong. But in the simplified spec atleast I did not find any information on how a write exactly occurs.

But this seems like the only way in which you could corrupt on my file system atleast. My block driver has alot of extra code in it to check for these types of errors.

...

But since both our block drivers don't use CRC we could have bits flipped during transmission. But, getting CRC to work will seriously eat into preformance.

Anyway,

▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔
Nyamekye,

Duane Degn · 2010-03-01 22:47

Kye - I’ve been playing with your demo. How fun! This is great stuff. A couple little things I’ve noticed. Here’s a copy of the terminal window.

>_ time

Running command: time

Enter the time.

Year (2000 - 2127): 2010

Month (1 - 12): 3

Day (1 - 7): 2

Date (1 - 31): 1

Hours (0 - 23): 12

Minutes (0 - 59): 2

Seconds (0 - 59): 50

>_ date

Running command: date

Monday, March 01, 2010 12:03:59 AM

I had intended to set the time for 12:02:50 PM. I think the problem with the wrong hour showing up is the way the DS1307 can store the time in either 12 or 24 hour format. Bit 6 of the hours register indicates the format. I purchased the Real Time Clock module from SparkFun.com. It comes preset to mountain time (which is my time zone as well) and with the 12 hour mode set. I can change the mode with another program and it should then work correctly with your driver, but I thought you’d like to know about the possible problem and have your driver check for the mode or just make sure it sets it to the 24 hour mode as it sets the time.

After looking at the demo code, I think I’ve figured out why the minute field was different from what I had intended. I think the between the time it took from setting the minutes to the seconds the original second field had rolled over increasing the minute field by one. I know I hadn’t waited over a minute from the time I typed "50" to the time I typed date. I had been trying to synchronize the time with my watch and had typed in 50 and waited a couple of seconds to press the enter key for my watch to show 50 just as I pressed the enter key. I doubt it matters much for the demo but I might want to hold the minutes and maybe even the hours field in a buffer and set all fields at once to make synchronization easier in programs using these objects.

For your information I was using a Kingston 1 GB microSD in the following test.

Wrote 32,768 bytes at 0000008291 bytes per second.

Read 32,768 bytes at 0000009731 bytes per second.

Now testing block write write and read speed...

Wrote 32,768 bytes at 0000032592 bytes per second.

Read 32,768 bytes at 0000094525 bytes per second.

A 2 GB Crucial microSD card was used in the following test.

Wrote 32,768 bytes at 0000008544 bytes per second.

Read 32,768 bytes at 0000009730 bytes per second.

Now testing block write write and read speed...

Wrote 32,768 bytes at 0000029001 bytes per second.

Read 32,768 bytes at 0000094310 bytes per second.

A Panasonic 8 MB gave "Disk I/O Error" on mount. It won’t work with fsrw either. I had just formatted it. I don’t care about not being able to use the 8 MB card; I just thought you’d like to know.

A Toshiba 2 GB SD test results are:

Wrote 32,768 bytes at 0000008792 bytes per second.

Read 32,768 bytes at 0000009743 bytes per second.

Now testing block write write and read speed...

Wrote 32,768 bytes at 0000037335 bytes per second.

Read 32,768 bytes at 0000096828 bytes per second.

Kye - It has been a couple of hours since I started writing this post (I stopped and worked on other things for a while) and the date feature is now giving the correct time. Maybe it just has a problem with the noon hour.

Duane

Kye · 2010-03-02 02:02

The code I have in the demo program reads and sets the time pieces one by one at an I2C rate of 33Khz. So its very possible to have slight errors during edge cases.

But, time shouldn't be a very sensitive things. If it is then you can always change the "readClock" rountine in my driver to get the time from somewhere else.

Thanks,

▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔
Nyamekye,

Kye · 2010-03-05 17:15

Spring break's here, time to get some work done.

▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔
Nyamekye,

mpark · 2010-03-05 19:30

Kye said...
Spring break's here, time to get some work done.

Spoken like a true nerd!

/or geek if you prefer
//meant as a compliment either way

Kye · 2010-03-05 20:01

Ah, I'm stuck on my schools campus and everyone else I know is going home.

Plus I don't have a TV or a kicking computer anymore. Not like I can goof off.

▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔
Nyamekye,

Duane Degn · 2010-03-21 17:50

Kye,
There is a small error in the way the time is displayed in the demo program.
In the "programDate" method you have:

    result := rtc.getHours
    stringPointer := string(" AM", 13)
    if(result > 12)
      stringPointer := string(" PM", 13)
      result -= 12

This causes the time between noon and 1:00 PM to be displayed "AM" when it is really considered "PM."· Here's my fix, you probably have a better way of fixing it.

    result := rtc.getHours
    stringPointer := string(" AM", 13)
    if(result > 11)
      stringPointer := string(" PM", 13)
      if(result > 12)
        result -= 12

Thanks for all your great programs.
Duane

Kye · 2010-03-22 13:50

Thanks, for the heads up. I have already fixed that however.

I'll be posting the file system sooner or later. I've had to rewrite some of my helper objects to make everything pin flexible and it prevented me from hitting my target release date. And then school started up again for me sooooooo, it will be a while.

Attached is my current progress. ( You'll need to have the string/com/and rtc engine·to compile).

▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔
Nyamekye,

Duane Degn · 2010-03-22 15:00

Kye,

Have you posted the newer versions of your string/com and rtc engine? The demo will not compile with the original versions.

Thanks for the updates.

Duane

Kye · 2010-03-22 16:50

Yeah, That's why I'm not done yet. I've been changing the way the RTC Engine Works.

I belive the old RTC Engine should still work with the demo.

▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔
Nyamekye,

Back to Work On File System. Last chance for suggestions.

Comments