SD Card Usage Survey

DavidZemon

Does anyone write to and read from the same file in one program?

Specifically, does anyone write to and read from the same file repeatedly and in quick succession?

I'm considering going the C++/Java route for PropWare. That is: having separate classes for input and output. I think this will have positive effects for code size.

Thanks,
David

Electrodude

Disclaimer: I almost never use PropGCC.

I've never done this (whether on or off of a Prop), but I can come up with cases where it would be useful, for example, if you need to transmit lots of packets over a lossy radio connection and have to be able to store them all on an SD card (they don't fit in RAM) and then mark them as acknowledged when you get acknowledges (by writing only one byte of a huge file). If the reception was any worse on a live telemetry system I made, I would have had to do this (or get longer range XBees, but that's much too simple a solution).

Could you make a read-only variant, a write-only variant, and a read-write variant, and only use the read-write variant if it's needed? The only problem with this is that, if the read-write variant is ever needed, you would probably want to replace all instances of the read-only and write-only variants with wrappers for the read-write variant. Obviously, it should automatically pick whether to use the read-write variant or to use the read-only and the write-only variants, but I can't see how you would do this with C++ without lots of preprocessor magic.

DavidZemon

The File class already has a "flush()" method. If it also contained a "reload_buffer()" method, then FileWriter could call flush() and FileReader could call reload_buffer() and then you could have an instance of each class open on the same file and keep them synchronized. I could probably even write a single class that contains both a Reader and Writer object which does this synchronization automagically. The code is already in place for two File instances to share the same buffer in RAM, so you wouldn't even be wasting RAM space by having two File instances in one.

SRLM

Well, I haven't really written anything new using an SD card in quite a while. But my most recent program just used the SD card as a data store: I logged values to the card. I never had a need to have both read/write access.

I agree with the breaking apart of the input and output. I worked on something similar here.

DavidZemon

SRLM wrote: »

Well, I haven't really written anything new using an SD card in quite a while. But my most recent program just used the SD card as a data store: I logged values to the card. I never had a need to have both read/write access.

I agree with the breaking apart of the input and output. I worked on something similar here.

I was just looking at your Stream-related classes over the weekend and thinking "wow! this is just like my Printer/PrintCapable classes!". The only difference is, I had to make up my own dumb class names instead of going with standard stuff. I might rename them before an official 2.0 release :P

SRLM

I must admit that I stole the InputStream and OutputStream names. Why I choose `Get` and `Put` for function names is a mystery, though.

msrobots

@swimdude0614,

In another thread you asked for multiple SD card and the need to support this. So a while ago @MacTuxLin build me some 4 SD-card boards for the Quick-Start. They where (are?) for sale at propellerpowered.

The plan was a RAID system for and with the Propeller. Sadly in 2 years I sold none as far as I know. Around $3000 down the drain.

So my guess is that nobody is really interested in having multiple SD-cards on one Propeller.

Multiple files open at the same time is a must for most of my projects, Kyes Fat_Engine and FSWR both support this by including the objects multiple times. But the RAM cost of multiple File Control Blocks and Buffers is hard to the small HUB. Especially since support for FAT-16 and FAT-32 is implemented in both of them. So the objects are not small to begin with.

Both File System Objects support reading and writing of open files and seek() to go to a position in the file. But I did run into a lot of trouble while writing. Somehow writing somewhere inside the file would also change the file size to the end of the last written byte. I had to modify Kyes Fat_Engine to get around this problem. FSRW can just seek in one cluster, pretty useless on bigger files.

I did not followed your PropWare's file access closely, but for a SD block driver, nothing beats @LoneSock's PASM driver included in FSRW. I do not know if you are using it already - if not, take a serious look. It uses a internal 512byte buffer in the COG to support Read Ahead and Write Behind. Very cool PASM. So if writing a sector, the call returns after your HUB buffer is copied to the COG ram. Then written to SD. After reading a Sector from SD it gets copied to HUB and then the next sector gets pre-fetched into COG ram in case you need it next.

@LoneSocks PASM code is really cool. Somehow I also see @Kuroneko's handwriting in there, but have no proof of that.

I had some basic ISAM DataBase written for the Prop. Working nice, but hitting all them problems. Multiple files, random access, files larger then a cluster.

Then real work kicked in and I am not allowed to play with the propeller since a while. At the time I got self employed, I was not aware of what an A..hole my Boss would be. Constantly driving myself to overtime. No vacations, things like that.

TLDR

Yes, reading and writing the same open file is quite important. Fast seeking of position in a file also. Multiple files open at a time is also important, multiple SD cards not so.

Enjoy!

Mike

DavidZemon

Lots of good stuff to read! Thanks!

I'm sorry to hear that you didn't sell a single board for the RAID system .

The FSRW driver sounds slick. We'll have to see how PropWare's performance compares after I implement the array read/write functions in my SPI driver... hopefully it's able to come somewhat close to FSRW.

If it doesn't compare, I'm afraid it's probably not something I'll be fixing. What makes PropWare's SD classes special is that there are multiple levels of abstraction - something completely different from any other implementation on the Propeller. Because of that abstraction, it would be very difficult to make something like FSRW's driver work [well] with my classes. And what would be the point? We already have libpropeller's SD class if you decide you need FSRW in C++.

As for code size: no worries! That I have very well covered. An instance of FatFileReader takes up only 84 bytes! So, open lots of files, no problem! And I wrote the following snippet as a cool example of multii-file read:

BlockStorage::Buffer secondaryBuffer;
    const SD driver(SPI::get_instance(), Pin::P0, Pin::P1, Pin::P2, Pin::P4);
    FatFS filesystem(&driver);
    filesystem.mount();

    FatFileReader reader(filesystem, "fat_test.txt");
    FatFileReader reader2(filesystem, "fat_tst2.txt", &secondaryBuffer);
    reader.open();
    reader2.open();

    while (!reader.eof()) {
        pwOut << reader.get_char();
        pwOut << reader2.get_char();
    }

`reader` uses the shared buffer which is statically allocated in the FatFS instance. This is great for tight memory requirements. On the other hand, `reader2` gets its own, dedicated buffer, as you can see by the third parameter to the constructor. This completely alleviates thrashing the SD card . Of course, if memory is more of a problem than speed, simply remove the third parameter from `reader2` and both file instances will share access to a single buffer - say hello to 512 more bytes of hub RAM!

tl;dr
I think you're gonna love SD card access with PropWare. I'm sure proud of what's done so far, and it's only going to get better. There's lots more to come!

-edit-
Oh yea - that's pretty silly about seeking causing problems. There won't be any issues with that here. The seek operation itself is nearly instant - just a single write to a variable. Reading in the new buffer and flushing out the old one doesn't happen until absolutely necessary. If you use seek() to a different part of the file, the read or write function will notice the buffer is incorrect and reload it. This was a necessary aspect of shared buffers, but also made implementing seek() really easy

msrobots

So when not providing a second buffer I can save space, but the speed of access will go down because of buffer trashing. Nice to know. I have a hard time switching to C/C++. Just love the simple way Spin and Pasm do things. Not back to the future but back to the past.

On the other hand I see the value of C/C++ even on the Prop1. It just looks more like being at work then fun. I have barely time to follow the forum and no time for Propeller Fun. Sad.

But I follow PropWare with great interest.

Enjoy!

Mike