Concurrent SD card access

ke4pjw

Has anyone been able to access the SD card via two separate cogs at the same time? For example both cogs read the same file, but at different positions in the file?

Thanks,
Terry

evanh

Bound to be a fail. At the very least the SD card will get confused. What should work, though, is operating two separate cards at once.

evanh

PS: That's the job of an OS. To deal with multiple tasks wanting access to multiples files from the one shared filesystem.

ke4pjw

I think I have a workaround. It just requires a little book keeping in my specific use case. If the SD card objects supported file handles, then it could do the book keeping. A full blowen OS doesn't need to be written to do this. I have locks that manage multiple cogs accessing the Wiznet chip. No OS needed.

Rayman

FAT32 has a seek function:
fileSeek(position)

That might let you jump around inside the same file...

rogloh

If the SD card application is read only and you take/give a lock between sector accesses from different COGs you should be able to share reads from two or more COGs. However as soon as you introduce writes you'd then need to be mindful about filesystem structures and exactly where your lock/unlock points are so you don't clobber the files with two writes to the same sector or otherwise read stale data etc.

ke4pjw

@Rayman said:
FAT32 has a seek function:
fileSeek(position)

That might let you jump around inside the same file...

This is exactly what I am going to have to do, though I didn't think of just keeping the file open. In this specific case, that works.

I wonder if a wrapper can be written to do file handles. The wrapper could keep track of files that are "opened", the mode they were opened in, their file position, and prevent a single file from being opened for writing by two "processes". The wrapper would have to open and close the files upon each access, have the bookkeeping in hubram, and use locks to ensure only one COG has access. Access times might not be great. This would lend itself to having a webserver that has more than 1 available port and multiple threads in separate cogs.

At the moment, this is all a thought exercise. I think it might be useful.

ke4pjw

OK, I think a wrapper is doable. And the great thing about doing it as a wrapper, you can easily swap out the engine if another one comes along.

evanh

Of course you can certainly code such functionality that doesn't yet exist. Yes, the locks are a minimum.

pik33

2 cogs + 1 SD card = trouble

A driver for the card is needed, something like PSRAM driver, so one cog controls the SD and responds to requests from other cogs.

AJL

Depending on the processing load for those cogs, you could run one cog with a sequence of:
process own cog SD request
process own cog web server code
process other cog SD request

It would probably be a bit slower than using 3 cogs, but if you are short on cogs…

Edit: I suppose you could also have one cog running the driver and the other serving both web sessions if there’s enough cog ram in the web server cog to hold the pointers to the buffers for both sessions.

ke4pjw

I started on a filesystem wrapper that provides filehandles and things are progressing. I think this might be useful for others.
Quick and dirty test of open.

RossH

@ke4pjw said:
Has anyone been able to access the SD card via two separate cogs at the same time? For example both cogs read the same file, but at different positions in the file?

Thanks,
Terry

Probably not what you are after, but Catalina can do this - i.e. you can start C functions in different cogs and each cog can then open and read from a file - either the same file or a different file.

Here is an example I just compiled and ran (it's a modified version of the multiple_cogs.c demo, and uses the utility functions from the demos\multicog folder).

This version only has one cog accessing the file at any one time, but if multiple cogs may do so, just use a lock to prevent contention (you could use the same lock as is used to prevent contention on the serial port).

/***************************************************************************\
 *                                                                           *
 *                         Multiple Cogs SD Access Demo                      *
 *                                                                           *
 * This program shows how to use the cog function coginit_C to start         *
 * many cogs running C functions that can each read files from the SD        *
 *                                                                           *
 * Compile with a command like:                                              *
 *   catalina multiple_SD.c cogutil.c -p2 -lcx -lm                           *
 *                                                                           *
 \***************************************************************************/

/*
 * include the definitions of some useful multi-cog utility functions:
 */
#include "cogutil.h"
#include <stdio.h>

#define MAX_LINE_LENGTH 128

// need a lot of stack space to use stdio!
#define MY_STACK_SIZE (STACK_SIZE + 500)

/*
 * define some global variables that all cogs will share:
 */
static int read;
static int lock;

/*
 * read_function : C function that can be executed in a cog.
 *                (the only requirement for such a function is that it 
 *                be a void function that requires no parameters - to 
 *                share data with it, use common variables)
 */
void read_function(void) {
   int me = _cogid();
   char name[] = "fileX.dat";
   FILE *file;
   char line[MAX_LINE_LENGTH];

   cogsafe_print_int(lock, "Cog %d started!\n", me);
   name[4]=0x30+me;

   file = fopen(name, "r");

   if (!file) {
      cogsafe_print_int(lock, " ... cog %d failed to open file\n", me);
   }
   while (1) {
      if (read == me) {
         if (fgets(line, MAX_LINE_LENGTH, file) != NULL) {
            cogsafe_print(lock, line);
         }
         read = -1;
      }
   }
}

/*
 * main : start up to 7 cogs, then signal each one in turn
 */
int main(void) {
   int i = 0;
   int cog = 0;
   unsigned long stacks[MY_STACK_SIZE * 7];

   // assign a lock to be used to avoid plugin contention
   lock = _locknew();

   // start instances of read_function until there are no cogs left
   do {
      cog = _coginit_C(&read_function, &stacks[MY_STACK_SIZE*(++i)]);
   } while (cog >= 0);

   // now loop forever, signalling each cog in turn
   while (1) {
      for (cog = 0; cog < 8; cog++) {
         read = cog;
         // slow things down a bit so we can see the messages
         wait(200); 
      }
   }

   return 0;
}

ke4pjw

Attached is my wrapper for concurrent SD card access. When you open a file, it returns a filehandle. All operations will need to provide that file handle to read and write to the file. The methods in this object map to Nyamekye's FAT32 object, but it could be modified to support other filesystem objects.

ToDos:

• Directories are currently not supported. I may need to restructure the array for filenames or create a new structure for paths. This may further affect performance.
• Error checking and return values are not great. I just don't know the best way to code this. Making it deal with aborts the bubble up out of Nyamekye's object is cooking my noodle.
• There are many missing methods that are supported by Nyamekye's object.
• Write methods have not been tested. Use at your own risk.
• I had an SD card get scrambled using this wrapper. Seems to be SD card specific, as others have not had the issue.

I will develop this further as needed. Currently it is used in my April Fools gag, and seems to work OK. Performance is not great, but it does work.

--Terry

evanh

@ke4pjw said:

• Write methods have not been tested. Use at your own risk.
• I had an SD card get scrambled using this wrapper. Seems to be SD card specific, as others have not had the issue.

My recent experience with revamping one of the many bit-bashed drivers is that they are all broken at sequencing the SD card correctly. This particularly shows up with unreported randomly failed block writes. And not because of a crappy SD card either. It's primarily because of missing card busy checks in the sequencing.

Here's one that's fixed up.