SD Card Speed

Discovery

Using the Propeller development card that includes a microSD memory chip...how fast can the propeller write and read data from the device?

Sincerely,

Discovery

Discovery

My application calls for writing one Mega words of data into the SD memory on the Propeller Activity Board at varying rates but the data must be read out at very high speed. I am looking for a read data rate around 500 kHz.

I was wondering if anyone has experience using one Cog of the propeller to read out data as fast as it can. The data also must be uniformly spaced in time, i.e. no data bursts.

My application is written in C.

Sincerely,

Discovery

DavidZemon

Discovery wrote: »

My application calls for writing one Mega words of data into the SD memory on the Propeller Activity Board at varying rates but the data must be read out at very high speed. I am looking for a read data rate around 500 kHz.

I was wondering if anyone has experience using one Cog of the propeller to read out data as fast as it can. The data also must be uniformly spaced in time, i.e. no data bursts.

My application is written in C.

Sincerely,

Discovery

I can throw some numbers at you, but it will be apples to oranges at the moment. Are you writing raw blocks to the SD card (without regard for a filesystem), or do you need your SD card to be formatted as FAT 16/32 so that a computer or other system can read the data at a later point? The benchmark results are drastically different depending on your answer.

Also, thank you for providing your application's language. The SD card and FAT routines that are shipped with PropGCC and the Simple Libraries are not very fast. No matter your answer to my above question, you will probably want to either use PropWare (if it's fast enough for you), convert an existing Spin object with Spin2cpp, or roll your own, highly-optimized version.

Peter Jakacki

Discovery wrote: »

My application calls for writing one Mega words of data into the SD memory on the Propeller Activity Board at varying rates but the data must be read out at very high speed. I am looking for a read data rate around 500 kHz.

I was wondering if anyone has experience using one Cog of the propeller to read out data as fast as it can. The data also must be uniformly spaced in time, i.e. no data bursts.

Maximum clock speed for an SD card in SPI mode is 25MHz although the most that the Prop can do is 20MHz @80MHz clock. However SD cards by their very nature do not output a continuous regular stream of data, the Flash memory is managed by a controller chip on the card which maps and wear levels sectors etc. So at the SPI layer commands are issued to the SD card controller (SD we'll call it) and then waits for however long that SD needs after which the SD sends back a token to say it's ready. Then and only then do you get to read the sector. If the file is non-fragmented then the sectors or blocks can be addressed sequentially without having to read the FAT table.

If you really want sustained high speed reads I suggest that you try dual QSPI Flash but then you might only have 256M bytes or so even with high density chips.

I know that when I play back WAV audio files from an SD that you can't have any breaks so after one buffer is filled and being played back another buffer is being filled so that they alternate functions. This is fine as long as the playback or read out rate is slower than the average SD read data rate. So if you are able to ensure contiguous blocks in a FAT32 file which is normally the case anyway, you could write a highly optimized driver over a couple of cogs to do what you want and maybe run the Prop at 100MHz and maybe it might work.....maybe.

Discovery

Very good. It then appears that getting data clocked out at 500 kHz should be possible. I have no idea what would be the right format on the SD. I just have raw numbers that will be stored in sequence at a rate depending upon the length of time it takes to complete the calculations. For cutting thick steel, the cutting speed will be about 15 inches per minute as opposed to 22 gauge steel at 822 inches per minute so I expect to change the clock rate to the SD for cutting materials.

I will be out of contact with the Forum for February and a few days into March. Then I will get back and try to work with you on writing the spin code. I have never written spin programs only C.

Sincerely,

Discovery

Peter Jakacki

Spin won't cut it, excuse the pun, but it's too slow for that. I have my own way of doing it but I'm sure the C guys will help you out.

DavidZemon

Discovery wrote: »

Very good. It then appears that getting data clocked out at 500 kHz should be possible. I have no idea what would be the right format on the SD. I just have raw numbers that will be stored in sequence at a rate depending upon the length of time it takes to complete the calculations. For cutting thick steel, the cutting speed will be about 15 inches per minute as opposed to 22 gauge steel at 822 inches per minute so I expect to change the clock rate to the SD for cutting materials.

I will be out of contact with the Forum for February and a few days into March. Then I will get back and try to work with you on writing the spin code. I have never written spin programs only C.

Sincerely,

Discovery

Sorry - I guess my real question got a bit lost in there. Do you need the SD card to be read by a computer or other system after the Propeller writes to it?

Mickster

Discovery wrote: »

Very good. It then appears that getting data clocked out at 500 kHz should be possible. I have no idea what would be the right format on the SD. I just have raw numbers that will be stored in sequence at a rate depending upon the length of time it takes to complete the calculations. For cutting thick steel, the cutting speed will be about 15 inches per minute as opposed to 22 gauge steel at 822 inches per minute so I expect to change the clock rate to the SD for cutting materials.

I will be out of contact with the Forum for February and a few days into March. Then I will get back and try to work with you on writing the spin code. I have never written spin programs only C.

Sincerely,

Discovery

You need 500KHz for a machine tool? Doesn't sound right.

Discovery

In answer to Mickster...if you understood the design you would say okay.

In answer to DavidZemon...I was hoping to use the propeller to readout the words from the SD as implemented in the Propeller Activity Board. As a test, I loaded a series of "1s" and "0s" in the SD then clocked them out to an oscilloscope to measure the speed. The output speed was 39.0625 kHz.

My interest now is to locate a shift register that is four bits (a nibble) by 46 giga nibbles deep. The propeller would do the calculations and load the shift register to the depth necessary to cut out the part and the readout would use a dedicated circuit. I noticed a German CMOS shift register of 128 bits deep. A really big shift register would be perfect but I have not located one.

I wonder if the propeller could write the nibbles to a large SD memory and a dedicated circuit could rapidly (say a 500 kHz clock) readout the values. I don't know what the dedicated circuit would entail.

Any ideas.

Discovery

DavidZemon

Let me ask in a different way: do you need a standard partitioning scheme/filesystem (like FAT) on the SD card? Having a filesystem like FAT on the card allows you to read the card on a PC with ease. Without the filesystem, it will be more difficult to get the data off of the SD card and onto a computer if that ever becomes necessary.

Peter Jakacki

Discovery wrote: »

In answer to Mickster...if you understood the design you would say okay.

In answer to DavidZemon...I was hoping to use the propeller to readout the words from the SD as implemented in the Propeller Activity Board. As a test, I loaded a series of "1s" and "0s" in the SD then clocked them out to an oscilloscope to measure the speed. The output speed was 39.0625 kHz.

My interest now is to locate a shift register that is four bits (a nibble) by 46 giga nibbles deep. The propeller would do the calculations and load the shift register to the depth necessary to cut out the part and the readout would use a dedicated circuit. I noticed a German CMOS shift register of 128 bits deep. A really big shift register would be perfect but I have not located one.

I wonder if the propeller could write the nibbles to a large SD memory and a dedicated circuit could rapidly (say a 500 kHz clock) readout the values. I don't know what the dedicated circuit would entail.

Any ideas.

Discovery

46 giga nibble shift register hey? Do you know what you are doing? I'd say you are trying to describe the problem by a method that you think is best, a common mistake and one that will paint you into a corner. How about just describing the application, what the current problems are, and what you would like to achieve. You talk of a read data rate of 500kHz but is that bits or bytes and why is this necessary and how much data before you stop or repeat etc?.

Discovery

All you need to know is that the CNC machine uses a propeller to calculate and output the steps to x, y, and z stepper motors. The x and y steppers are high speed motors but the propeller cannot compute and output the steps fast enough to cut thin materials. The CNC works perfectly well at moderate speeds. The need is to modify the existing design using the propeller to do the calculations and find some way to increase the output data rate. It is clear that the propeller cannot accomplish this task by itself. I therefore thought of generating a plot file after all the steps were computed by the propeller. The propeller's limited memory is not sufficient for the task so I thought that the SD memory on the Propeller Activity Board might be the answer. I generated a series of nibbles representing the stepper motor clocking sequence and loaded them into the SD memory on the Propeller Activity Board...however, the SD would only hold 300 nibbles, the output was slow, and the output nibbles were not spaced uniformly in time...i.e. bursts of output. I then thought of the simplest method of storing an array of four-bit words so that they could be written into storage at a non-uniform slow rate then readout at a uniform high-speed rate. That device is a shift register. The four bit nibbles are coded for all combinations of moving the x and y steppers in clockwise or counterclockwise directions, x only, y only, as well as x and y at the same time. There are specifically eight values that are decoded (2, 3, 8, 10, 11, 12, 14,and 15). Therefore, each nibble represents one step of the motors as the CNC cutter describes the shape of the object to be cut. The cutter can handle steel or aluminum raw stock 7 feet long by 3.5 feet wide with a precision of 0.0005"; therefore, the number of nibbles gets to be quite large. Yes, Peter...I do know what I am doing. I do not understand how to format a memory stick to accept a simple linear array of numbers. Clocking nibbles into a shift register then clocking them back out in reverse through a fast decode then driving the motors is the simplest concept I can think of for making a modification to the existing design. I would rather find a modification to the existing design than scrap it and start over.

Discovery

kwinn

Discovery wrote: »

.......

My interest now is to locate a shift register that is four bits (a nibble) by 46 giga nibbles deep. The propeller would do the calculations and load the shift register to the depth necessary to cut out the part and the readout would use a dedicated circuit. I noticed a German CMOS shift register of 128 bits deep. A really big shift register would be perfect but I have not located one.

............
Any ideas.

Discovery

What you are looking for is a "FIFO", First In First Out, and I have not heard of one even close to the size you want.

Large FIFO's can be built using ram chips and some logic chips or a fast micro, or an FPGA.

kwinn

As per Peter's post a better description of the application and problems will get you better more focused suggestions.

Peter Jakacki

"My interest now is to locate a shift register that is four bits (a nibble) by 46 giga nibbles deep"
"I noticed a German CMOS shift register of 128 bits deep. A really big shift register would be perfect but I have not located one."

This is what I mean by "do you know what you are doing".

46 giga nibbles would be 23 GB of memory which is totally impractical in terms of RAM for instance where you could have programmable logic cycling through the memory. In terms of SD memory it is achievable but then you would have to buffer it with one processor and read it out with DMA at the desired rate after a large enough amount has been buffered.

When you can only think "lots of nibbles" as a way to solve the problem it just means you haven't thought the problem through enough. Think outside the square even if it seems far fetched.

It is possible to read out data from an SD card at this rate but of course you would have one cog filling the buffers and another cog reading alternate nibbles from a long in the buffer and outputting that every 2us. But then you also have to tell the SD cog to wait as well to allow the "nibbler" to finish with its current buffer etc.

BUT, this is where you need to think this through. You are talking about writing 23GB to the SD card. Even at 250kB/sec that would take a full day to write. Do you really mean 46 "giga nibbles"?

Tracy Allen

@discovery, Here is another insightful thread about achieving fast reads.
http://forums.parallax.com/discussion/146447/what-is-the-current-state-of-sd-card-interface-driver-implementations

Your requirements started at 1 megaword (word?), then somehow evolved to 46 giga-nibbles. Quantitative disconnect. Huh?

Discovery

The original concept was based on cutting a retangle of 7 feet by 3.5 feet arriving at 1. 075 mega nibbles. Then I took a look at including in the same cutting pass several circles, squares, triangles, and special conic sections. That put the number up to 46 giga nibbles.

There is no reason that separate cutting passes can be made and keep the single cutting pass at 1.075 mega nibbles.

Discovery.

Peter Jakacki

So many ways of doing this but I know I can read 256kB/sec with Tachyon on P1 and fragmentation is not an issue anyway I find on SD cards. So a sector by sector read every 2ms or so would keep a circular buffer filled while another cog simply reads from the other end of it and delivers nibbles to the target every 2us. The nibbler cog also signals when it is finished with a sector so that the SD cog can top it up again. You won't be limited to a mere 1M nibbles and the limit has more to do with what is practical.

BTW, some SD card drivers may even read a sector using a 20MHz clock but the limit is mainly the latency of the SD card getting the sector ready and also the overhead of the software in processing the data.

DavidZemon

You're definitely going to be writing this one in assembly, so I'll let our other forum members like Peter help you out the rest of the way. I'll keep an eye on the thread and jump in if you need help getting your assembly hooked into the rest of your C/C++ application.

DavidZemon

Hm... perhaps you don't need to code this in pure assembly. I just wrote a quick little program that uses PropWare's SD card driver, running in LMM mode @ 80 MHz and it averages 278 kB/sec

Here's the output:

Total time:            17974918 us
Average transmit rate: 284841.343750 B/s

and the code

#include <PropWare/utility/utility.h>
#include <PropWare/hmi/output/printer.h>
#include <PropWare/memory/sd.h>

using PropWare::SD;
using PropWare::Utility;

const unsigned int SECTORS       = 10000;

int main() {
    PropWare::ErrorCode err;
    const SD            sd;
    const uint16_t      sectorSize = sd.get_sector_size();
    uint8_t             buffer[sectorSize];

    check_errors(sd.start());

    const unsigned int startTime      = CNT;
    for (unsigned int  i              = 0; i < SECTORS; ++i)
        sd.read_data_block(i, buffer);
    const unsigned int endTimeUs      = Utility::measure_time_interval(startTime);
    const float        endTimeSeconds = endTimeUs / 1000000.;

    pwOut << "Total time:            " << endTimeUs << " us\n";
    pwOut << "Average transmit rate: " << SECTORS * sectorSize / endTimeSeconds << " B/s\n";

    return 0;
}

Cluso99

The video circuit is capable of serially outputting up to 32 bits at a time, concatenation additional 32 bit blocks. Or it can output 32 bits serially as 8x nibbles. IIRC 80 MHz clocking can be achieved without overclocking.

So as long as you can feed the data into the prop fast enough, you can convert and output it fast enough.

As Peter has said, describe your actual requirements because you're not thinking about your problem properly.

4given

I've been dreaming of building a CNC machine for some time, but never thought to store every bit change like you have done. I have been with pencil and paper for a bit here and realizing what it means to have to store 4000 nibbles to move an XY machine one inch! I am impressed that you have made this work, and it speaks of your enthusiasm for the project. Rough calculations, your 46G nibble plot file would fill 5 DVD's. Just managing and storing these plot files and having to pre-compute them is going to be prohibitive.

Often a stepper motor driver is used to take a little load off the controller. It translates simple 'step' and 'direction' into the necessary coil drives rather than tasking the main controller with that mundane and granular task. I didn't see the pattern I was expecting with your step sequences of 2,3..14,15 and cant tell if you know about step and direction driver boards. In the off chance that you don't, and for the benefit of others, there is a hobby friendly board for $20 here https://www.sparkfun.com/products/12859 'big easy driver' 2A/Phase.

Just because you can see the peak and the whole side of the mountain, doesn't mean there isn't a better way to conquer it even if it involves what is unknown.
The propeller is an amazing and very powerful processor, you won't be disappointed. Consider investing some of that energy into reading G Code, or learning more about how to unleash the power of the prop. In case you haven't, check out Don Starkey's Propeller CNC control package. You are sure to learn from it. http://obex.parallax.com/object/85

Do I know what I am doing? I don't know, this is my first post ever on any forum. I offer some secret insight to the high project success rate that I now enjoy when working with electronics: The ugly frustrations and failures of the present, stand to make you beautiful in the future, if you persevere. Don't give up!

Ltech

Use the SD card only to uplodade a ram/ eeprom @ low speed. And do the job from the faster ram /eeprom after the uploade ?
Or use two SD cards and mix the data to get the speed ?

Peter Jakacki

On a Sandisk Ultra 8GB I get sector read times which includes command response phase and reading in 512 bytes of 1.6ms. I had an older Sandisk which was taking over 2ms.

---

( 0031 $3C78  ok )   $40.0000 $40 ADO CR I .LONG SPACE I LAP SECTOR LAP .LAP LOOP

0040.0000 624 cycles at 96000000Hz  or 6.500us 
0040.0001 154688 cycles at 96000000Hz  or 1.611ms 
0040.0002 154688 cycles at 96000000Hz  or 1.611ms 
0040.0003 154688 cycles at 96000000Hz  or 1.611ms 
0040.0004 154688 cycles at 96000000Hz  or 1.611ms 
0040.0005 154688 cycles at 96000000Hz  or 1.611ms 
0040.0006 154688 cycles at 96000000Hz  or 1.611ms 
0040.0007 154688 cycles at 96000000Hz  or 1.611ms 
0040.0008 154688 cycles at 96000000Hz  or 1.611ms 
0040.0009 154688 cycles at 96000000Hz  or 1.611ms 
0040.000A 154688 cycles at 96000000Hz  or 1.611ms 

Discovery

Responses:

When I make the upgrade to my CNC plasma cutter I will use very high speed step and direction stepper motors produced by ClearPath which have built-in algorithms for controlled direction reversal without missing a beat. They operate on 75 VDC with 5 volt logic control.

DavidZemon...that is what I am looking for. 278 kB/sec. That is good enough for my application. I will try to duplicate your SD driver code above on my Propeller Activity Board and send driver signals to my ClearPath stepper motor as a test. Thank you.

David...I don't have your ability to sit down and write code from scratch. All of the code that I use is from users who supply a piece of code that I integrate into my controls.

Sincerely,

Discovery

Discovery

David...I clicked on your PropWare link in your response above then clicked on Download, Windows. The down load began but Agast found a problem in the download from your file. I directed Agast to fix the problem then downloaded the PropWare again. The .exe file was loaded.

Upon running your code on my machine...I get an error stating that #include <PropWare/utility/utility.h> can not be found.

What do you suppose is the problem?

Discovery

Discovery

David...The problem is that Avast detected a threat in your downloaded software and blocked it from executing on my machine.

I will try another source.

Discovery

DavidZemon

Discovery wrote: »

David...I clicked on your PropWare link in your response above then clicked on Download, Windows. The down load began but Agast found a problem in the download from your file. I directed Agast to fix the problem then downloaded the PropWare again. The .exe file was loaded.

Upon running your code on my machine...I get an error stating that #include <PropWare/utility/utility.h> can not be found.

What do you suppose is the problem?

Discovery

What is your programming environment? If you're using SimpleIDE, you should following the instructions at the very top, with the heading "SimpleIDE (Any Operating System)". The file you downloaded is only if you're using a command line for compiling your code, or an IDE such as CLion or Eclipse.

Also, I'm a fan of keeping threads clean and on-topic. So if you have any more issues with PropWare, let's talk about them in this thread.

Discovery

I am using SimpleIDE.

Discovery

Discovery

David...I read your documentation on PropWare and it sounds impressive; however, I have over 100 industrial projects written using SimpleIDE and I have concerns that if I setup PropWare on the same computer, something undesirable could happen to them.

I am thinking about purchasing a small laptop and using it for PropWare project development...just to be absolutely safe.

Discovery

DavidZemon

You could... doesn't sound necessary to me but it wouldn't hurt. The downloaded recommended by my SimpleIDE instructions does not contain any executables: just header files and static libraries (so there's no need to worry about conflicts with other projects from this aspect). Then the instructions for making PropWare work with SimpleIDE must be repeated on a per-project basis - which means making one project work with PropWare will not have any affect on another project in SimpleIDE. So, it will be perfectly safe to have PropWare installed on a machine with non-PropWare projects.