Data Logging Options for the Propeller
Duane Degn
Posts: 10,588
I generally use a SD card when I want to log data with the Propeller but I'm currently working on a project which needs higher reliability than what I think SD cards provide.
Every so often when I use a SD card for data logging the card becomes corrupt and I end up needing to reformat the card in order to use it with the Propeller again. This doesn't happen often but it happens frequently enough to be a concern for this current project.
I was just looking over the Propeller Memory Card specs and I think the card's Winbond W25Q32FV chip looks like a good option for data logging. If I'm reading the specs correctly the chip retains its memory between power cycles. I think 4 Megabytes is probably enough memory but I'm not sure yet.
Do you guys have favorite memory chips? Any suggestions?
I should say I generally use Kye's SD driver. I'm pretty sure Kye's driver is picky about any sort of corruption on the SD card. I'm I ruling out SD cards too quickly? Can they be used reliably without any need to remove the cards?
Every so often when I use a SD card for data logging the card becomes corrupt and I end up needing to reformat the card in order to use it with the Propeller again. This doesn't happen often but it happens frequently enough to be a concern for this current project.
I was just looking over the Propeller Memory Card specs and I think the card's Winbond W25Q32FV chip looks like a good option for data logging. If I'm reading the specs correctly the chip retains its memory between power cycles. I think 4 Megabytes is probably enough memory but I'm not sure yet.
Do you guys have favorite memory chips? Any suggestions?
I should say I generally use Kye's SD driver. I'm pretty sure Kye's driver is picky about any sort of corruption on the SD card. I'm I ruling out SD cards too quickly? Can they be used reliably without any need to remove the cards?
Comments
My no brainer has been the Memory Stick Datalogger but it may not be in your budget or in you space constraints.
No problem up to 2 GB.
Besides, you are correct, it would be way too expensive. I'm looking for memory to into a board to be produced on a relatively large scale (at least compared with my many hobby projects).
We ran many industrial systems from Compact Flash cards for ages and I don't recall them ever failing. But that is a big interfacing problem. Can you even still get them ?
Winbond sounds like it's worth investigating. You can always use more than one for further capacity.
What I do is preformat the card with fixed files (8.3 names) and fixed sizes filled with zeros and in the process there is no fragmentation either and this I also rely on too. Now I encode all data as text with a null terminatation so I don't update any file sizes either, only the current sector may be rewritten several times before it moves onto the next. This simplifies the runtime operation of the datalogger and on power-up it just needs to scan through to find the latest file and a quick binary search to find the end-of-file termination.
Now it's just a question of what happens to this when you try to read the files on a PC as it won't complain about the formatting since it hasn't been touched. Text files that aren't full will say 1MB file size but might only have 100k in data but that is very easy to work around and is in fact viewable on most editors. Of course you could have an eject option on your datalogger which updates the file size etc.
Some of these dataloggers are accessed via FTP in which case I just report back the data file size rather than the FAT file size, no problem.
Alternatively since you are considering serial Flash you obviously are not worried about FAT so you can do exactly the same with SD, just treat it as one big memory, which is in fact how my EASYFILE system references it, either the whole card as up to 4GB or each file up to 4GB.
Normally I'd agree but Peter makes it sound like it's possible to use SD cards reliably.
I have a couple CF cards. I doubt this is a practical option nowadays.
I've been looking at some of the memory chips available and I'm a bit overwhelmed by the options.
This sure sounds interesting. I don't think I really mind keeping track of where all the data is located. I doubt we'll use SD cards in this project but I want to try your method sometime.
I don't think we would want to have a slot for SD card access. I think all the data will be retrieved with a USB connection (now without resetting the Prop), over a RS-485 Modbus connection or over Bluetooth.
Does FTP imply a ethernet connection?
I'll need to research FTP to see if it has advantages I can use with our present communication protocols.
I'll need to try your EASYFILE system sometime. I think I have other projects where it would be useful.
Anyway the main point is that SD cards are just Flash like any other Flash but accessed with the SD SPI protocol. There are some things under the hood to do with wear levelling etc but nothing that affects the access methods.
I use FTP for dataloggers using WIZnet chips such as the W5200 and W5500 or even the older W5100 as used on the old Spinneret. In fact if you click on the FTP link in my sig from your browser you should be able to connect to a Prop running EASYFILE and EASYNET software. You can even Telnet in on port 10001 and talk to to the system and even type an "ls -l" to list the files and details etc.
So if you only ever have an internal memory you don't ever have to worry about FAT compatibility, but it sure does help initially in getting files onto it in the first place if you need to. The EASYFILE software takes the pain out of it and in fact once you open a file you can access it as if it were virtual memory without having to be concerned about any file system stuff.
If your files are large you will find the FTP connection the fastest for transferring them but I have also accessed them over RS485 and Bluetooth as well. What I need to do next is interface one of the ESP8266 modules and get it working with FTP.
My IoT5500 +P8 Ethernet module, the size of a magjack incorporates a Prop and microSD (or Flash) and is all setup for datalogging.
Here's a local Telnet session to this unit although it is still running an older version of the software (I need to update it).
https://www.sparkfun.com/products/9530
If your data rate isn't too high, you can simply use FDS to interact with it.
IoT is great for adding tons of reliable storage to a data stream.
On my BASIC Stamp OWL platform I used the Atmel AT45DB081 serial dataflash chip (1Mbyte per 8-pin soic). It was very reliable. That IP is now in the hands of Adesto Technologies. Interesting product line. Not sure, but I think it works in a manner similar to the Winbond chips.
In a current project, I'm using eeprom, the AT24CM01 (128kByte) for both the boot 32k and 96k for logging. I had to convince the customer that it will be advantageous to limit the memory requirements. Let the Prop pre-process and compress the data into that smaller memory. One issue is how long it takes to offload data via USB via a serial bridge (FTDI).
The world is awash in too much data!
Don't you think that would be kind of insulting to the Prop? The Prop can certainly handle logging the data to the SD card directly. And at $25 a pop I have a feeling you didn't read the whole thread (something I'm often guilty of). $25 is much more than we would want to pay.
I believe the sensor can sample in the kilohertz range but I doubt we'll need to sample more than 10Hz. I think 80 bytes per second will be the close to the maximum amount of data we'll want to store.
I'm not sure what you mean by this. Does this assume a constant connection to some cloud storage? Is this done over WiFi? We've wondered about adding WiFi but I doubt that's going to happen. Bluetooth seems to work fine. We're wondering about having a smartphone collect the data via BT and then having the smartphone email the data. I suppose if we had WiFi on the sensor itself, we could have the sensor send the email directly.
It would be nice if we could do this too but I think 96k is just too little memory. I think we want a megabyte or two.
I don't think we want the deal with SD cards. There won't be a slot in the enclosure so the cards won't be accessible to the end user. Besides, I'm pretty sure the memory chip will use less board space which is an added bonus. I think for a few megabytes, a memory chip will probably cost less than adding SD card hardware and purchasing reliable cards (though I don't know enough about this to be sure).
On top of Peter's EASYFILESYSTEM in Forth, PropWare in C++ gives you direct access to the SD card for easy read/write of blocks as well.
I don't see a problem with using other MCUs along with the Propeller. Just because the Propeller can do something, doesn't mean that it's the best choice for the job. If the Propeller was that easy to use with SD, this thread wouldn't exist in the first place.
True, it is cost prohibitive (I did see your comment about the USB logger). I was more thinking along the lines of using the schematic to implement your own variant.
But, to your other point, I did somehow get caught up on the wrong part of your post. I focused on the SD card issue, not the part where you were saying that a dedicated IC would be good enough. The advantage of using the referenced winbond part is that there is already code for it. It looks like you could get a 16MB drop-in replacement for less that $2 each (actually, up to 64MB, but 16MB was the largest digikey sold in unit quantities).
I've only used SD cards with the FAT filesystem.
I'll take a look at your driver for other projects but I doubt we'll use a SD card for this project.
Am I correct to assume if all we need is a few megabytes of storage, an SD card would be overkill?
do a search for W25Q128 (or W25Qxxx, where xxx is the number of desired megabits).
That did it.
Thanks.
I believe the sensor can sample in the kilohertz range but I doubt we'll need to sample more than 10Hz. I think 80 bytes per second will be the close to the maximum amount of data we'll want to store.
I'm not sure what you mean by this. Does this assume a constant connection to some cloud storage? Is this done over WiFi? We've wondered about adding WiFi but I doubt that's going to happen. Bluetooth seems to work fine. We're wondering about having a smartphone collect the data via BT and then having the smartphone email the data. I suppose if we had WiFi on the sensor itself, we could have the sensor send the email directly.
[/quote]
I'm sure that the ESP8266module can do this, but you wouldn't be able to use the free online IoT sites like Thingspeak, they use 15 second intervals.
If it is feasible to use a smartphone, Android will take care of this. I'm not sure about sending the emails with data, that might require some custom software.
Here's a PDF on the Winbond chips.
As for SD cards, we're talking orders of magnitude larger capacity and built-in wear leveling. I have them often in boxes with no external access slot, and the socket real estate can often be found on the bottom side of the PCB where it doesn't seem so intrusive. I use a modified version of FSRW.
Glad to hear it. As always, do let me know if you have questions.
What's your definition of overkill? I'll check tonight, but the code size required to interface with an SD card shouldn't be very large. Most of the logic normally associated with SD cards come from the filesystem, not the physical storage device. This is all bogus until I can put some numbers to it though, so bear with me until I run some tests this evening.
Program 1, with SD routines
#include <PropWare/filesystem/sd.h> #include <simpletools.h> int main () { PropWare::ErrorCode err; uint8_t buffer[PropWare::SD::SECTOR_SIZE]; PropWare::SD sd(PropWare::SPI::get_instance()); err = sd.start(); if (err) { putStr("Error occurred during startup: "); putDec(err); putChar('\n'); return err; } const unsigned int addressOffset = 1024 * 1024; unsigned int currentAddress = 0; while (1) { err = sd.read_data_block(currentAddress, buffer); if (err) { printi("Error during read of address %d: %u\n", currentAddress, err); return err; } err = sd.write_data_block(currentAddress + addressOffset, buffer); if (err) { printi("Error during write of address %d: %u\n", currentAddress, err); return err; } ++currentAddress; } return 0; }And Program 2, without SD routines
#include <PropWare/filesystem/sd.h> #include <simpletools.h> int main () { PropWare::ErrorCode err = CNT; uint8_t buffer[PropWare::SD::SECTOR_SIZE]; if (err) { putStr("Error occurred during startup: "); putDec(err); putChar('\n'); return err; } const unsigned int addressOffset = 1024 * 1024; unsigned int currentAddress = 0; while (1) { err = CNT; if (err) { printi("Error during read of address %d: %u\n", currentAddress, err); return err; } err = CNT; if (err) { printi("Error during write of address %d: %u\n", currentAddress, err); return err; } ++currentAddress; } return 0; }And the results:
Hard to beat $0.43/ea
http://www.ebay.com/itm/10pcs-W25Q32FVSSIG-W25Q32FVSIG-25Q32FVSIG-4M-Memory-Flash-SOP-8-SMD-/201098254024?hash=item2ed263d6c8
Thanks for the comparison. I know I'll need to start using C more at some point but my current project is in Spin PASM.
It sure would be nice to have an easy way of reading code from external memory.
I can read in PASM "cogjects" from upper EEPROM but this is a really awkward way of creating large programs.
I think C has a lot to offer but I'm really pleased with what I can do with Spin/PASM and it's hard for me to want to go back to not knowing what I'm doing by programming in C. Though I suppose periodically not knowing what one is doing is a good way to keep life interesting.
Those are tempting for my own projects but there are so many counterfeit chips on ebay, I don't think I'd trust them in a project I'm doing for someone else. Thanks for the link.
I don't know whether this is applicable for your project, but you can also put an SPI memory chip into MicroSD form, which gives you a choice into the future for being able to plug in larger memories/other memory technologies
http://forums.parallax.com/discussion/144082/microsd-party-board-spi-widgets-for-microsd-enabled-prop-boards
As for the Tachyon memory model it still has plenty of memory left over even with FAT32 and datalogging handling up to 4 open files. It is only when I implement WIZnet drivers and Ethernet server protocols that I use part of the upper half of the 64k EEPROM for dictionary storage so that all code fits nicely in RAM, no other tricks although SD can be used in for the dictionary in place of the EEPROM if need be.
The nice thing about developing in Tachyon is that you can develop and test interactively and even while the datalogger is running you can access the files and buffers to see what is going on, even inject data etc. This also applies once the system is deployed and being able to Telnet in is a bonus, even add new functionality to it via the network.
Thanks for the reminder about those cool gizmos. I doubt we'll be putting a uSD socket on this board but I'll remember those cool tricks for future projects.
You're going to get me to try this sometime. I've never used Forth but I loved my old RPN calculator and I hear the thought process of using Forth is similar to RPN.
You'll need to forgive me if I don't suggest a switch to Forth for this project.
if you do not try Tachyon soon ;-)
and yes - having used a RPN calculator helps
It's funny how people look at the Forth stack as being "reverse" when in fact the common method is the reverse from the Forth point of view. I mean if we parse the infix notation 12 + then we have to ask "what's next?" and proces this and then find out what to do with the result? In Forth there is never this problem as + does not have to compile anything, it just adds two numbers that are processed and sitting there ready on the stack. A conventional compiler has to perform mutiple passes of the source code to "make sense of it" and break it down into efficient runtime code. In Forth the source does not need to be broken down, it is already in its most efficient form ready to run much like PASM. A benefit of this is that you know what is actually happening rather than scratching your head only to find out that the compiler needs whitespace or the braces in the right places
To me the stack is very natural for programming as I find passing parameters on the stack easier although I don't prefer RPN calculators. It is awkward though to manipulate parameters by position which is also why I minimise the number of parameters passed and refuse to implement PICK and ROLL or stick to "tradition" when it isn't practical.
Anyway it would be remiss of me to not mention the use of Forth or more importantly Tachyon for this project as all the pieces are in place and have been put together and work. No need to try and make it fit and work and I've successfully used SD cards and Forth for datalogging in commercial volumes for many years.
I'm a hands-on guy and I love that the code is alive, that I can interact with it on the target in-situ rather than being tied to the PC compiler tools and dealing with the project in an abstract and almost bureaucratic way.
Yes, as MJB says, the only one you need to forgive is yourself