Seeking Project Input: Can the a Prop (or two) handle this?
pgbpsu
Posts: 460
I've been working with the Prop on and off for ~3 years. I think it's a great chip and I'm constantly amazed at what people (especially someone like me with very limited programming experience) are able to do with it. I've now reached a point in my project development where I think it might be time to move to another chip/ucontroller. I'm hoping folks here can convince me that there's still room for the prop to be the main player in my design, but I think there's enough evidence to suggest I move to something else. Based on all the info available about the Prop II, it would solve my problem, but I can't really wait another year.
With a lot of help from this forum, I've built a 4-channel, 24-bit wireless data-logger with the prop at the center of it all. The system has two main modes: continuous recording (at 1250 sps) or triggered (at 10Khz). I use FSRW to store data to an SD card, I've got 2 2-channel 24-bit ADC that pass data into the prop at 80Khz on 4 channels. There it's averaged down to the desired sample rate (8->1 for triggered mode or 64-> 1 for continuous mode). It also has a precision GPS onboard which I used for accurate timing and general location.
The point is I think we've done an awful lot with what we've got, but I'm bumping up against a few things I just can't solve easily. The biggest issue is need of a FIR/decimation filter. I just don't see anyway to fit this into the current prop. I'm out of cogs (although I'm sure a better coder could squeeze one back), and even if I had one I don't think the prop has the speed to do a 6-tap (integer) filter on 4 channels at 80-Khz. The other big issue is moving data from the SD card off to a laptop. The direction the project is going has the SD card residing in the unit, not easily to be removed. We'll install a USB bulkhead that the user would plug into to retrieve data. The data rates and volumes are high so we'd like to somehow connect the SD card directly to a USB device so when one connects to it, it simply looks like a thumb/jump drive on the laptop.
There are other issues but they are minor. I wonder if folks here have thoughts on how I can solve this FIR filter and USB issue. If anyone has experience with other IC that would maybe help offload some of these problems, the prop could stay at the center of the action. That's my preference because I know how to work with it. I don't have experience with other micros although I'm working on learning Verilog/Xilinx/FPGA.
Thanks for reading this and for any feedback.
Regards,
Peter
With a lot of help from this forum, I've built a 4-channel, 24-bit wireless data-logger with the prop at the center of it all. The system has two main modes: continuous recording (at 1250 sps) or triggered (at 10Khz). I use FSRW to store data to an SD card, I've got 2 2-channel 24-bit ADC that pass data into the prop at 80Khz on 4 channels. There it's averaged down to the desired sample rate (8->1 for triggered mode or 64-> 1 for continuous mode). It also has a precision GPS onboard which I used for accurate timing and general location.
The point is I think we've done an awful lot with what we've got, but I'm bumping up against a few things I just can't solve easily. The biggest issue is need of a FIR/decimation filter. I just don't see anyway to fit this into the current prop. I'm out of cogs (although I'm sure a better coder could squeeze one back), and even if I had one I don't think the prop has the speed to do a 6-tap (integer) filter on 4 channels at 80-Khz. The other big issue is moving data from the SD card off to a laptop. The direction the project is going has the SD card residing in the unit, not easily to be removed. We'll install a USB bulkhead that the user would plug into to retrieve data. The data rates and volumes are high so we'd like to somehow connect the SD card directly to a USB device so when one connects to it, it simply looks like a thumb/jump drive on the laptop.
There are other issues but they are minor. I wonder if folks here have thoughts on how I can solve this FIR filter and USB issue. If anyone has experience with other IC that would maybe help offload some of these problems, the prop could stay at the center of the action. That's my preference because I know how to work with it. I don't have experience with other micros although I'm working on learning Verilog/Xilinx/FPGA.
Thanks for reading this and for any feedback.
Regards,
Peter
Comments
However, if you have not been exposed to interrupts yet, you will find moving to another micro another big challenge.
Having said this, there are some things a Prop cannot do. When you reach this stage, another chip has to be sought.
If you plan to travel the FPGA route, I suggest $50 is well spent on the Avnet Spartan 3A kit for starters. There are others with the DSP blocks that are more expensive, but this is a cheap way to take a look at FPGA and decide if the effort warrants it. You will have to learn either Verilog or VHDL. Here, you are making the processor and other equipment as well as having to program it, so the task is another magnitude of complexity.
Having said that the Prop is constantly being used to do things that are commonly agreed to be impossible:)
I think perhaps you could say more about your app to see where this leads.
For example how are you getting the analog data into the prop? Is that 4 channels of 20Ksps making 80Ksps or 80Ksps each. You mention a max rate of 10Ksps and then averaging down to one eighth or one sixtyfourth of that. So why the filter needs to run at such a high rate. Do you really need 24 bit, seems excessive? And so on.
USB is probably better tackled by an external chip.
BTW do you ever really get 24bits of real resolution? In my experience it is hard to get the noise floor low enough to make use of such high resolution. I just ask because going to 16bit might let you do some clever coding, packing two measurements into a long or something.
Graham
These "impossible" things are much easier to tackle if you can make the spec. more realistic.
I have have an FPGA development board on my bench already which has the Spartan 3E. I'm working through a number of labs/demos to get the hang of this. My current idea is to team an FPGA with the Prop. The FPGA can get data directly from the ADCs filter it, decimate it and hand over a much lower bandwidth signal to the Prop. I think it's possible to build a USB thingy on the FPGA that would read data straight from the SD card and pass it out to the USB bulkhead. Problem is I have no idea how to do that, but a month ago I couldn't blink any lights on my dev board and now I can put color bars on an VGA monitor.
@Heater-
I'm using 2 AD1871 ADC chips. Each chip produces an 2 channels of 80Khz data. They want to run at 96Khz, but I've tuned them down so I can keep up. Running at this rate allows me to oversample and allows me to decrease noise by averaging. However, I believe I should really be filtering these data before averaging them to avoid aliasing. So the filtering needs to be done at 80Khz. Any packing of the data can't be done on the raw data. It could be done after the filter has been applied. I do actually pack the data a bit before writing to disk. It saves a bit of space and speeds up the write to disk side of things. Any suggestions for an external chip that does USB?
@Heater & Leon-
Believe it or not I have one of Leon's toys on my bench from last spring. I worked through a few examples, but I'm not well versed in C (or XC) and found the sledding rather hard. The documents are pretty thick and I had trouble finding example code that showed how to share data in "HUB" memory/between cores. Pretty much the opposite experience to what I've had with Parallax. I suspect you are right and that solution would work, but for me the learning curve there is extremely steep. I may have to revisit that.
@Graham-
The reason we don't post filter is because we can't keep all those samples around. If I could store the 80Khz signal then we could filter it later on a PC, but there's no way to capture all that data to an SD card. It simply comes too fast. Once I've averaged or decimated it, those samples are gone and can no longer be filtered. If you can convince me that I don't need to filter I'll be nearly in the clear, but we've been round and round about it here and it seems necessary for more than simply anti-aliasing. We'd actually like to install a downsampling filter that would allow a number of different sample rates. Currently we are limited to 10Khz (8->1), 5Khz (16->1), 2.5Khz (32->1), 1250hz (64->1) below that we're splitting samples which starts screwing the sampling.
I really believe my problems would all go away if Prop II were here. I'm afraid by the time I get one of these other solutions working it will be here. But I can't sit and wait. It will never come. If I get another working solution, Prop II will be available the day before or after I get it all working!
Any and all thoughts are welcome.
You share data via the links. Try the support forum if you have problems, people there are very helpful.
Also, if you tristate all the driver pins to the SD card from the prop, and disconnect the power and ground pins with a relay, I'm betting you could even have a ribbon cable pass-thru from the SD socket on-board to a "SD jack", sort of a dummy SD card to plug directly into a PC's SD card slot.
Jonathan
We're using 4 ADC channels and one timer channel (generated internally by capturing the number of system clock counts since the last Pulse/second from our GPS). So the data rate without any compression would actually be 5 channels * 96Khz * 4 bytes/sample = 1.92Mbytes per second. I think that's well beyond even the magic currently being done with FSRW. The added problem is that there is a LOT of other stuff going on so memory is pretty much all used up. We take these incoming data and strip out a VERY low sample rate version to send back over a radio link just for QC. We also have 4 serial ports running (2 GPS based ones, one debug, and the XBee radio). There's also a ~300bytes/second data stream coming from the GPS that we store. My solution so far has been to buffer that for as long as possible, then quickly close the seismic file on disk, open the gps one, write, close and reopen the seismic file. All that monkeying around means buffer space so I don't lose any samples.
If writing all the samples to disk were possible we'd consider doing it, but it does create a problem with storage. A 16Gbyte disk that would have lasted months at 500sps now only lasts days....
I can't pass this opportunity to thank you for all you've done on FSRW. It's been a lifesaver. Any interest in doing an ext2 filesystem?
http://forums.parallax.com/showthread.php?t=125255&highlight=ext2
Peter
Can you post the exact filter you are wanting to implement? Also, if you wanted to zip and email me a representative data set, I could use that to play around with, see if any other ideas pop up. (same user name as here, at gmail dot com)
thanks,
Jonathan
That's very kind and I'll take you up on it. I don't have a raw dataset to work with because I haven't yet captured that 96Khz stream to disk. I've always taken it into the Prop and averaged 8 samples into 1. I need to write some code (as a one off) that will let me capture several seconds of high speed data, even on one channel. As seismic data it's mostly noise until there's something interesting. There are lots of good compression schemes, but with all that's already going on in our code, there's not a whole lot of space to do anything intense. I'll see what I can capture in the next week.
Thanks,
Peter
Jonathan
I think this is quite reasnable to do the 6-tap FIR if you have 1 cog available or better yet if you can dedicate two cogs to each fetch half the ADC samples and filter half the ADC samples.
First a few questions. How precise do the FIR coefficients have to be? And are the FIR coefficients symmetrical?
Assuming that 8-bits is enough for the FIR coefficients, I'd make a custom unrolled 24-bit X 8-bit = 32-bit multiply in assembly. Using the basic 2 instruction per bit multiplication algorithm shown in the "propeller guts.pdf" it should take 2x8 instructions plus like a 7 instruction overhead for a total of 23 per tap per sample. Lets say using a 64-bit accumulator adds an extra 3 instructions per tap per sample. So for the full filter you'd have 156 instructions per sample. Expand this out to full data rate on two channels and this is 24,960,000 insturctions/sec. Just feasable with an overclock to 100MHz. I've also likely overestimated the overhead needed by a few instructions so you could eke out some more breathing room.
If the taps are symmetric, we should be able to cut the number of instructions per sample nearly in half by remembering intermediate results. I'll assume the filter looks roughly like this => I(k)*a + I(k-1)*b + I(k-2)*c + I(k-3)*c + I(k-4)*b + I(k-5)*a = filtered_sample*2^n Then the I(k)*a term for the current sample is the I(k-5)*a term 5 samples later. Also the I(k-1)*b term is the I(k-4)*b three samples later. Finally the I(k-2)*c is the I(k-3)*c term when the next sample arrives. In all these cases there is no reason to re-calculate the values if they can be stored somewhere. If we assume this trick increases speed 1.8x the 6-tap filter should now take 13,870,000 instructions each second for half the data. Hopefully this then leaves enough instructions to interleave reading the ADC and writing the results to a buffer in hub-ram.
Third, if the filter coefficients are fixed constants, a specialized super fast multiply routine can be made. Something like x*7 = x<<3 - x or x*13 = x<<4 - x<<2 + x.
Otherwise, a second Propeller Chip should be able to handle this or more complex filtering easily.
Lawson
You seem to have a very good handle on this, better than I do at this point. Can you point me to a reference for building FIR filters? At this point, with very little study, I don't think the filter needs to be anything fancy although I would like to be able to resample, not just down sample.
It sounds like there might be a two prop solution here somewhere. At the moment I have use 2 cogs to read 4 channels. The way I2S works (the ADC output protocol) there's an L/R clock that indicates which channel the data belong to. Within that L clock, there are 32-bits but only 24 are real data. In the time between the 24th bit and the 32nd, I have ~1.4uS before I have to be ready for the R sample. It's during this time that I currently do the averaging and write the sample to HUB, set some flags, etc. So maybe there's time here to do some filtering. I don't have spare cogs in this prop, but maybe in a second one....
Thanks.
First off when I say I(k) I mean I(k*T) which is the input function I(t) sampled every 'T' seconds with 'k' as the sample index. A simple moving average difference equation may be sufficient. i.e. F(k) = a*F(k-1) + (1-a)*I(k) where a = e^(-T/tau), F(K) is the filtered output, and tau is the time constant of the filter. Assuming you have an anti-aliasing filter on the input to the ADC, the averaging you're currently doing may be enough to prevent aliasing when decimating the sample stream. ( it would be worth testing this with a function generator set to frequencies > fs/2)
Lawson
Otherwise, the FPGA as a frontend may be a good solution. I would expect you could also sell the pcbs here for others to play with.
If I understand the FTDI correctly, it's serial-to-USB. The whole reason we want to include a USB port is do pull data from these units (rather than have to remove the SD card which would require some kind of door which can be difficult to make water proof). Datasets will be large (4Gbytes and greater) so download speed is important. All things considered, it looks like it will be useful to figure out how to mate an FPGA to the prop. The FPGA can (hopefully) do the DSP, the USB, and hopefully some allow me to interface with some kind of external RAM chip. Given all these requirements, I think I'll focus my energies on Verilog and on working out the DSP requirements rather than trying to find a way to shoehorn all this stuff into one (or more props).
Thanks all for you comments and suggestions.
Peter
For mode switching, I've wondered about putting an address latch external to the prop which selects which EEPROM to load from. On a blackout power up, the latch would default to EEPROM $00 but on subsequent resets, the latch could be set to light up another EEROM, say $01 from the previously running code. This might facilitate moving from acquisition to dump mode. For this application a Flip Flop might even work using the address lines on the EEPROMs.
This solution assumes there can be gaps in the acquisition process but depending on how you sync them it might only be a couple of clock cycles at the ADC.
With a little creative programming you might be able to overlap the sampling so the incoming 'gatherer' prop starts collecting data before it actually takes control of the ADC clocking.
I think I'd look into using a Viniculum chip to control an actual thumb drive in that case... Maybe would also save the cog used for FSRW...
I started there about a year ago. I couldn't get the Vincilum chip to work the way it was supposed to and when I did get it working the speeds weren't very good. Maybe things have recently....
An "SD jack" sounds in-elegant. Still, this "switched" SD card sounds like an excellent solution. I'd take an inexpensive usb SD card adapter and solder it to your board. (or I'd solder down the guts of a usb SD card adapter) I'd then use a few analog switch chips to shuffle the SD card between the Propeller and the SD adapter as needed. Seeing +5 volts on the external usb port would be a indicator that the propeller should un-mount the SD card and switch it over to the SD card usb adapter.
This chip
http://search.digikey.com/scripts/DkSearch/dksus.dll?Detail&name=ADG884BRMZ-ND
or this chip
http://search.digikey.com/scripts/DkSearch/dksus.dll?Detail&name=296-9138-1-ND
or quite a few 74xx or 4000 series chips should work well as an SD card switch.
Lawson
I was actually thinking along the same lines. I don't really need to get involved in the whole USB thing. I simply want my SD card to be connected to an SD-to-USB device once in a while. There are lots of ways to tell if it's time to connect the SD card to the card reader including your suggestion. I'm sure our EE will have suggestions, but how do I shove a card reader into each unit? Given the range and volume of SD card readers, this can't be too hard.
I found this:
http://www.maxim-ic.com/datasheet/index.mvp/id/6226/t/do
but it's not actually available. I guess I need to crack open an SD card reader and see what's actually in there....
That would solve one big part of my problem. I was just looking over CIC filters for use in large sample rate changes. There might be something there I can do to filter and decimate my data....
p
No Captain Obvious comments here. Not only is that the simplest solution, it's the cheapest, and what we are currently doing. Problem is, making the case such that the SD card is accessible and watertight, is difficult/expensive. To address this we've decided on a USB port that could be connected to through which data would be pulled off. The SD card would not normally be removed and the case would not normally be opened.
@Rayman-
Can you provide any more info on that device?
Thanks,
Peter
If not, then perhaps a pair of pushbuttons to select another mode. They can be sealed. You could have a wireless zigbee or bluetooth and transmit wirelessly.
For now, a 2 prop solution seems to be a nice solution as you have the major code already working in the prop.
If you need waterproof connectors, what if you were to use some multi-conductor round connectors like 1/8" four connector. The round profile would facilitate O ring seals on both the jack and you could fashion a screw on cap.
If it's a steel shell then you could just weld 1" pipe stub on it and put a cap over the hole for diving. The SD card is rectangular but there is no reason the opening to get to it has to be,,, right?
How deep does this thing go?
Jim-
http://search.digikey.com/scripts/DkSearch/dksus.dll?Detail&name=638-1088-ND
and another
http://search.digikey.com/scripts/DkSearch/dksus.dll?Detail&name=CYWB0224ABM-BVXI-ND
Though it's going to be hard to beat buying a <$10 usb to SD adapter and stripping out the circuit board on a cost/development-time basis.
I Read what Wiki had on CIC filters. Very nifty little filter. It looks like the full sample rate portions of a decimating CIC filter would fit in your I2C cogs while the low speed section of the CIC filter would be <200 instructions of assembly (mostly depending on the compensating FIR filter quality) and may fit in the cog that deals with the decimated samples. ( this Wikipedia reference seems particularly useful. http://www.embedded.com/columns/showArticle.jhtml?articleID=160400592 )
You never mentioned if the ADC's have an input anti-aliasing filter? If you don't have an analog anti-aliasing filter there is nothing a digital filter can do to "fix" the data after it's sampled. (though, a 159KHz signal to alias back to 1kHz would be pretty rare on a seismometer...)
Lawson