Smoothing Sensor Readings
SailerMan
Posts: 337
I'm reading four Distance Sensors and I would like the best and most efficient way to smooth the readings that are coming from the sensors. I am using Pulsin and my readings are between 0 and 255. But sometimes the readings fluctuate wildly and I need to make sure that can't happen.
Does anyone have a good algorthm?
Regards,
Eric
Post Edited (SailerMan) : 12/18/2007 4:09:30 PM GMT
Does anyone have a good algorthm?
Regards,
Eric
Post Edited (SailerMan) : 12/18/2007 4:09:30 PM GMT
Comments
I don't have time right now to dig it out, but there was a lengthly and detailed discussion about this a month or so ago. I think it was in the Sandbox forum (Might have been Basic Stamp), and was discussing accuracy of distance (I think Ping))) sensors. They were also getting swings in readings and apparently unrepeatable distance measurements. I think they basically ended up averaging a number of readings.
▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔
John R.
Click here to see my Nomad Build Log
At my work we do this:
Take 7 readings. Remove the highest and lowest, then average the remaining 5.
Bean.
▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔
Cheap used 4-digit LED display with driver IC·www.hc4led.com
Low power SD Data Logger www.sddatalogger.com
SX-Video Display Modules www.sxvm.com
Coming soon! Propeller based OSD module www.hittconsulting.com
·
Note that you need at least two sensor readings to have a usable value when using this strategy.
John R. I thought I did a proper search I will look more.
Bean...I was thinking along those lines, but couldn't think of a good way to do that.
JonnyMac... Unique Idea I will try that and see if it works for my problem.
Post Edited (SailerMan) : 12/18/2007 4:10:41 PM GMT
http://www.emesystems.com/BS2math5.htm
This method allows you to dynamically set the number of samples without using the variable space to hold each sample.
New = {Sensor reading} DataBuffer = DataBuffer - Avg + New Avg = DataBuffer / SamplesIf 'Avg' is a BYTE variable, and 'DataBuffer' defined as a WORD, you can effectively hold an Average consisting of a maximum of 255 samples. If 'Avg' is a NIB, then 4096 samples can be achieved.
The variable called 'Samples' can be a constant therefore not impacting the available variable space.
▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔
Beau Schwabe
IC Layout Engineer
Parallax, Inc.
Nice link... Thanks,
Eric
That is short and to the point, I wrote a small program in "Blitz Basic" to test it and it looks like it's going to be a perfect little piece of code.
Thanks,
Eric
I think you have a viable solution(s) already, but here's the thread I was thinking of:
http://forums.parallax.com/showthread.php?p=623170
▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔
John R.
Click here to see my Nomad Build Log
The imprecision comes from truncation. Let's say you have 15 samples at 8 bits, and 14 of them are value 101 and 1 of them is value 100. The average is 100.93, but if you are carrying out the average with 8 bit integer math, you will still be seeing a filter output of 100. As i said, the bias is toward low values, so as an input decreases and increases, there is a hysteresis effect as the average jumps between values, more easily down than up.
To lessen that effect, multiply the input by some factor (say 16 or 256) and take the average of that. The extra bits can slosh around and in effect interpolate to the right of the radix point, so you can get an interpolated reading as an output, or at least get a better round-off.
This doesn't matter if you simply what to reduce wild fluctuations and the least significant bit is not that significant anyway. But it does matter if you are using the average to do any kind of interpolation of noisy data.
Signals that have pop noise or occasional outliers are greatly improved by throwing away the min and max before the average, as Bean suggested, or by making a first stage filter output the median value to a second stage filter that takes the average.
If there is a strong frequency component in the noise, say 60 hertz, the most effective filter will take several samples carefully timed to cover one period of the interference. That is hard to do with something like a distance sensor, but look for possible effects of 60hz pickup.
▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔
Tracy Allen
www.emesystems.com
just tried this code from Beau
New = {Sensor reading}
DataBuffer = DataBuffer - Avg + New
Avg = DataBuffer / Samples
and don't know really how to implement that into the software. Does the samples variable increase with every measurement ? So first measurement means Samples=1, second measurement means Samples=2 and so on?
What is the variable I am actually giving out as a result after certain amount of measurement, is it DataPuffer?
Thanks guys! [noparse]:)[/noparse]
Matthias
The variable Samples is fixed to the number of samples that you want to average.
The value on this depends on how you define DataBuffer and what the highest value you get from New
For instance, if you can only define DataBuffer as a WORD (the highest possible value being 65535), and the highest value you get from New is 50, then the maximum number of Samples that can be averaged is 1310. [noparse][[/noparse]65535 / 50 = 1310.7
Likewise, if you define DataBuffer as a WORD, and the highest value you get from New is 255, then the maximum number of Samples that can be averaged is 257. [noparse][[/noparse]65535 / 255 = 257
Your Input variable is New, while your Output variable is Avg
▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔
Beau Schwabe
IC Layout Engineer
Parallax, Inc.
Post Edited (Beau Schwabe (Parallax)) : 9/3/2009 4:14:51 PM GMT
thank you very much. I now understand the idea of the code. However I got one issue left: how much samples do I have to do to get an reasonable result? I just quickly checked in excel and there, with a constant sensor input reading of 200, I need at least 25 samples to get something around 200 back.
I am measuring a moving object (ball rolling down a 2 feet long beam). It's not fast movement, but will the SX be able to calculate this 25 samples quick enough to be able to still provide a 'live' measurement?
In theory, the algorithm can handle 218 samples, so I can handle the 25 required ones. (with DataPuffer as a WORD and a maximum sensor reading of 300, 65535/300 = 218). But sometimes the sensor skips occasionally to higher values (like 600 or even 3031, the maximum value). So should I built in a restriction first that the maximum value the sensor gives out is 300?
Would I build a for...next statement around the code and make 25 loops for 25 samples?
Thanks Beau. Lot's of questions for the genius code [noparse]:)[/noparse]
Matthias
Post Edited (Matthias09) : 9/3/2009 5:47:17 PM GMT
Your 25 samples assumes that the DataBuffer and Avg start out as Zero.
On your very first sample do this...
Avg = New
DataBuffer = Avg * Samples
... and then apply this to any samples that follow...
DataBuffer = DataBuffer - Avg + New
Avg = DataBuffer / Samples
"So should I built in a restriction first that the maximum value the sensor gives out is 300?" - Yes, otherwise clipping will occur.
▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔
Beau Schwabe
IC Layout Engineer
Parallax, Inc.
Post Edited (Beau Schwabe (Parallax)) : 9/3/2009 6:25:45 PM GMT
Thank you!
I am curious: what is the advantage of this code over building just an average value out of the 5 values (except that I safe variable space)?
Matthias
▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔
Beau Schwabe
IC Layout Engineer
Parallax, Inc.
Here is an example of something I'm working on right now written in SPIN using a derivative of this algorithm.
The code eventually plays out to displaying 6 digits to the right of the decimal and is intended to capture the fractional difference and determine a bias on the LSB (least significant bit).
VAR long RawValue,DataBase, Average, Samples PUB MainProgram ADC.Start '' Initialize ADC Term.start(31, 30, 0, 38400) '' Initialize serial communication to the PC SerialDisplay PUB SerialDisplay Samples := 100 Average := ADC.Raw Average := ADC.Raw 'Read value twice the first time DataBase := Average * Samples repeat Samples Calibration Samples *= 100 DataBase *= 100 repeat Samples Calibration Samples *= 100 DataBase *= 100 repeat Calibration PUB Calibration RawValue := ADC.Raw ' Read RAW 10-bit Adc value DataBase := DataBase - Average + RawValue Average := DataBase / Samples DisplayData PUB DisplayData Term.tx(1) ' Position cursor at the Home position Term.dec(RawValue) Term.tx(13) Term.dec(DataBase)▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔
Beau Schwabe
IC Layout Engineer
Parallax, Inc.
thank you very much for the extensive information.
I understand the function of the code, however, have problems with implementing it into SX Basic. Main point: The output data varies a lot, independent from the amount of samples I take (tried 1 to 25). Actually I have more variance in the output data when using the smoothing algorithm than without. My Databuffer has a lot of fluctuation. I have the assumption, that the value read in the first loop is vastly different from the other, created ones. (I measured this) Do you have any idea?
Here is what I've build:
FUNC RUN_ULTRASOUND US_Black PIN RC.6 'black labeled Ultrasonic sensor ball_position VAR Word 'ball position 2 DataBuffer var word avg var word samples var byte samples = 10 FOR i = 1 TO samples '-----Reading----- US_Green = 0 'set port to low PULSOUT US_Green, 5 PAUSEUS 100 PULSIN US_Green, 1, ball_position '-----Smoothing----- 'smooth in raw data, adjust only the final result. save time with that. IF i = 1 THEN 'at first reading, initialize the variables Avg = ball_position 'Avg holds the averaged values, while ball_position holds the recent reading DataBuffer = Avg * Samples 'Databuffer is a non-physical value, that is required for calculation ELSE 'start to smoothen with 2nd reading onwards DataBuffer = DataBuffer - Avg DataBuffer = DataBuffer + ball_position Avg = DataBuffer / Samples ENDIF NEXT '-----Adjusting----- 'convert the final result in mm. Distance = Avg * 5 Distance = Distance ** RawToMm '-----Transmitting----- RETURN Distance '-----Resetting----- Distance = 0 ENDFUNCThanks!
Matthias
Post Edited (Matthias09) : 9/13/2009 11:51:28 PM GMT
Since you are only taking a certain number of samples to begin with before displaying the data, then it doesn't really make sense to use the smoothing algorythm when you can just divide your total number of samples by the sum of your data.
The smoothing algorythm is designed for streaming continuous dataflow.
PULSIN is sensitive to 2us ... I wonder what the hardware end looks like? can you post a schematic?
▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔
Beau Schwabe
IC Layout Engineer
Parallax, Inc.
I will try the idea from Bean then (take a couple of readings, remove highest and lowest and average the rest). However, do you have any idea how I quickly can determine the highest and lowest value in e.g. 5 samples, all stored in an array?
Thank you!!
Matthias
How are you using the two PING sensors? You can only use one at a time or else they will produce false signals or noise.
Also, looking at the documentation for the PING, I wonder what the reflective acoustic properties of the ball are, and how big the ball is....
The PING))) sensor cannot accurately measure the distance to an object that: a) is more than 3 meters
away, b) that has its reflective surface at a shallow angle so that sound will not be reflected back towards
the sensor, or c) is too small to reflect enough sound back to the sensor. In addition, if your PING)))
sensor is mounted low on your device, you may detect sound reflecting off of the floor.
...So in an earlier post you said that your getting a value of 300 from PULSIN ... if I did that correct, that's about 8.14 inches?... does that correspond with your setup? ... and do your numbers report valid values if you manually move the ball? .. or do the numbers jump around?
My experience with the PING is that it's very stable, It seems as if your getting a huge amount of variation, then your PING might be seeing sonic reflections. Can you post any pictures of your setup and maybe describe how you are using two PING sensors?
▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔
Beau Schwabe
IC Layout Engineer
Parallax, Inc.
I just let them run one after the other and give 2000ms time in between. So:
Ping1 PulseOut
Pause 100ms
Ping1 PulsIn
Pause 2000ms
Ping2 PulseOut
Pause 100ms
Ping2 PulseIn
So they won't interfere. I have pretty smooth read outs of both sensors, only when I start to smoothen with the algorithm, then I get a lot of jumps.
I get a maximal value of 300mm, the reading is already adjusted. 300mm are about 11.8in. The value is correct and works great with my setup.
Find a picture enclosed of the BnB System. Other than in the picture, I adjusted the sensors so they now look up in the sky with approx. 25deg. I did these as now they have a much bigger range (maybe due to less reflections from the beam itself). I don't actually know why they have a bigger range (also in single mode, means I only use one sensor) when I don't target them directly towards the object, but a little bit above, but it works fine.
THis is my code. I ccmmented out the smoothing process as it doesn't currently work.
FUNC DRIVE_US_SENSOR 'Driver. Has to be called for each sensor. Gives back the position of the ball RELATIVE to the sensor. Provides Data smoothing: 'make [noparse][[/noparse]samples] readings, built average, then give out the result. When called for one sensor, [noparse][[/noparse]samples] values are taken for 'this one before switching to next one occurs. temp_gs var bit temp_gs = __param1 FOR i = 1 TO 5 'amount of readings before giving out result. '-----Reading----- IF temp_gs = Green THEN led3 = ison led4 = isoff US_Green = 0 'set port to low PULSOUT US_Green, 5 '[noparse][[/noparse]PIN][noparse][[/noparse]Pulselength]. set port high for width of trigger variable (5). creates the pulse that I will 'later receive. after the pulse, port gets back to initial status (low). PAUSEUS 100 'Pause 100 microseconds PULSIN US_Green, 1, ball_position(i) '[noparse][[/noparse]PIN][noparse][[/noparse]State][noparse][[/noparse]Variable to store result]. measure the width of an incoming pulse. 1 sets the transition of 'the measured pulse (Flankenerkennung und Triggerung: port war auf low gesetzt. nun 'warte ich auf eine Veraenderung auf HIGH durch einfallenden Puls, den ich mit PULSOUT gesendet habe. 'PULSIN triggert also auf steigende Flanke. Der einfallende Puls wird als highpulse bezeichnet, dessen 'Laenge gemessen).The length of the measured pulse is stored in the word ball_position ELSE led4 = ison led3 = isoff US_Black = 0 PULSOUT US_Black, 5 PAUSEUS 100 PULSIN US_Black, 1, ball_position(i) ENDIF PAUSEUS 700 'required for short distance measurements, so the sensor doesn't get signal from the previous cycle '-----Smoothing----- IF ball_position(i) > 150 THEN 'maximal reading is around 150 (raw). Cut above, so sensor input data for smoothing is more reliable ball_position(i) = 150 ELSEIF ball_position(i) = 0 THEN 'value stored in byte variable gets 0, when value > 256 (I define this as out of range). Give sensor special code (252, I chose that) ball_position(i) = 252 'so that I can read from the sensor output, that the sensor is out of range (assuming that he never will ENDIF 'give back 252 as a averaged value) DON'T KNOW YET IF I NEED THIS FEATURE NEXT b1 = ball_position(1) 'store saved sample data b2 = ball_position(2) b3 = ball_position(3) b4 = ball_position(4) b5 = ball_position(5) {cut highest and lowest} <-- yet to be implemented {average the rest} {save the value in the variable Avg} '-----Adjusting----- 'convert the final result in mm. see documentation for details Avg = Avg * 5 Avg = Avg ** RawToMm PAUSEUS 2000 'to prevend that one sensor reads the other, give some time before function returns 'and can be called again. '-----Transmitting----- RETURN Avg '-----Resetting----- Avg = 0 ENDFUNCMatthias
Post Edited (Matthias09) : 9/23/2009 10:47:49 PM GMT
I'm still looking forward to a sleek way to smoothen the sensor data of the ONE Ping Sensor I'm using (two made too much noise, thanks Beau!).
So what I want to do:
1) take 5 measurements and save them into an byte array (max value is about 100)
2) cut lowest and highest value
3) average the remaining 3 results
4) give out the averaged value to my controller function
5) next 5 measurements
Step 1 is easy, obviously. But how can I do step 2)? And is there a sleek way for step 3)?
Guys, hope you can help me with that! Without smoothing, the whole system (see picture in above's post) get's really crazy!
Thank you!
Matthias
#2 and #3 combined
' {$STAMP BS2} ' {$PBASIC 2.5} N VAR Nib DataIn VAR Word MaxValue VAR Word MinValue VAR Word DataTotal VAR Word Average VAR Word MaxValue = 0 MinValue = 65535 DataTotal = 0 FOR N = 1 TO 5 IF DataIn > MaxValue THEN MaxValue = DataIn ENDIF IF DataIn < MinValue THEN MinValue = DataIn ENDIF DataTotal = DataTotal + DataIn NEXT Average = (DataTotal - MaxValue - MinValue) / 3▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔▔
Beau Schwabe
IC Layout Engineer
Parallax, Inc.
thanks again! I'll use the piece of code, it's simple yet powerful!
Now, during going on with programming last night I found out something: The reason, why my Ping sensors is jumping so much in his output data is not the sensor, it's the my motordriver function. So far my program looks like this:
DO
1) read the ball position with the Ping
2) determine angle to move with controller function
3) move beam to angle with stepper motor
LOOP
I played a little bit around and turned 3) off, and voila: the sensor reading becomes very smooth instantly (when ball is in the middle of the beam, the sensor reads now constantly 54, while before he read 136, 137, 54, 136, 145, 55, so also jumped between values around 55 (which is the actual position of the ball) and much higher values of 137). So I went on on searching and figured out that not the motor is the problem, but the pause I have after every step I move (PAUSE 18). deactivating the pause makes the stepper not move anymore (as I switch to fast between the steps), but prevents the jumping of the values of the Ping.
So now my question: how can the Pause command in step 3 impact the Ping sensor in step 1? I mean, the ping sensor doesn't start working until step 3 is done (with working I mean sending, waiting and receiving the wave) and step 3 is done only after the Pause command is done, which results in that step 1 is basically just delayed by step 3. But somehow the pause command delays also the Ping sensors, means the time between sending and receiving the data (otherwise I cannot imagine how the sensor can read 136 when actually the ball is at 55).
Any idea?
How can I solve that?
Should I basically quit this software architecture and go to an interrupt based one, like Zoot suggested here: http://forums.parallax.com/showthread.php?p=849214
Thanks!!
Matthias
Post Edited (Matthias09) : 10/24/2009 11:54:34 PM GMT
So my question are [noparse]:)[/noparse]
1) is there any way how I can read the values the ping is giving back without impacting them at the same time? (what I do with the break command in combination with the watch command).
2) how can commands outside of the Ping routine impact it's results at all? I mean, everything is running in a consecutive order, so PAUSE or BREAK shouldn't have any impact in the Ping routine when called before or afterwards.
3) Is the Ping sensor giving different values back when running in debug mode with 4Mhz compared to run in normal Mode with 50 Mhz (as I use the PAUSEUS statement within the routine to set the delay between sending and receiving the wave).
4) How much time should I give the Ping between PULSOUT and PULSEIN? So far I have 100us (PAUSEUS 100).
Thanks!
Matthias
Problem: the 450us waiting period suggested between Pulseout and Pulsein in this thread under Capt. Quirks post http://forums.parallax.com/showthread.php?p=647889.
I reduced it from my 100us to 5(!)us and voila. No problems with any pause or break statements! Also now same values in debug mode and real mode (tested it with a led that turns on on a certain distance).
However, I'm still curious how this long waiting time can impact the result? Does the ping just catch old bursts?
Matthias
Long range
Long new
Long Databuffer
Long Avg
New := range
Databuffer := Databuffer - Avg + New
Avg := Databuffer / 16