How to discard hi/low in averaging
AGCB
Posts: 327
I have a program that monitors freezer temperatures and sounds an alarm if temp gets too hi. Occasionally there is a glitch in the reading which causes the alarm to sound falsely. A simple average doesn't work because the glitch is so high that it ups the average beyond the alarm threshold.
Is there a way to average readings but discard the out of range high ones?
Thanks
Aaron
Is there a way to average readings but discard the out of range high ones?
Thanks
Aaron
Comments
Each new sample only contributes 1/60th of the average.
A less memory consuming way to get similar effect is to instead do what's called an IIR (Infinite Impulse Response) filter. The simplest form of this type is, each cycle, subtract a portion of the filtered reading and add the newest sample.
And, yes, crafting code is fun but I sometimes need 'outside the box', (the box being my brain) ideas.
That one I do not understand!
Aaron
Thanks Peter
What Peter has said is valid although you need to also give it a threshold it has to exceed before clamping to the average. The threshold could be relative to the average or it could be an absolute threshold if the glitch is always a max'd out sample.
The OP did not indicate what micro is being used. If it is vanilla BS-2 (no PUT/GET storage), there may not be enough variable space to keep a fews tens of samples. In that case, Peter's increment/decrement may be the only solution.
Occasionally (guessing 3 - 5 times per day) one of the readings will be bogus. No problem with any but 3 sensors in freezers which trigger an alarm when too high. It's annoying and worthless to have alarms going off falsely.
I can't see anything that could happen simultaneously in the main method that could cause interference.
Attached is a screen shot of TV display.
Thanks for the help!
Aaron
A simpler form of the above is just to use the middle value from the array (the median) and forget the averaging. Also, see my thread here about computing a running median:
https://forums.parallax.com/discussion/103778/computing-a-running-median
Any time you include the outliers in any kind of average, including FIR and IIR filters, they will skew the result.
-Phil
PUB median3(a, b, c)
return a #> b <# (a <# b #> c)
You need to keep a record of the most recent three readings. This works to reject a single glitch outlier. Fails if there are two out of the three. The output of a median filter can feed into an FIR or IIR filter.
I'd forgotten about that one. I wonder how long the expression would be with five readings.
-Phil
PUB Median5 Sort(0, 1) Sort(3, 4) Sort(0, 3) Sort(1, 4) Sort(1, 2) Sort(2, 3) Sort(1, 2) result := value[2] PRI Sort(a, b) if value[a] > value[b] temp := value[a] value[a] := value[b] value[b] := tempData stored in a 5 element array value. Here is C code reference document.
John Abshier
PantrySens:= readTemperature(H_temp) 'pantry freezer PanOut:=output 'output is sign extended PanLast:=PanNew 'update previous reading PanNew:=PanOut 'save newest reading IF (||PanNew < 106) AND (||PanLast < 106) 'absolute value of ~~dsOut 106=20°F lower value = higher temp FreezWarnThanks for all the code ideas. I'll test them and probably save in my personal OBEX.
Aaron
Aside from electrical prevention, error detection is usually also applied to data streams in case something still manages to corrupt the data. I'm guessing it won't have anything like a CRC though, so not an option. What you are attempting now with spike removal is likely about as good as you can do in software.
Temperature is very slow. You could do something as simple as ( a long way to write it out )
Tempbuffer = readsensor If tempbuffer < tempavg + 5 If tempbuffer > tempavg - 5 Tempavg = tempbufferI do this on a thermal sensor used for human detection. If the temp changes slightly the avg changes with it. If the temp changes a lot then it’s a person and the avg does not update. It’s not really an average though. Just the latest read. In my case there’s no need to avg out the last 10 reads and shift out the oldest. If it’s within a window it’s valid as is.
The simple code shown in my last post seems to be working as I haven't heard alarm in 36 hours.
Glad you got it going. Clever wife!!
@ Phil,
Yeah, 5 is an ugly bear of a puzzle.
Nevertheless, simply iterating the method on multiple circular buffers of three can reject longer strings of glitch values. One median of three rejects a single glitch. Feed the output of that to a second median of three, and it can reject a pair of successive outliers at the input. And so on. A single median of 5 is better though than the chain of two sets of three, even though the median of 5 too can reject only 2 outliers. It gives overall lower standard deviation given mostly normal random input values.
My thought exactly....any particular reason why this was apparently not considered?
The tricky part, of course, is defining "a certain range." But this is what the median operation takes care of handily. The "range" is defined dynamically by the middle values in a sorted list of the last N readings. So, yes, this technique was considered and discussed.
-Phil
You would want avg to be fractions of a degree; if you take 1 reading per second 1/10 or even 1/100 of a degree might be appropriate, since freezer temperature can't physically change that fast. This drags the average toward the sample instead of adding the sample in to anything, and sharply limits the damage that even a train of transients can do.
The formulae are as follows:
Moving Average: yi = yi-1 * 0.9 + xi * 0.1
Average of Three: yi = (xi-2 + xi-1 + xi) / 3
Median of Three: yi = median(xi-2, xi-1, xi)
Increment/Decrement: yi += 0.1 * sign(xi - yi-1)
The moving average shows a definite phase lag and is affected by the wild outliers. The average-of-three shows almost no phase lag but is even more affected by outliers. The median shows no phase lag and is unaffected by single outliers. The increment/decrement is immune to outliers and shows no phase lag until the data's rate of change is larger than the adjustments it's making, at which point it can't keep up.
-Phil