How to discard hi/low in averaging

AGCB

I have a program that monitors freezer temperatures and sounds an alarm if temp gets too hi. Occasionally there is a glitch in the reading which causes the alarm to sound falsely. A simple average doesn't work because the glitch is so high that it ups the average beyond the alarm threshold.

Is there a way to average readings but discard the out of range high ones?

Thanks
Aaron

Brian Fairchild

You need to average over a much longer time period so that any contribution from a glitch has a much smaller effect on the average value. I can't see the temperature in a freezer varying that quickly so if you took a reading every second and averaged that over one minute (60 samples) then any spike will not affect the long-term reading.

Each new sample only contributes 1/60th of the average.

AGCB

The problem that I see with that is it increases the loop time of the rest of the program. How to avoid that?

evanh

Ah, the fun of crafting code. Each cycle is a partial adjustment to the filtered reading. Each new cycle is typically also a new sample. The sample becomes one of many, in a rolling buffer, that are all summed to a single more precise value. How much this then gets scaled is whatever you want. The result is the filtered reading. It's known as a moving average, or box filter. It's the most basic of a class of filters called FIR (Finite Impulse Response). There is also a low MIPS version of the moving average that doesn't need to sum up every sample in the buffer on every cycle.

A less memory consuming way to get similar effect is to instead do what's called an IIR (Infinite Impulse Response) filter. The simplest form of this type is, each cycle, subtract a portion of the filtered reading and add the newest sample.

AGCB

Can you guide me to an example of this IIR filter in SPIN?
And, yes, crafting code is fun but I sometimes need 'outside the box', (the box being my brain) ideas.

Peter Jakacki

Or simply increment or decrement bit by bit towards the current reading so that even a huge glitch will not upset it.

AGCB

Peter
That one I do not understand!
Aaron

Peter Jakacki

If the new sampe is higher than your "average" then simply increment rather than trying to add in a portion of that value. So your average is sitting at 95 and the new value is 200 so your average becomes 96 and if the glitch is gone next time then it goes back to 95 otherwise it continues to climb towards 200 it it persists. Call it inertia.

AGCB

Got it!
Thanks Peter

evanh

Oh, I hadn't read the OP. Only the part about loop time in reply. You probably should be attempting to find out why the glitches are occurring at all. Filters are not really a deglitch system.

What Peter has said is valid although you need to also give it a threshold it has to exceed before clamping to the average. The threshold could be relative to the average or it could be an absolute threshold if the glitch is always a max'd out sample.

tomcrawford

Yeah, if you cannot eliminate the bogus readings, you should at least understand, in detail, why they are occurring. They may be an indicator of some more serious problem.

The OP did not indicate what micro is being used. If it is vanilla BS-2 (no PUT/GET storage), there may not be enough variable space to keep a fews tens of samples. In that case, Peter's increment/decrement may be the only solution.

AGCB

OK. So I'm reading 8 DS18B20s and a thermocouple on a MAX31850 with a PROP 1. I'm using 3 different instances of 'spin one wire' on 3 different pins because of long wires (70+') going in opposite directions. Sensors are read one at a time at 1/10th second intervals.

Occasionally (guessing 3 - 5 times per day) one of the readings will be bogus. No problem with any but 3 sensors in freezers which trigger an alarm when too high. It's annoying and worthless to have alarms going off falsely.

I can't see anything that could happen simultaneously in the main method that could cause interference.

Attached is a screen shot of TV display.

Thanks for the help!
Aaron

Phil Pilgrim (PhiPi)

I had an app with the same issue. I solved it by reading the sensor values into a sorted array and averaging the readings in the middle half. That effectively excluded both the low and high outliers.

A simpler form of the above is just to use the middle value from the array (the median) and forget the averaging. Also, see my thread here about computing a running median:

https://forums.parallax.com/discussion/103778/computing-a-running-median

Any time you include the outliers in any kind of average, including FIR and IIR filters, they will skew the result.

-Phil

Dave Hein

If a new value is outside of a certain range just don't update the average.

DigitalBob

If you post your spin code maybe we could see how it reads. I had a problem using the MCP3204 and the BS2 the bits where unstable and jittery causing fluctuations in my readings. So I added a pause of 275 milliseconds between shiftout and shitin. along with that I'm using an array to average my readings. Now I only get about 1 bit jitter on occasion.

localroger

Sanity check your readings. It is not possible for temperature to change very fast, so if the new reading is more than slightly different from the previous one / existing running average, ignore it instead of adding it in.

Tracy Allen

The median of three values is returned by this cryptic method that makes use of the #> and <# operators in Spin:

PUB median3(a, b, c)
return a #> b <# (a <# b #> c)

You need to keep a record of the most recent three readings. This works to reject a single glitch outlier. Fails if there are two out of the three. The output of a median filter can feed into an FIR or IIR filter.

Phil Pilgrim (PhiPi)

Tracy,

I'd forgotten about that one. I wonder how long the expression would be with five readings.

-Phil

John Abshier

From some old code on my hard disk.

PUB Median5
  Sort(0, 1)
  Sort(3, 4)
  Sort(0, 3)
  Sort(1, 4)
  Sort(1, 2)
  Sort(2, 3)
  Sort(1, 2)
  result := value[2]

PRI Sort(a, b) 
  if value[a] > value[b]
    temp := value[a]
    value[a] := value[b]
    value[b] := temp    

Data stored in a 5 element array value. Here is C code reference document.

John Abshier

AGCB

Because 2 successive readings 2 seconds apart are probably not both bogus, I did it this way. Got idea from my wife (not a programmer). Could easily add more readings.

    PantrySens:= readTemperature(H_temp)                               'pantry freezer
    PanOut:=output     'output is sign extended
    PanLast:=PanNew    'update previous reading
    PanNew:=PanOut     'save newest reading

    IF  (||PanNew  < 106) AND (||PanLast < 106)     'absolute value of ~~dsOut   106=20°F  lower value = higher temp                                   
      FreezWarn 

Thanks for all the code ideas. I'll test them and probably save in my personal OBEX.
Aaron

evanh

AGCB wrote: »

...'spin one wire' on 3 different pins because of long wires (70+') going in opposite directions. Sensors are read one at a time at 1/10th second intervals.

Occasionally (guessing 3 - 5 times per day) one of the readings will be bogus....

That'll be electrical issues corrupting the data stream. There is some risk of electrical damage over time if not protected. Shielded cables can be an answer. Data rated isolators combined with twisted pairs is another well trod method.

Aside from electrical prevention, error detection is usually also applied to data streams in case something still manages to corrupt the data. I'm guessing it won't have anything like a CRC though, so not an option. What you are attempting now with spike removal is likely about as good as you can do in software.

T Chap

Could be a compressor kicking on.

Temperature is very slow. You could do something as simple as ( a long way to write it out )

 Tempbuffer = readsensor
  If tempbuffer < tempavg + 5
     If tempbuffer > tempavg - 5
        Tempavg = tempbuffer 

I do this on a thermal sensor used for human detection. If the temp changes slightly the avg changes with it. If the temp changes a lot then it’s a person and the avg does not update. It’s not really an average though. Just the latest read. In my case there’s no need to avg out the last 10 reads and shift out the oldest. If it’s within a window it’s valid as is.

Peter Jakacki

When you need fast response it pays to implement more advanced filtering but measuring freezer temperatures never needs this. Just slow and steady working a bit like an integrator with the simplest form being to increment or decrement, even fractionally which also helps to filter out jitter.

AGCB

evanh wrote: »

I'm guessing it won't have anything like a CRC though

DS18B20 does have CRC but I've never used CRC and looking at the OBEX it looks complicated for a simple coder like me. I wouldn't know how to start.
The simple code shown in my last post seems to be working as I haven't heard alarm in 36 hours.

evanh

It's not a bad solution either. Good work.

Tracy Allen

@ Aaron,
Glad you got it going. Clever wife!!

@ Phil,

Tracy,
I'd forgotten about that one. I wonder how long the expression would be with five readings.

The median of three values is returned by this cryptic method that makes use of the #> and <# operators in Spin:

PUB median3(a, b, c)
return a #> b <# (a <# b #> c)
You need to keep a record of the most recent three readings. 
This works to reject a single glitch outlier. 
Fails if there are two out of the three. 
The output of a median filter can feed into an FIR or IIR filter.

-Phil

Yeah, 5 is an ugly bear of a puzzle.

Nevertheless, simply iterating the method on multiple circular buffers of three can reject longer strings of glitch values. One median of three rejects a single glitch. Feed the output of that to a second median of three, and it can reject a pair of successive outliers at the input. And so on. A single median of 5 is better though than the chain of two sets of three, even though the median of 5 too can reject only 2 outliers. It gives overall lower standard deviation given mostly normal random input values.

JonnyMac

DS18B20 does have CRC but I've never used CRC and looking at the OBEX it looks complicated for a simple coder like me. I wouldn't know how to start.

My 1-wire object has CRC methods in it; one in Spin by Micah Dowty, the other in PASM by Cam Thompson (yes, I will borrow good code whenever I can). I'm using the DS18B20 in a work project so this has been on my desk lately.

Mickster

Dave Hein wrote: »

If a new value is outside of a certain range just don't update the average.

My thought exactly....any particular reason why this was apparently not considered?

Phil Pilgrim (PhiPi)

Mickster wrote:

Dave Hein wrote:

If a new value is outside of a certain range just don't update the average.

My thought exactly....any particular reason why this was apparently not considered?

The tricky part, of course, is defining "a certain range." But this is what the median operation takes care of handily. The "range" is defined dynamically by the middle values in a sorted list of the last N readings. So, yes, this technique was considered and discussed.

-Phil

localroger

Something very simple and similar to what Peter was suggesting would be like this:

tempbuffer := readsensor
if tempbuffer > avg
  avg++
elseif tempbuffer < avg
  avg--

You would want avg to be fractions of a degree; if you take 1 reading per second 1/10 or even 1/100 of a degree might be appropriate, since freezer temperature can't physically change that fast. This drags the average toward the sample instead of adding the sample in to anything, and sharply limits the damage that even a train of transients can do.

Phil Pilgrim (PhiPi)

I tried and plotted several smoothing methods on some fake sample data that includes different rates of change, a couple wild readings, and a step function. The results are plotted here:



The formulae are as follows:

Moving Average: y_i = y_i-1 * 0.9 + x_i * 0.1
Average of Three: y_i = (x_i-2 + x_i-1 + x_i) / 3
Median of Three: y_i = median(x_i-2, x_i-1, x_i)
Increment/Decrement: y_i += 0.1 * sign(x_i - y_i-1)

The moving average shows a definite phase lag and is affected by the wild outliers. The average-of-three shows almost no phase lag but is even more affected by outliers. The median shows no phase lag and is unaffected by single outliers. The increment/decrement is immune to outliers and shows no phase lag until the data's rate of change is larger than the adjustments it's making, at which point it can't keep up.

-Phil