How to discard hi/low in averaging

I have a program that monitors freezer temperatures and sounds an alarm if temp gets too hi. Occasionally there is a glitch in the reading which causes the alarm to sound falsely. A simple average doesn't work because the glitch is so high that it ups the average beyond the alarm threshold.

Is there a way to average readings but discard the out of range high ones?

Thanks
Aaron
«1

Comments

  • Brian FairchildBrian Fairchild Posts: 505
    edited 2019-04-05 - 10:02:23
    You need to average over a much longer time period so that any contribution from a glitch has a much smaller effect on the average value. I can't see the temperature in a freezer varying that quickly so if you took a reading every second and averaged that over one minute (60 samples) then any spike will not affect the long-term reading.

    Each new sample only contributes 1/60th of the average.
  • The problem that I see with that is it increases the loop time of the rest of the program. How to avoid that?
  • evanhevanh Posts: 6,882
    edited 2019-04-05 - 11:07:29
    Ah, the fun of crafting code. :) Each cycle is a partial adjustment to the filtered reading. Each new cycle is typically also a new sample. The sample becomes one of many, in a rolling buffer, that are all summed to a single more precise value. How much this then gets scaled is whatever you want. The result is the filtered reading. It's known as a moving average, or box filter. It's the most basic of a class of filters called FIR (Finite Impulse Response). There is also a low MIPS version of the moving average that doesn't need to sum up every sample in the buffer on every cycle.

    A less memory consuming way to get similar effect is to instead do what's called an IIR (Infinite Impulse Response) filter. The simplest form of this type is, each cycle, subtract a portion of the filtered reading and add the newest sample.
    "There's no huge amount of massive material
    hidden in the rings that we can't see,
    the rings are almost pure ice."
  • Can you guide me to an example of this IIR filter in SPIN?
    And, yes, crafting code is fun but I sometimes need 'outside the box', (the box being my brain) ideas.
  • Or simply increment or decrement bit by bit towards the current reading so that even a huge glitch will not upset it.

    Tachyon Forth - compact, fast, forthwright and interactive
    useforthlogo-s.png
    --->CLICK THE LOGO for more links<---
    P2 +++++ TAQOZ INTRO & LINKS +++++ P2 SHORTFORM DATASHEET
    P1 +++++ Latest binary V5.4 includes EASYFILE +++++ Tachyon Forth News Blog
    Brisbane, Australia
  • Peter
    That one I do not understand!
    Aaron
  • Peter JakackiPeter Jakacki Posts: 8,216
    edited 2019-04-05 - 11:44:42
    If the new sampe is higher than your "average" then simply increment rather than trying to add in a portion of that value. So your average is sitting at 95 and the new value is 200 so your average becomes 96 and if the glitch is gone next time then it goes back to 95 otherwise it continues to climb towards 200 it it persists. Call it inertia.

    Tachyon Forth - compact, fast, forthwright and interactive
    useforthlogo-s.png
    --->CLICK THE LOGO for more links<---
    P2 +++++ TAQOZ INTRO & LINKS +++++ P2 SHORTFORM DATASHEET
    P1 +++++ Latest binary V5.4 includes EASYFILE +++++ Tachyon Forth News Blog
    Brisbane, Australia
  • Got it!
    Thanks Peter
  • evanhevanh Posts: 6,882
    edited 2019-04-05 - 12:07:10
    Oh, I hadn't read the OP. Only the part about loop time in reply. You probably should be attempting to find out why the glitches are occurring at all. Filters are not really a deglitch system.

    What Peter has said is valid although you need to also give it a threshold it has to exceed before clamping to the average. The threshold could be relative to the average or it could be an absolute threshold if the glitch is always a max'd out sample.
    "There's no huge amount of massive material
    hidden in the rings that we can't see,
    the rings are almost pure ice."
  • Yeah, if you cannot eliminate the bogus readings, you should at least understand, in detail, why they are occurring. They may be an indicator of some more serious problem.

    The OP did not indicate what micro is being used. If it is vanilla BS-2 (no PUT/GET storage), there may not be enough variable space to keep a fews tens of samples. In that case, Peter's increment/decrement may be the only solution.
    Re-inventing the wheel is not a waste of time if, when you are done, you understand why it is round.
  • OK. So I'm reading 8 DS18B20s and a thermocouple on a MAX31850 with a PROP 1. I'm using 3 different instances of 'spin one wire' on 3 different pins because of long wires (70+') going in opposite directions. Sensors are read one at a time at 1/10th second intervals.

    Occasionally (guessing 3 - 5 times per day) one of the readings will be bogus. No problem with any but 3 sensors in freezers which trigger an alarm when too high. It's annoying and worthless to have alarms going off falsely.

    I can't see anything that could happen simultaneously in the main method that could cause interference.

    Attached is a screen shot of TV display.

    Thanks for the help!
    Aaron
    3918 x 2326 - 1M
  • Phil Pilgrim (PhiPi)Phil Pilgrim (PhiPi) Posts: 22,263
    edited 2019-04-05 - 17:05:06
    I had an app with the same issue. I solved it by reading the sensor values into a sorted array and averaging the readings in the middle half. That effectively excluded both the low and high outliers.

    A simpler form of the above is just to use the middle value from the array (the median) and forget the averaging. Also, see my thread here about computing a running median:

    https://forums.parallax.com/discussion/103778/computing-a-running-median

    Any time you include the outliers in any kind of average, including FIR and IIR filters, they will skew the result.

    -Phil
    “Perfection is achieved not when there is nothing more to add, but when there is nothing left to take away. -Antoine de Saint-Exupery
  • If a new value is outside of a certain range just don't update the average.
  • If you post your spin code maybe we could see how it reads. I had a problem using the MCP3204 and the BS2 the bits where unstable and jittery causing fluctuations in my readings. So I added a pause of 275 milliseconds between shiftout and shitin. along with that I'm using an array to average my readings. Now I only get about 1 bit jitter on occasion.
  • Sanity check your readings. It is not possible for temperature to change very fast, so if the new reading is more than slightly different from the previous one / existing running average, ignore it instead of adding it in.
  • Tracy AllenTracy Allen Posts: 6,317
    edited 2019-04-05 - 19:01:12
    The median of three values is returned by this cryptic method that makes use of the #> and <# operators in Spin:

    PUB median3(a, b, c)
    return a #> b <# (a <# b #> c)

    You need to keep a record of the most recent three readings. This works to reject a single glitch outlier. Fails if there are two out of the three. The output of a median filter can feed into an FIR or IIR filter.
  • Tracy,

    I'd forgotten about that one. I wonder how long the expression would be with five readings. :)

    -Phil
    “Perfection is achieved not when there is nothing more to add, but when there is nothing left to take away. -Antoine de Saint-Exupery
  • From some old code on my hard disk.
    PUB Median5
      Sort(0, 1)
      Sort(3, 4)
      Sort(0, 3)
      Sort(1, 4)
      Sort(1, 2)
      Sort(2, 3)
      Sort(1, 2)
      result := value[2]
    
    PRI Sort(a, b) 
      if value[a] > value[b]
        temp := value[a]
        value[a] := value[b]
        value[b] := temp    
    

    Data stored in a 5 element array value. Here is C code reference document.

    John Abshier
  • Because 2 successive readings 2 seconds apart are probably not both bogus, I did it this way. Got idea from my wife (not a programmer). Could easily add more readings.
        PantrySens:= readTemperature(H_temp)                               'pantry freezer
        PanOut:=output     'output is sign extended
        PanLast:=PanNew    'update previous reading
        PanNew:=PanOut     'save newest reading
    
        IF  (||PanNew  < 106) AND (||PanLast < 106)     'absolute value of ~~dsOut   106=20°F  lower value = higher temp                                   
          FreezWarn 
    

    Thanks for all the code ideas. I'll test them and probably save in my personal OBEX.
    Aaron
  • AGCB wrote: »
    ...'spin one wire' on 3 different pins because of long wires (70+') going in opposite directions. Sensors are read one at a time at 1/10th second intervals.

    Occasionally (guessing 3 - 5 times per day) one of the readings will be bogus....
    That'll be electrical issues corrupting the data stream. There is some risk of electrical damage over time if not protected. Shielded cables can be an answer. Data rated isolators combined with twisted pairs is another well trod method.

    Aside from electrical prevention, error detection is usually also applied to data streams in case something still manages to corrupt the data. I'm guessing it won't have anything like a CRC though, so not an option. What you are attempting now with spike removal is likely about as good as you can do in software.
    "There's no huge amount of massive material
    hidden in the rings that we can't see,
    the rings are almost pure ice."
  • T ChapT Chap Posts: 3,947
    edited 2019-04-06 - 02:58:51
    Could be a compressor kicking on.

    Temperature is very slow. You could do something as simple as ( a long way to write it out )
     Tempbuffer = readsensor
      If tempbuffer < tempavg + 5
         If tempbuffer > tempavg - 5
            Tempavg = tempbuffer 
    
    

    I do this on a thermal sensor used for human detection. If the temp changes slightly the avg changes with it. If the temp changes a lot then it’s a person and the avg does not update. It’s not really an average though. Just the latest read. In my case there’s no need to avg out the last 10 reads and shift out the oldest. If it’s within a window it’s valid as is.
  • When you need fast response it pays to implement more advanced filtering but measuring freezer temperatures never needs this. Just slow and steady working a bit like an integrator with the simplest form being to increment or decrement, even fractionally which also helps to filter out jitter.

    Tachyon Forth - compact, fast, forthwright and interactive
    useforthlogo-s.png
    --->CLICK THE LOGO for more links<---
    P2 +++++ TAQOZ INTRO & LINKS +++++ P2 SHORTFORM DATASHEET
    P1 +++++ Latest binary V5.4 includes EASYFILE +++++ Tachyon Forth News Blog
    Brisbane, Australia
  • evanh wrote: »
    I'm guessing it won't have anything like a CRC though
    DS18B20 does have CRC but I've never used CRC and looking at the OBEX it looks complicated for a simple coder like me. I wouldn't know how to start.
    The simple code shown in my last post seems to be working as I haven't heard alarm in 36 hours.
  • It's not a bad solution either. Good work.
    "There's no huge amount of massive material
    hidden in the rings that we can't see,
    the rings are almost pure ice."
  • Tracy AllenTracy Allen Posts: 6,317
    edited 2019-04-06 - 21:27:03
    @ Aaron,
    Glad you got it going. Clever wife!!

    @ Phil,
    Tracy,
    I'd forgotten about that one. I wonder how long the expression would be with five readings. :)
    The median of three values is returned by this cryptic method that makes use of the #> and <# operators in Spin:
    PUB median3(a, b, c)
    return a #> b <# (a <# b #> c)
    You need to keep a record of the most recent three readings. 
    This works to reject a single glitch outlier. 
    Fails if there are two out of the three. 
    The output of a median filter can feed into an FIR or IIR filter.
    
    -Phil

    Yeah, 5 is an ugly bear of a puzzle.

    Nevertheless, simply iterating the method on multiple circular buffers of three can reject longer strings of glitch values. One median of three rejects a single glitch. Feed the output of that to a second median of three, and it can reject a pair of successive outliers at the input. And so on. A single median of 5 is better though than the chain of two sets of three, even though the median of 5 too can reject only 2 outliers. It gives overall lower standard deviation given mostly normal random input values.
  • JonnyMacJonnyMac Posts: 6,183
    edited 2019-04-08 - 22:15:53
    DS18B20 does have CRC but I've never used CRC and looking at the OBEX it looks complicated for a simple coder like me. I wouldn't know how to start.
    My 1-wire object has CRC methods in it; one in Spin by Micah Dowty, the other in PASM by Cam Thompson (yes, I will borrow good code whenever I can). I'm using the DS18B20 in a work project so this has been on my desk lately.
    Jon McPhalen
    Hollywood, CA
    It's Jon or JonnyMac -- please do not call me Jonny.
  • Dave Hein wrote: »
    If a new value is outside of a certain range just don't update the average.

    My thought exactly....any particular reason why this was apparently not considered?
    Failure is not an option...it's bundled with the software.
  • Mickster wrote:
    Dave Hein wrote:
    If a new value is outside of a certain range just don't update the average.

    My thought exactly....any particular reason why this was apparently not considered?

    The tricky part, of course, is defining "a certain range." But this is what the median operation takes care of handily. The "range" is defined dynamically by the middle values in a sorted list of the last N readings. So, yes, this technique was considered and discussed.

    -Phil
    “Perfection is achieved not when there is nothing more to add, but when there is nothing left to take away. -Antoine de Saint-Exupery
  • Something very simple and similar to what Peter was suggesting would be like this:
    tempbuffer := readsensor
    if tempbuffer > avg
      avg++
    elseif tempbuffer < avg
      avg--
    

    You would want avg to be fractions of a degree; if you take 1 reading per second 1/10 or even 1/100 of a degree might be appropriate, since freezer temperature can't physically change that fast. This drags the average toward the sample instead of adding the sample in to anything, and sharply limits the damage that even a train of transients can do.
  • Phil Pilgrim (PhiPi)Phil Pilgrim (PhiPi) Posts: 22,263
    edited 2019-04-09 - 22:01:13
    I tried and plotted several smoothing methods on some fake sample data that includes different rates of change, a couple wild readings, and a step function. The results are plotted here:

    various_smoothing_methods.png

    The formulae are as follows:

    Moving Average: yi = yi-1 * 0.9 + xi * 0.1
    Average of Three: yi = (xi-2 + xi-1 + xi) / 3
    Median of Three: yi = median(xi-2, xi-1, xi)
    Increment/Decrement: yi += 0.1 * sign(xi - yi-1)

    The moving average shows a definite phase lag and is affected by the wild outliers. The average-of-three shows almost no phase lag but is even more affected by outliers. The median shows no phase lag and is unaffected by single outliers. The increment/decrement is immune to outliers and shows no phase lag until the data's rate of change is larger than the adjustments it's making, at which point it can't keep up.

    -Phil
    929 x 647 - 26K
    “Perfection is achieved not when there is nothing more to add, but when there is nothing left to take away. -Antoine de Saint-Exupery
Sign In or Register to comment.