digital low pass filter

ManAtWork

Until now, I have used simple moving average filtering to suppress noise from sensor signals because it is easy to implement. Although it is good for signal processing because it has no overshot and constant phase delay its transfer function is not ideal. It has large "humps" in the stop band.

I think a well designed FIR or IIR filter can do much better and give better stop band attenuation for the same delay. Unfortunatelly, the propeller is not the best choice for general FIR or IIR filters because multiplications are expensive, at least for PASM implementations required for high speed.

Does anybody know a good "trick" for improved low pass filtering? Cascade multiple moving average filters? Clever adjustment of the coefficiens of an IIR filter so that only a few shifts/adds would do?

I have a PID control loop with a noisy sensor input signal. A larger averaging window size reduces the noise but also adds delay which is not good for the stability of the PID loop. I have to find the best compromise of low noise vs. delay. I have no influence on the analogue side of the sensor circuit. The sensor is "digital" i.e. has a built-in ADC. I receive the values via serial communication with 20kHz sample rate and 12 bits resolution.

rbehm

For noisy data I use two simple filters, that are easy to implement, are fast and do not demand high cpu load:

1. Peak suppression: Collect n samples like your moving average, throw away the largest and smallest and compute the average for the remaining values. This eliminates any peaks (in both directions) which otherwise can only be suppressed by a long averaging filter.

2. "Trend filter": This is good for slow changing noisy signals, e.g temperature. Compare the newest value with the last. If it is larger for at least N samples take it (or the an average of this and the old one). The same for the newest being lower than the last for N samples. If the direction of change was not stable for N samples, just keep the old value.
If you have noisy data, this will keep the noise out because changes continuously go in both directions. If there is a trend in one direction, this will follow this trend quickly.

1 and 2 can be combined.

ManAtWork

Basically good ideas but they have a drawback: non-linear and even non-constant phase delay and a "latch-effect".

If there is a strong "trend" (change per sample higher than noise) the newest sample will always be the lowest or highest, so filter 1. is completely ineffective. Same for 2. the filter doesn't suppress anything if the change is higher than the noise component.

This becomes even worse when the noise is not "white" but has dominating frequencies (resonances, 50/60Hz and harmonic EMI...). Then, the slope of the signal becomes lower/higher periodically so that the filter response flips forth and back (latch effect). This makes the noise actually worse for the PID control loop because a smooth waveform is transformed into a stair shape with steep slopes and more energy in the high order harmonics.

I think the peak and/or trend filter should be combined with a standard (bessel type or similar) low pass filter. The latch effect could possibly be avoided if the filter is applied to the differentiation of the signal and the output is integrated to restore the original signal.

Rayman

See bottom of this page:
http://www.pulsedpower.net/Info/RC/RC_Filter.htm#Numerical_Low-Pass_RC_Filter

If you are getting a stream of data, just pick an alpha factor like .25 and then:
Current value = .75*old value +.25* new sample

ManAtWork

Ok, this is the simplest possible IIR filter with only one tap. If alpha is the an inverse power of two the multiplication can be done with a shift. But is it better than a moving average? I guess for a noticable improvement we need a higher order filter.

ManAtWork

Another issue with the trend filter: It does sort of a "rectification". A step forward is passed through, a step backward is suppressed. This can result in positive gain in a closed loop and can cause oscillation.

I think anything that does a "hard decision" of dropping a sample or not is "evil" and should be avoided. Each sample is composed of a true signal part and a noise part. If we completely drop a sample we also throw away the signal part producing unwanted artefacts in the signal shape.

A dynamic limiter would be much better. If we have a robot and a distance sensor, for example, we know that the robot or the sensed object can't jump but has inertia and therefore limited velocity and acceleration. If there is a sudden jump in the sensed distance we can limit the velocity and acceleration to realistic values. This way we cancel out unrealistic peaks but still keep the trend without adding too much delay.

MJB

you can use a FIR or IIR filter design program -
there are some available online
design a filter you like.
then you can approximate the constant filter coefficients by power of 2 factors
which allow shift/add implementation to a level you like.
Being a compromise between exact representation of the filter coefficients and the effort to implement it.
You can even check the resulting approximation by using the
approximated coefficients in the filter simulation.
Of course this is them 'hard-coded'. No easy change of filter coefficients.

I read a very good example long ago using an ATtiny as a digital filter processor
for a current regulator in a ~8 A power supply.
- can't find it a.t.m. sorry

here it is - found it on my disk: .... hmmm ... how do I make an attachment?

here the link:
ww1.microchip.com/downloads/en/DeviceDoc/current_supply.pdf

just had a look again ... it is not really a FIR / IIR application here but maybe give you a hint

ManAtWork

Hmm, the limiter idea doesn't work well. The acceleration is in saturation most of the time leading to large errors and overshot. I've made a comparison. There is no remarkable difference between the moving average and the RC filter approach. For example, a averaging window size of 8 samples and an alpha factor of 0.25 give nearly the same delay and noise suppression.

geo_leeman

Not necessarily inexpensive, but I really like the Savitzky-Golay filter for things that have steps I want to preserve, but smooth.

Phil Pilgrim (PhiPi)

Try my FIR filter generator:

https://forums.parallax.com/discussion/133173/fir2pasm-automatic-fir-filter-code-generator/p1

-Phil

MIchael_Michalski

"virtually parallel". Hmmmmm


"My wife thinks I'm cheating on her with a woman at work, but I told her, there's virtually nothing between us... so she has virtually nothing worry about"

My boss caught the two of us on the security camera. I explained it wasn't what it looked like. We were virtually fully clothed.

I applied for unemployment, but the unemployment department turned me down. They said losing my job was my own fault. I'm going to appeal because I was virtually blameless.

My wifes divorce lawyer got my car, my house and even my dog. Now I'm living in a cardboard box and eating out of the trash. I'm virtually homeless and virtually penniless.

The guy in the box next to mine said he knew a guy who would pay me a bunch of money to bring something back from Mexico for him. It was virtually risk free.

I was accused of smuggling 50kg of cocain across the Mexican border in the gas tank of my old pickup truck, but I'm virtually innocent"

I was convicted and sentenced to 30 years in a federal prison, but I'll be out in virtually no time at all.





The word "virtually" can make quite a difference...

Phil Pilgrim (PhiPi)

And your post was in virtually the correct thread.

-Phil

K2

Once upon a time I implemented a mixer and a series of FIR filters in a widely used baseband modem. The compute platform was nothing more than the SX chip. Def no hardware multiplier there! The mixing involved table look-ups. The FIR filters were nothing more than a whirling dervish of additions punctuated by the occasional ÷2 (a.k.a. shift-right).

Tracy Allen

About the graph you posted, LowpassComp.png. Is the x axis in units of samples? Not a lot of oversampling to work with, no?, in relation to the rate of change. How about a filter where the IIR or FIR filter constant is adaptive as a measure of confidence that changes or lack thereof are real. An elementary Kalman filter if you will that seeks to hug tighter to the data stream.

msrobots

Phil Pilgrim (PhiPi) wrote: »

And your post was in virtually the correct thread.

-Phil

Could be moved, but being in the wrong thread made me smile also.

And basically Michael_Michalski made a very valid statement here...

Enjoy

Mike

SaucySoliton

ManAtWork wrote: »

Does anybody know a good "trick" for improved low pass filtering? Cascade multiple moving average filters?

That will approach a Gaussian filter, which might be exactly what you want. :cool:

Long moving average filters can be efficiently computed by a difference equation. If you wanted to average 100 samples, add the new sample to an accumulator and subtract the sample you added 100 steps ago. You could start with 4 filters of length 4. I don't know if it's worth it to do the difference equation for filters that short.

Here's a quick Octave script to simulate a filter like this:

% Simulation of Gaussian filter from cascaded moving average filters  

plotwidth=20
width=4
impulse_response=[ones(width,1) ; zeros(plotwidth-width,1)]/width;

for n=2:4
  convout = conv(impulse_response(:,n-1),impulse_response(:,1),'full');
  impulse_response(:,n)=convout(1:plotwidth);
end

step_response = cumsum(impulse_response);

%plot( impulse_response )
plot(    step_response )

Mark_T

Python's scipy.signal library has a comprehensive set of functions for designing digital filters and plotting
their responses, if you ever played with Python's matplotlib and numpy you'll realise this makes it pretty
easy to experiment with filter design.