Platform feature: Low pass filter
The IoT and analytics platform hetida platform offers a wide range of standard workflows for efficient data pre-processing. One of these is the low-pass filter – a proven method for smoothing out high-frequency fluctuations in time series while preserving the low-frequency trend in the data. In this article, we show how the filter works and how it can be specifically applied to water level data in water management.
Initial situation
The Ruhrverband measures the water level of the Ruhr in Hattingen every 15 minutes. Such data is interesting both for flood early warning systems and for determining the frequency of low water due to drought. However, short-term fluctuations occur in the 15-minute raw data, for example due to passing ships, blocking branches or measurement errors. Such fluctuations can affect the further hydrological use of the data. One method of smoothing out high-frequency fluctuations while retaining the low-frequency tendency of the data is a low-pass filter. The hetida platform – fuseki’s data analysis tool with connected dashboarding system – is well suited for this: it provides standard workflows for data pre-processing, including high-pass and low-pass filters. In the following, we use the hetida platform to smooth the level data of the Ruhr with a low-pass filter and thus prepare it for further analysis.
To do this, we first illustrate the basic functionality of a low-pass filter using artificially generated data. We then load the level data from the Ruhr in Hattingen into the hetida platform and demonstrate the hetida platform’s low-pass filter workflow on real data. We visualize the result in a dashboard of the platform.
How the low pass filter works
We imagine the measurement of a signal (this can be an audio signal, but also any other type of time series) that is composed of two other signals: a low-frequency main signal and a high-frequency interference signal:
We are only interested in the low-frequency main signal. However, as we can only measure the composite signal, we have to filter out the high-frequency interference signal. This is precisely the purpose of a low pass filter: it filters out high-frequency signals so that only the low frequencies can pass through the filter (hence the name “low pass filter”). The hetida platform has a standard workflow that implements such a low pass filter with a Butterworth filter.
Butterworth filter
A Butterworth filter is a filter whose frequency response is defined by the formula on the left. The argument ω is the incoming frequency as a radian speed. The cut-off frequency is denoted by ωc, i.e. the frequency for which signals with a lower frequency ω < ωc are to be retained and signals with a higher frequency ω > ωc are to be filtered out. The parameter n denotes the order of the filter; the higher the order, the steeper the transition from the passband to the stopband in the frequency response. As a result of this formula, low incoming frequencies ω < ωc are not greatly altered by the filter (|H(ω)| ≈ 1), while high frequencies ω > ωc are filtered out (|H(ω)| ≈ 0).
In the hetida platform, we configure the “cut-off frequency” and “order” parameters of the Butterworth filter.We also specify the time unit to which the frequency refers – for example, periods per second, minute, hour or day.We also specify whether the filter should work on one or both sides:When applied on one side, the filter only takes into account current and previous values for each filtered value.With the two-sided application, on the other hand, both earlier and later values are included in the calculation.
The low-frequency main signal in our artificial example has a frequency of 0.1 Hz, the high-frequency interference signal has a frequency of 1 Hz. The cut-off frequency is therefore selected in between, in this case 0.25 Hz. We retain the default settings for the time unit (frequency_as_periods_per_unit = “s” for second), the order of the filter (order = 1) and the two-sided application (“forward_backward” = True). The result looks like this:
In reality, there is never such an ideal signal composed of two perfectly regular signals. We therefore try out the same low-pass filter on a composite signal that also contains random noise. The result is similarly satisfactory:
Application of a low pass filter to level data in the hetida platform
We imagine that a hydrologist wants to investigate the frequency and duration of low water events in the Ruhr region. She defines a low water event as a water level falling below a certain level for a longer period of time. If a measured water level rises for a short time due to a single rainfall, this is often of no great significance. Other random factors such as a passing ship or a blocked channel can also cause short-term fluctuations. If the water level briefly exceeds the limit value as a result, this does not change the ongoing low water situation.
The hydrologist therefore wants to smooth her measurement data before analyzing low water events. To do this, she uses a low-pass filter to eliminate such short-term fluctuations. We implement this in the hetida platform below.
The water level data of the Ruhr in Hattingen, which is measured by the Ruhrverband at 15-minute intervals, is made freely available as a .csv file. After downloading, the .csv file can be easily imported into the hetida platform (more information on the platform interfaces).
Import of raw data via CSV file
Once the data has been successfully imported, a channel containing this data can be created. The hetida platform generates a data preview for each channel created at the corresponding point in the Explorer. Here we can already see that there was a longer period of low water levels in the Ruhr in Hattingen in 2003. From March to November, the water levels were below the average low water level at this measuring point, which is 102 cm. However, there were short-term exceedances of this limit during this period. If low water was defined using such a limit value, the period would not be fully classified as a low water period, which is undesirable.
We therefore filter the raw data through a low pass filter. We set the cut-off frequency so that events with a duration of less than two weeks are filtered out. We regard these short events as interference signals. Longer-term events, on the other hand, should be retained.
To do this, we select “d” for days as the unit of frequency. We set the cut-off frequency to 1/14, i.e. 0.0714. The result of this filter is written to a new channel whose data preview looks like this:
Result after applying the low pass filter
The result of this filter is written to a new channel whose data preview looks as follows:
And indeed, we can see that the water level adjusted by the low-pass filter remains below the 102 cm mark throughout the entire period from April to December. Finally, we create a dashboard to visually check the plausibility of the filter result. The result looks like this:
We have seen how we can apply a low pass filter to the level data of the Ruhr with just a few clicks using the standard workflows of the hetida platform. Now we can use the data for further analysis, either in the platform or after exporting data with another tool.
Request a free demo now!
Experience the IoT and analytics platform live and make an appointment.
Our experts will get in touch with you shortly.
We look forward to hearing from you.
