Regularization by Convolution

The estimate 8 can be re-written as a convolution of the kernel with the true density function;

$$f(s) \star K(s) \equiv \int_{-\infty}^{+\infty} K(s - x) f(x) dx$$ (10)

$$= E_{f(x)} \left( K(s - x) \right)$$

$$\approxeq \frac{1}{n} \sum_{i=1}^n K(s - x_i)$$

$$= \hat{f} (s)$$

and so in a sense, the kernel density estimate is approximately a deconvolution from the true density; in [DH73] we see that in the limit as the number of random samples approaches infinity, $\hat{f}(s)$ converges to $f(s) \star K(s)$. We can also write the density estimate as a smoothing (we refer to smooth and smoothing in two contexts; smooth functions have continuous derivatives and smoothing operations remove localized features) convolution of the impulsive density function (which assigns a probability mass of $\frac{1}{n}$ to each of the $n$ random samples; $\hat{f}^i(x) = 1/n$) with the kernel function:

$$\hat{f}^i(s) \star K(s) = \sum_{i = 1}^{n} \hat{f}^i(x_i) \; K( s - x_i)$$ (11)

$$= \sum_{i = 1}^{n} \frac{1}{n} \; K( s - x_i)$$

$$= \hat{f} (s)$$

Essentially the convolution is a regularization of the estimate through the addition of `smoothing' noise in the regions where the impulsive density is undefined.

It is interesting to note that the Parzen Method has been shown [ZPR05] to be equivalent to a Regularized Least Squares Method (or Tikhonov Regularization) where the regularizing functional is taken to be the norm of the resulting estimated density. One benefit [MZ00] of `regularizing by convolving' is that there are no explicit regularization parameters that need to be estimated and then re-trained.