Imagemagick Resampling

Introduction

Some of you may be familiar with the blog post I made about mpv's scaling filters a while ago, but it was never really meant to be shared as much as it did. That page was originally the result of an assignment that I had during my undergrad, when I was formally studying digital image processing for the first time. Naturally, it was full of mistakes and the results weren't particularly scientific. I got a lot of feedback and the page ended up evolving in an organic way, but it still has some fundamental issues that can not be fixed without a major change in the methodology.

From the top of my head I can enumerate the following problems:

1. The resampling tests were only done for 1 or 2 test images.
2. I kept adding meme metrics since people requested them.
3. I also kept adding meme shaders since people requested them.
4. I created a bunch of different test cases but didn't really have the motivation to keep all of them up to date.
5. I originally used Matlab to compute the metrics and Excel to plot the tables, which means updating the page was a major pain in the ass.
6. I used catmull-rom to downscale the test images.

So, to address these issues:

1. I chose to use the entire Manga109 dataset this time, which is probably the best widely-used dataset we have for line-art.
2. I chose to stick to the standard distortion metrics.
3. I chose to exclude all meme shaders. I'll stick to resampling filters this time.
4. I chose to only test 2x upsampling and 0.5x downsampling.
5. I chose to fully automate everything so updating the numbers should be trivial.
6. I chose to use the box filter to downsample this time. Since I'm downsampling to 0.5x this should be a simple average of 4 pixels, which is the best case scenario and doesn't introduce any blurriness or ringing.

With that out of the way, we can proceed with the real introduction.

Resampling

Resampling is the process of changing the number of samples of a discrete signal to obtain a new discrete representation of the underlying continuous signal. This definition comes from the idea of having a sensor of some kind producing an analog continuous-time voltage/current variance which is then periodically sampled and quantised into predefined amplitude levels so we can store it in bits/bytes.

The easiest and most classic way of resampling to a higher sample rate is via linear interpolation, if you want to find a value between two points you can simply draw a line between them. Linear interpolation can be done in a cartesian plane, through both axis, creating what we call "bilinear" interpolation. Bilinear interpolation is the simplest interpolation algorithm, the easiest to compute and probably the most widespread one.

But can something as simple as just drawing a line between 2 points give us good results? Sometimes it does, sometimes it doesn't. It really depends on the signal. Instead of taking 2 points and drawing a line, we could take more than 2 points and draw a higher-degree curve. The shape of the curve depends on the weights used in the calculation, and these weights depend on the chosen filter. The number of input samples in the calculation depends on the length/radius/support of the filter. If you want to understand how this is actually done, I suggest simply reading this explanation.

In short, the most common way of resampling images is treating each row/column as an independent 1-D signal and simply going over all of them until you have resampled the entire image. This means you have to choose a dimension to resample first but this is pretty much inconsequential to the end result. The resampling algorithm itself is pretty simple, for each output sample you simply centralise the filter on top of it and see which input samples end up inside of the window after you compute their equivalent positions, then you multiply these inputs by their corresponding weights depending on their distance to the centre.

There's a different method, usually called polar/cylindrical/elliptical resampling, that does the operation in the 2-D domain directly. The only difference here is that all samples that fit inside the 2-D filter will now be weighted simultaenously, which may drastically change how some filters behave since we're only calling the filter once per output pixel rather than twice as before.

In this page I'll include results for both orthogonal and polar resampling. It's important to note that polar resampling implementations are generally slower and most filters weren't really designed to be used in this way (the most "famous" exception being polar lanczos, since we replace the Sinc function with its 2-D "equivalent", the Jinc/sombrero function).

Upsampling Methodology

As stated before the entire Manga109 dataset will be used in this comparison. This dataset has manga covers that look like this:

The dataset is downsampled with:

The dataset is then brought back up with the following command for orthogonal resampling:

For polar resampling, -resize 200% is replaced by -distort resize 200%.

Likewise, for the results in sigmoid light, the resampling arguments are wrapped by -colorspace lineargray +sigmoidal-contrast 7.5 and -sigmoidal-contrast 7.5 -colorspace gray.

The {resampling_filter} argument is replaced by all available resampling filters: ['Bartlett', 'Blackman', 'Bohman', 'Box', 'Catrom', 'Cosine', 'Cubic', 'Gaussian', 'Hamming', 'Hann', 'Hermite', 'Jinc', 'Kaiser', 'Lagrange', 'Lanczos', 'Lanczos2', 'Lanczos2Sharp', 'LanczosRadius', 'LanczosSharp', 'Mitchell', 'Parzen', 'Point', 'Quadratic', 'Robidoux', 'RobidouxSharp', 'Sinc', 'SincFast', 'Spline', 'CubicSpline', 'Triangle', 'Welch']

For a better explanation of these filters, please check this page from Imagemagick.

The result is then evaluated with MAE, PSNR, SSIM and MS-SSIM. All calculations are done after grayscale conversion and in double-precision floating-point for accuracy. Metrics are normalised between [0, 1] and then averaged together.

Please note that some of those filters are just aliases to other filters or aliases to Sinc/Jinc with different windows.

LanczosRadius is a Jinc-windowed Jinc squished to have its third zero crossing at 3 instead of 3.2383154841662362. Likewise, LanczosSharp and Lanczos2Sharp are also just squished Jinc-windowed Jincs.

Upsampling Results

Gamma Light Upsampling

Sigmoid Light Upsampling

Upsampling Commentary

Unsurprisingly, the Lanczos family is at the top. Lanczos is widely considered to be one of the best resampling filters, so this makes sense.

There are a few things worth talking about. The first one is that orthogonal LanczosSharp ended up on top of the table, a filter preset that is not meant to be used as an orthogonal filter. If you want to give it a try on mpv, you can use --scale=lanczos --scale-blur=0.9812505644269356. Keep in mind, however, that this blur value is arbitrary nonsense in this context. It is meant to be used with Polar_Lanczos.

The second is that there are some Lanczos variants on top of the normal Lanczos: Hamming, Cosine and Welch. For the last two all you gotta do is set --scale=lanczos --scale-window=window, but for the first you also have to set --scale-radius=4. These are just Sinc filters with different windows, so naturally they all look very similar.

The third is that Polar_Catrom is the best filter with only 2 lobes. For the orthogonal contenders, Lanczos2 beats Catrom, and Lanczos2Sharp beats them both. To use Polar_Catrom on mpv, you can use --scale=ewa_robidoux --scale-param1=0.0 --scale-param2=0.5. For Lanczos2 you can simply do --scale=lanczos --scale-radius=2, and for Lanczos2Sharp add --scale-blur=0.9549963639785485.

You might have also noticed that Polar_LanczosRadius is on top of Polar_LanczosSharp. Polar_LanczosRadius is sharpened to have its third zero crossing exactly at 3, which corresponds to a blur factor of 0.9264075766146068 (3.0/3.2383154841662362). To try it on mpv you can use --scale=ewa_lanczos --scale-blur=0.9264075766146068.

As expected, upsampling in sigmoid light helps and makes all metrics go up. This is admittedly a mathematical gimmick, but it's hard to argue against the numbers.

As usual, the filters are ranked based on full-reference distortion metrics that may not always correlate with the human perception of quality, and your personal preference is entirely subjective. Please also keep in mind that only the named filters were scored, but you can make "custom" filters that may score higher using the expert controls.

Downsampling Methodology

The problem with downsampling evaluation is that we do not have a "ground truth" image, making it impossible for us to use the standard full-reference quality metrics. However, in the previous section I already made the concession that, at a 0.5x scaling ratio, the box filter is as good as it gets without producing any artifacts.

If we make the leap of faith that resampling filters will more or less keep their character regardless of scaling factor, we can use the output of box as the reference and compare other filters against it. So, in short, we want to find a filter that, at any scaling factor, will behave similarly to box at 0.5x. I'm calling this a leap of faith because this hypothesis most likely falls apart with extreme scaling factors, but as long as you keep it close to 0.5x it probably makes sense.

With that said, now we can proceed to the actual methodology. The reference is created with:

The dataset is then brought downsampled with the following command for orthogonal resampling:

For polar downsampling, -resize 50% is replaced by -distort resize 50%.

Likewise, for the results in linear light, the resampling arguments are wrapped by -colorspace lineargray and -colorspace gray.

The {resampling_filter} argument is replaced with all available resampling filters. Everything else about the process stays the same, so we can proceed to the results.

Downsampling Results

Gamma Light Downsampling

Linear Light Downsampling

Downsampling Commentary

The results were relatively predictable.

Catrom is known to be good at downsampling, and since Lanczos2 is just a slightly sharper Catrom, it ends up as the best orthogonal filter that isn't a meme.

For polar filters, Polar_RobidouxSharp is our top scorer in gamma light but Polar_Hermite takes the lead in linear light. I'd personally favour Polar_Hermite between the two, as it's faster and doesn't ring.

Both Lanczos (orthogonal Sinc-Sinc) and Polar_Lanczos (polar Jinc-Jinc) ended up with mediocre scores. That's a good thing because it shows the methodology can actually punish filters that are too ringy.

I think this test solidifies what was already common knowledge. BC-Splines are very good at downsampling, and pretty much all of them are near the top.

Outro

Nicolas Robidoux has a personal page with some recommendations, if you want a more qualitative approach.