mpv Resampling

Introduction

This page is meant to be treated as a follow-up to this scaler comparison done with ImageMagick. I'm only going to talk about niche topics here, so just refer to that other page if you only want to read about scalers.

The Effect of the Downsampler in Upsampling Evaluation

Downsampling in linear light is widely accepted as the "usually correct" approach, it can better preserve bright highlights which can make a huge difference depending on what you're downsampling. That doesn't mean that linear light downsampling will absolutely always look better, if you're downsampling black text on a white background you're probably better off doing it in gamma light to prevent the background from dilating into the text for example, since that would hurt legibility.

This usually isn't a problem when using the built-in filters because mpv doesn't upsample in either linear or gamma light by default, it does it in sigmoid light instead to reduce ringing artifacts. However, that's usually not true for shaders, specially not when they're trained with artificial data (the LR image is created by downsampling the HR reference). If the test procedure includes downsampling in linear light, it's obvious that any model trained with images downsampled in linear light is going to have an unfair advantage.

In short, downsampling in linear light dilates bright structures while eroding dark ones (relative to downsampling in gamma light). This is generally accepted as correct but it might not always be ideal depending on the content.

The problem arises when we have to decide how to downsample training data before feeding it to a model. If we feed images that have been downsampled in linear light, the model will produce images with dilated dark highlights and eroded bright features. The exact opposite of what happens when you downsample in linear light. The model is simply learning how to "undo" what linear light downsampling "does", so this behaviour makes perfect sense.

Interestingly though, most models seem to show better reconstruction quality when you feed them LR images that have been downsampled in gamma light, and pretty much all mpv shaders have been trained with data downsampled in gamma light (including NNEDI3, FSRCNNX, RAVU, etc).

For these reasons, the upsampling tests will be done with an image downsampled in gamma light.

Upsampling Shaders

I still do not have a good way of automating mpv tests and therefore I'll have to stick to a single test image, which is going to be aoko.png:

The small number of samples under test makes this very unscientific, but it is what it is.

Upsampling Methodology

The following shaders were "benchmarked":

1. ArtCNN
2. RAVU
3. FSRCNNX

The test image is downsampled with:

It is then converted to grayscale with:

The original image is also converted to grayscale using the same command to create the reference:

We need to convert them to grayscale because some of these shaders do not have RGB support.

The images were then upsampled back with:

Upsampling Results

Upsampling Commentary

As we can see in the table above, ArtCNN is seems to be the best option when it comes to luma doubling. I personally think the C4F16 version of it should be good enough for most video content, but if your hardware can handle the bigger models and you don't mind the heat/noise then why not.

For lower scaling factors between 1x and 2x you can expect this list to remain mostly the same as long as you use a sharp downsampling filter for the luma doublers. The choice of filter actually has a huge impact on how sharp the final image will be, so sticking to the hermite default makes all doublers much softer than ravu-zoom for example. Also keep in mind that the difference between the shaders also becomes smaller as you decrease the scaling factor.

Chroma Shaders

The following shaders were "benchmarked":

1. ArtCNN
2. Chroma From Luma Prediction
3. JointBilateral
4. KrigBilateral

Chroma Methodology

A "near lossless" 420 version of aoko.png was created:

avifenc aoko.png --min 0 --max 0 -y 420 420.avif

mpv options remains the same with the exception that we don't need --window-scale=2.0 anymore, and since we're comparing chroma here, the images weren't converted to grayscale.

Chroma Results

Chroma Commentary

If we look at the numbers, ArtCNN is easily the best option. I actually wouldn't recommend using your resources on this unless you're playing native resolution video and your GPU has nothing better to do.

The other shaders all suffer from chromaloc issues and they can be easily fooled when there's no correlation between luma and chroma. I'd personally skip all of them.

Antiring

Antiringing solutions is a topic that I hadn't covered in the previous iteration of this page, but now that we have more than a single option we can also compare them.

In short, antiringing filters attempt to remove overshoots generated by sharp resampling filters when they meet a sharp intensity delta. What is commonly referred to as ringing is simply consequential to the filter's impulse response.

The following image shows this very well:

The negative weights in the filter are there for it to be able to quickly respond to high-frequency transitions, but it makes the filter overshoot a little bit before reaching its final destination. The "intensity" of the ringing is directly related to the magnitude of the secondary lobes. The second lobe, which is almost always negative, is responsible for the overshooting in can see in this example, but filters with more lobes ring once per lobe, and the ringing can be "positive" as well (within the range set by the original pixels) with positive lobes. The "length" of the rings is directly related to the length of the lobes, which is why filters like polar lanczos have "longer" rings (the zero crossings don't fall exactly at the integers, but rather slightly after them).

Aoko.png isn't a good test image to see ringing problems so I'm switching to violet.png here. If you've never seen it before, this is what violet.png looks like:

Antiring Methodology

The methodology here almost is equal to the one used for upsampling, with the only difference being that we have to include --scale-antiring to use mpv's (or libplacebo's) native AR solution. We also don't have to convert to grayscale this time since the shaders support RGB images just fine.

AR is only really necessary when you're using sharp filters, it makes no sense alongside blurry filters because blurry filters don't ring hard enough for it to be noticeable. There are a few sharp memes that are worth trying with AR though if you feel adventurous, but enerally speaking I think polar lanczossharp are pretty well balanced (a bit blurry even).

I'm including my AR shader in this comparison because I think it's better at keeping everything but the overshoots intact, which does create some weird artifacts with sharp transitions sometimes, specially if you use it at ludicrous scaling factors, but the output is generally good enough on real content.

Antiring Results

Antiring Commentary

Previously, this section talked about the different strengths and weaknesses of Pixel Clipper and libplacebo's AR, but that's not really neccessary anymore. Libplacebo's AR has been recently updated and it looks great now. It's actually shocking how similar the solutions look at 2x considering how different the mechanisms are.

The only caveat is that libplacebo's AR is still a bit blurrier at similar strengths, so keep that in mind.

As you can see in the table above, the sweetspot for libplacebo's AR seems to be around 0.6 taking all the metrics into account. I could end the commentary here but if you pay attention you can also see that 0.6 isn't at the top of any of the individual metrics, but it does score above average on all of them. This is actually pretty interesting as these metrics tell us different things:

If we choose to focus on MAE, the sweetspot seems to be around 0.85. MAE is very good at telling us what is numerically closer to the reference without biasing this towards any specific type of artifacts regardless of how humans perceive them.

If you only want to know how bad the worst pixels are you can focus on PSNR instead, and in this case Pixel Clipper seems to be much better than libplacebo's AR (presumably because the latter is smoother). The sweetspot seems to be around 0.8 here for PC, and 0.5 for libplacebo's AR.

SSIM is good at telling us about sharpness and the whole structural profile of the picture, it doesn't care too much about brightness or contrast variations but it'll heavily punish blurriness or sharp transitions that shouldn't exist. We can see that Libplacebo's AR is better than PC here (again because it's smoother), with 0.75 strength at the top.

MS-SSIM is just SSIM at different scales to emulate a viewer looking at the picture from different distances. This metric tends to correlate better to the human perception of quality than the other 3, and if you're not very close to the display this is probably a better metric. For MS-SSIM, the picture with no AR at all is the top scorer. FIR filters need to overshoot a little bit to reach some intensity levels and in this case (since polar lanczossharp isn't that sharp to begin with), the ringing doesn't seem to be a big enough problem for it to hurt the score. This isn't true for all filters and it might not be true for all scaling factors either, but for polar lanczossharp at 2x you could perhaps make an argument that AR might be unnecessary.

To conclude, my opinion is that you can probably skip AR unless you're using a very sharp filter. If you want to use AR though, the ideal strength will depend on the choice of filter, scaling factor and content type, but something around 0.6 should be pretty safe (if you want to use PC for the higher PSNR you should probably use it at higher strengths like 0.8).

Downsampling Antiring

Pixel Clipper actually has a downsampling variant as well, which I believe is probably less broken than mpv's native --dscale-antiring solution for orthogonal filters. Libplacebo's new polar AR has been recently enabled for downsampling as well, so we can also test that here.

Downsampling Antiring Methodology

My current method to evaluate downsampling filters is to concede that at 0.5x linear light box is as good as it gets, since in this case we're just averaging 4 pixels together. We can't use box itself for other scaling factors though, so the aim is to find a filter that produces a similar result while also being usable at any other arbitrary scaling factor.

The same violet.png test image was used to test downsampling AR, and it was downsampled with ImageMagick as follows:

The images were basically generated using the following commands:

Please note that you have to set --scale-antiring=1.0 instead of --dscale-antiring=1.0. That's probably a bug but it is what it is.

Downsampling Antiring Results

Downsampling Antiring Commentary

The first thing we can see is that AR seems to be a net-positive in general when using ringy filters. Only Mitchell and Hermite got worse with AR.

It's also good to see that libplacebo's AR seems to work even better when downsampling, as it can remove ringing without introducing any significant blur. This consolidates that there's little reason to use Pixel Clipper over Libplacebo's native AR if you're using polar filters.

Personally, I'd say you probably need some kind of AR for anything sharper than catrom, but catrom itself is still perfectly usable without AR.

Performance Benchmarks

Benchmarking mpv performance is a bit tricky due to a few things:

1. Due to the relatively cheap load, it's difficult to predict at which power state the GPU will stay at while we benchmark. To virtually increase the load, in an attempt to level the ground, we can cheese mpv into outputting frames as quickly as possible, but that's not necessarily representative of a real-world scenario.
2. Hardware, OS, drivers, display servers, etc... All of those things affect mpv performance in complex ways that are usually unpredictable, so even if my benchmarks were perfect they'd still only be relevant for those with similar systems.

In any case, the following numbers were produced with these mpv settings with a 720p anime episode (~24 minutes long video) on a 5600X+6600XT+Windows11 setup:

Measure-Command { mpv --no-config --vo=gpu-next --gpu-api=vulkan --audio=no --untimed=yes --video-sync=display-desync --vulkan-swap-mode=immediate --window-scale=2.0 --fullscreen=yes }

Please keep in mind that the choice of built-in filter doesn't matter that much as the weights get stored in a LUT, so something like lanczos and spline36 have virtually identical performance. In short, filters with larger radii will obviously be slower, polar filters are slower than their orthogonal counterparts and AR obviously also makes things slower.

For the shaders, the number are more or less what I expected, but it's nice to see that in the grand scheme of things these shaders don't really slow the playback down that much. Good old polar lanczossharp with AR is already as slow as most shaders.

Still though, the conclusion here seems to be that if you're running mpv on a semi-decent computer, these shaders will never be a problem.

Outro

I want to make it clear though that you shouldn't take the results as gospel. Mathematical image quality metrics do not always correlate perfectly with how humans perceive image quality, and your personal preference is entirely subjective. You should take this page as what it is, a research that produces numbers, but you should not take these numbers for granted before understanding what they actually mean.