mpv Resampling

Introduction

This page is meant to be treated as a follow-up to this scaler comparison done with ImageMagick. I'm only going to talk about niche topics here, so just refer to that other page if you only want to read about scalers.

Upsampling Shaders

The mpv community has written several meme shaders you can use to upsample video in real time, and the rationale is that they can potentially achieve higher quality than the built-in methods. I still do not have a good way of automating mpv tests and therefore I'll have to stick to a single test image, which is still going to be violet.png just because I'm familiarised with it. If this is your first time reading these posts, this is what violet.png looks like:

The small number of samples under test makes this very unscientific, but it is what it is.

Upsampling Methodology

The following shaders were "benchmarked":

1. RAVU
2. NNEDI3
3. FSRCNNX
4. Anime4K
5. FSR

The test image is downsampled with:

It is then converted to grayscale with:

The original image is also converted to grayscale using the same command to create the reference:

We need to convert them to grayscale because some of these shaders do not have RGB support.

The images were then upsampled back with:

Upsampling Results

Upsampling Commentary

The funniest thing about these results is that, according to distortion metrics, Anime4K is actually worse at "recreating the ground truth" than simple resampling filters. The problem seems to be that the shader is too sharp, it does not only ring but also makes the lines thinner. This isn't necessarily bad depending on how blurry your source is, but my test image here isn't blurry at all.

The slower variant of FSRCNNX has also scored worse than its smaller sibling, presumably for the same reason. I don't think igv trained these networks with images that had been downsampled in linear light, so that's probably part of the problem.

The overly-sharp shaders would probably do much better on perceptual metrics, I'll evaluate whether adding a few of them to the benchmarking script makes sense on a later date.

EASU (which is FSR without RCAS) actually scores way better than FSR, and I think it's probably a good option if you're looking for an efficient shader.

It's nice to see that bjin came back from the dead with gold. The new ravu variants perform much better than their previous incarnations and they're also free of the half-pixel shift problem.

Ravu-lite-ar-r4 is the best luma-doubler according to the metrics, and it's probably a safe choice if you're looking for a doubler. If you're not only doubling though, ravu-zoom-ar-r3 might be a better option. You can also edit their trigger conditions or use auto-profiles.lua to switch between them as needed.

Fractional Upsampling

While doing these tests at 2x makes sense since we can use a box filter to downsample, most people aren't always doubling their content. Scaling "performance" at lower scaling factors, such as 1.5x or 1.33x, are usually more interesting because this corresponds to 720p->1080p and 1080p->1440p, respectively. This section is supposed to better represent a real world scenario when you don't have a 4K display or don't watch comically low-resolution content.

Fractional Upsampling Methodology

Unfortunately, some of my choices here are only explained in the following sections. You'll have to bear with me and trust this makes (some) sense, as the alternative would be placing the fractional upsampling section at the end of the page instead of here, its most logical position.

In short, the entire methodology stays the same with a few modifications:

1. Downsampling was done from 1080p to 720p with catrom+PC to create the input image.
2. The doublers were downsampled with hermite (mpv's default) to reach the 1.5x scaling factor.
3. I've removed Anime4K from this because this is too far removed from what the devs intended it to be used for.

Fractional Upsampling Results

Fractional Upsampling Commentary

The numbers are all closer together at 1.5x, which shows that your choice of scaler becomes less relevant as you decrease the scaling factor.

It's a bit funny to see FSR get worse here, since a lower scaling factor corresponds to a higher quality preset in AMD's terms, but I think the problem is just that the shader is too sharp.

It's pretty easy to see that the slightly blurry downsampling step is hurting the doublers. This section previously used catrom+PC to downsample them (which is now only used to generate the "input"), but honestly that was a bit overkill. Using mpv's default downsampling filter makes more sense as that's what most users are probably using.

The conclusion is that at lower scaling factors the benefits these shaders bring become smaller, but the new ravu-zoom variants are all pretty damn good. I personally recommend sticking to the r3 variants, with or without AR.

Chroma Shaders

Using shaders to upsample chroma is an old meme, as igv ported Shiandow's KrigBilateral to mpv years ago. KrigBilateral implements kringing interpolation, which is commonly used in geostatistics. You can read more about it here.

bjin has also released chroma variants of ravu in the past, and I've written a few shaders myself in an attempt to learn GLSL (ended up being too fun and now I can't spend a week without editing these shaders...).

Anyway... My first (not very successful) chroma meme was called jointbilateral and it pretty much implemented this paper. In short, chromatic information from pixels with similar luminosities is preferred (simply uses luma-deltas as an extra weighting factor). The inherent flaw with this idea is that it can't overshoot, so it's literally incapable of reaching some intensity levels depending on how chroma subsampling is done.

My second attempt at this is what's generally known as "CfL", which was written taking inspiration from this paper. In short, Chroma from Luma (Prediction) is a technique that applies a linear regression to map the missing chromaticities based on known luminosities. This is paired with a sharp spatial filter that gets dynamically mixed-in based on the local correlation between chroma and luma.

Chroma Methodology

This test is a little different and it requires a slightly more elaborate process. The first problem with this is that, with real video content, we rarely get a 4:4:4 copy. This means that we generally don't have a reference point to work with, so we have to create one on our own.

I've semi-arbitrarily decided to use a downsampled version of this key visual:

Two versions of the image were created using FFmpeg:

The idea here was creating (nearly) lossless variants with and without chroma subsampling.

The "reference" image was obtained by screenshotting "444.mp4" on mpv.

The images under test were obtained by screenshotting "420.mp4" on mpv using the various chroma resampling filters/shaders.

mpv options remains the same with the exception that we don't need --window-scale=2.0 anymore, and since we're comparing chroma here, the images weren't converted to grayscale.

Chroma Results

Chroma Commentary

Chroma upscaling is often seen as less important since the human visual system is less sensitive to chromatic variations, but all shaders managed to score higher than the built-in solutions. The numeric deltas obviously aren't as bit this time, but the top scoring shader is still significantly ahead of Lanczos (main reference point since it's the default).

If we look at the numbers, CfL is the best shader on all metrics. The default 12-tap variant scores higher than all other shaders and only loses to itself on MAE and PSNR when the 4-tap regression is turned on. I've been chasing minor improvements for a while now and the shader became very good.

Krigbilateral comes right after that and the reason it's so low is because its PSNR score sucks. This shader is extremely hit or miss, and sometimes the pixels are just terribly off-target (easily seen in the hair ornament).

Ravu is also an option is you really dislike what the regression-based shaders do, as it's analogous to a very good polar filter and relatively free of weird artifacts.

You should probably skip Joint/FastBilateral unless you really can't run any of the other shaders due to performance constraints (and in that case you're better off with built-in filters anyway). This shader is kinda hit or miss just like Krig, but since it can't overshoot at all it tends to be worse than even the built-in solutions on very complex scenes.

To conclude, if you want to use a chroma shader you should probably use CfL. My opinion on this matter may be biased since I'm the author of the shader though, so feel free to do your own tests.

Antiring

Antiringing solutions is a topic that I hadn't covered in the previous iteration of this page, but now that we have more than a single option we can also compare them.

In short, antiringing filters attempt to remove overshoots generated by sharp resampling filters when they meet a sharp intensity delta. What is commonly referred to as ringing is simply consequential to the filter's impulse response.

The following image shows this very well:

The negative weights in the filter are there for it to be able to quickly respond to high-frequency transitions, but it makes the filter overshoot a little bit before reaching its final destination. The "intensity" of the ringing is directly related to the magnitude of the secondary lobes. The second lobe, which is almost always negative, is responsible for the overshooting in can see in this example, but filters with more lobes ring once per lobe, and the ringing can be "positive" as well (within the range set by the original pixels) with positive lobes. The "length" of the rings is directly related to the length of the lobes, which is why filters like polar lanczos have "longer" rings (the zero crossings don't fall exactly at the integers, but rather slightly after them).

Antiring Methodology

The methodology here almost is equal to the one used for upsampling, with the only difference being that we have to include --scale-antiring to use mpv's (or libplacebo's) native AR solution. We also don't have to convert to grayscale this time since the shaders support RGB images just fine.

AR is only really necessary when you're using sharp filters, it makes no sense alongside blurry filters because blurry filters don't ring hard enough for it to be noticeable. There are a few sharp memes that are worth trying with AR though if you feel adventurous, but enerally speaking I think polar lanczossharp are pretty well balanced (a bit blurry even).

I'm including my AR shader in this comparison because I think it's better at keeping everything but the overshoots intact, which does create some weird artifacts with sharp transitions sometimes, specially if you use it at ludicrous scaling factors, but the output is generally good enough on real content.

Antiring Results

Antiring Commentary

Previously, this section talked about the different strengths and weaknesses of Pixel Clipper and libplacebo's AR, but that's not really neccessary anymore. Libplacebo's AR has been recently updated and it looks great now. It's actually shocking how similar the solutions look at 2x considering how different the mechanisms are.

The only caveat is that libplacebo's AR is still a bit blurrier at similar strengths, so keep that in mind.

As you can see in the table above, the sweetspot for libplacebo's AR seems to be around 0.6 taking all the metrics into account. I could end the commentary here but if you pay attention you can also see that 0.6 isn't at the top of any of the individual metrics, but it does score above average on all of them. This is actually pretty interesting as these metrics tell us different things:

If we choose to focus on MAE, the sweetspot seems to be around 0.85. MAE is very good at telling us what is numerically closer to the reference without biasing this towards any specific type of artifacts (regardless of how humans perceive them).

If you only want to know how bad the worst pixels are you can focus on PSNR instead, and in this case Pixel Clipper seems to be much better than libplacebo's AR (presumably because the latter is smoother). The sweetspot seems to be around 0.8 here for PC, and 0.5 for libplacebo's AR.

SSIM is good at telling us about sharpness and the whole structural profile of the picture, it doesn't care too much about brightness or contrast variations but it'll heavily punish blurriness or sharp transitions that shouldn't exist. We can see that Libplacebo's AR is better than PC here (again because it's smoother), with 0.75 strength at the top.

MS-SSIM is just SSIM at different scales to emulate a viewer looking at the picture from different distances. This metric tends to correlate better to the human perception of quality than the other 3, and if you're not pixel peeping to catch the ringing this is probably a better representation of reality. For this metric, the picture with no AR at all is the top scorer. FIR filters need to overshoot a little bit to reach some intensity levels and in this case (since polar lanczossharp isn't that sharp to begin with), the ringing doesn't seem to be a big enough problem for it to hurt the score. This isn't true for all filters and it might not be true for all scaling factors either, but for polar lanczossharp at 2x you could easily make an argument that AR is probably unnecessary.

To conclude, my opinion is that you can probably skip AR unless you're using a very sharp filter. If you want to use AR though, the ideal strength will depend on the choice of filter, scaling factor and content type, but something around 0.6 should be pretty safe (if you want to use PC for the higher PSNR you should probably use it at higher strengths like 0.8).

Downsampling Antiring

Pixel Clipper actually has a downsampling variant as well, which I believe is probably less broken than mpv's native --dscale-antiring solution for orthogonal filters. Libplacebo's new polar AR has been recently enabled for downsampling as well, so we can also test that here. I'm skipping doing these tests at different strengths though, as throwing away unwanted information should be easier when downsampling.

The following image will be used for downsampling AR tests:

Downsampling Antiring Methodology

My current method to evaluate downsampling filters is to concede that at 0.5x linear light box is as good as it gets, since in this case we're just averaging 4 pixels together. We can't use box itself for other scaling factors though, so the aim is to find a filter that produces a similar result but is also usable at any other arbitrary scaling factor.

To eliminate any differences caused by how differently mpv and ImageMagick perform linear light conversion, the box reference was also obtained using mpv.

The images were basically generated using the following commands:

Please note that you have to set --scale-antiring=1 instead of --dscale-antiring=1. That's probably a bug but it is what it is.

Downsampling Antiring Results

Downsampling Antiring Commentary

The first thing we can see is that AR seems to be a net-positive in general, and the score differences we see here are much larger than their upsampling counterparts.

It's also good to see that libplacebo's AR seems to work even better when downsampling, as it can remove ringing without introducing any significant blur. This consolidates that there's little reason to use Pixel Clipper over Libplacebo's native AR if you're using polar filters.

It's also worth mentioning that downsampling AR only makes sense when you're using a sharp filter. Blurry filters such as mitchell do not benefit nearly as much from this (since there's almost no ringing to remove), and the remark that some mild ringing can be perceived as a positive thing by the human eye is also valid here. Personally I'd say you probably need some AR for anything sharper than catrom, but catrom itself is still perfectly usable without AR.

Performance Benchmarks

Benchmarking mpv performance is a bit tricky due to a few things:

1. Due to the relatively cheap load, it's difficult to predict at which power state the GPU will stay at while we benchmark. To virtually increase the load, in an attempt to level the ground, we can cheese mpv into outputting frames as quickly as possible, but that's not necessarily representative of a real-world scenario.
2. Hardware, OS, drivers, display servers, etc... All of those things affect mpv performance in complex ways that are usually unpredictable, so even if my benchmarks were perfect they'd still only be relevant for those with similar systems.

In any case, the following numbers were produced with these mpv settings with a 720p anime episode (~24 minutes long video) on a 5600X+6600XT+Windows 11:

Measure-Command { mpv --no-config --vo=gpu-next --gpu-api=vulkan --audio=no --untimed=yes --video-sync=display-desync --vulkan-swap-mode=immediate --window-scale=2.0 --fullscreen=yes }

Please keep in mind that the choice of built-in filter doesn't matter that much as the weights get stored in a LUT, so something like lanczos and spline36 have virtually identical performance. In short, filters with larger radii will obviously be slower, polar filters are slower than their orthogonal counterparts and AR obviously also makes things slower.

For the shaders, the number are more or less what I expected, but it's nice to see that in the grand scheme of things these shaders don't really slow the playback down that much. Good old polar lanczossharp with AR is already as slow as most shaders.

Still though, the conclusion here seems to be that if you're running mpv on a semi-decent computer, these shaders will never be a problem. Please note that you can make it go even faster with hardware decoding, but --d3d11-sync-interval=0 doesn't seem to work in fullscreen, so it is what it is. Until AMD fixes Vulkan hwdec in their drivers this is probably the best I can provide.

Outro

I want to make it clear though that you shouldn't take the results as gospel. Mathematical image quality metrics do not always correlate perfectly with how humans perceive image quality, and your personal preference is entirely subjective. You should take this page as what it is, a research that produces numbers, but you should not take these numbers for granted before understanding what they actually mean.