Glitchy Mode 5 when HD Mode 7 scaled at 1x and then upscaled at a later stage

[Azrael256]

I am talking here about the libretro implementation, which allows to upscale at the libretro/GUI stage, with shaders for pixel art upscaling. This allows HD Mode 7 in bsnes-hd to be set at 1x, offering great performances, most of the HD-Mode7 goodness, and the ability to use shaders such as hqx or nnedi. It works reasonably well in most situations, but seem to lead to glitchy screens in Mode 5.

Here's an example of the glitch in action with HD Mode 7 Scale set to 1x

And here's how it looks like with HD Mode 7 Scale set to Disabled

I may be wrong but I think HD Mode 7 should not be active at all in Mode 5. Unless I missed something, I do not think it yields any significant benefit and it seems to be mostly a cause for trouble. Otherwise, it does seem to be buggy at 1x and may need fixing. The problem disappears at 2x scaling or more.

[Azrael256]

Additional information from further testing:

The problem happens even when RetroArch is not set to further upscale what is rendered by the core. If BSNES-HD is set to render at 1x and RetroArch is set to render at 1x in windowed mode, the problem still happens. EDIT: I was finally able to test on BSNES-HD standalone too, not only the RetroArch port: exact same problem. When HD mode enabled but scaling set to 1x, all hell breaks loose.

What seems to be happening is that half the pixels are discarded following a vertical lines pattern, and then the remaining pixels are doubled/stretched. Kinda like a basic deinterlacing going on, except on columns rather than rows.

This does not happen in Mode 7 nor apparently in modes 1-4, but happens in Mode 5. It is worth noting that sprites don't seem to be affected by the bug, only background tiles. This leads me to believe it has to do with how this mode has an unconventional way of using even/odd pixels on the x-axis in 16-pixel wide bg tiles.

See here how the sprite is rendered perfectly, but the text and background, not so much.

It is likely that the bug also exists in Mode 6, but I don't know of any game using that mode for testing. Mode 5 can easily be tested using e.g. Secret of Mana or Seiken Densetsu 3.

[Azrael256]

Further testing: the bug started in beta 10.2.
Beta 9 and Beta 10 both display mode5 correctly.
Beta 10.2, 10.5 and 10.6 do not.

[DerKoun]

Thanks you for the report and especially for the extensive research. I really appreciate it.
However, I'm afraid this probably won't be fixed or at least not any time soon.
It is a design decision for bsnes-hd not to change it's resolution for hi-res modes. Instead it paints the pixels into the target resolution. This means that for 1x scale half the information will be lost.
I currently see no way around this. But I'm open to suggestions.

[Azrael256]

How did it work before 10.2? Because it seemed to be working fine in betas 9/10. Was this, paradoxically, a bug that was fixed from 10.2 onward?

Also, would it be possible and sensible to apply shaders before the upscaling stage, allowing for upscaling filters to work correctly whatever the final render resolution, and also make it much less costly in terms of resources... but I guess some shaders, e.g. CRT emulation shaders, would not necessarily work great that way?

[DerKoun]

I honestly have no idea how it could have worked. The resolution just isn't there. Maybe the scale or fast-PPU setting didn't work correctly.

Some time in the future I want to start (collaborative) work on a version of bsnes-hd that uses the GPU for HD Mode 7. One goal for that will be to enable upscaling shaders for everything but Mode 7 backgrounds and also for Mode 7 backgrounds before they are transformed. But that's likely far off.
CRT shaders generally are for low-res input AFAIK. So using 1x scale with super sampling and smoothed gradients (and maybe widescreen) is pretty much all the benefit you can achieve there.

[Azrael256]

Actually, you're right... even in 1x in beta 9/10, it still doesn't look correct, albeit much better than in later builds.

This is how it looks like in beta 9/10 vs HD Mode disabled or set at 2x: readable, but for example one can see on the letter "m" that the middle part is missing, as well as the rightmost part of the letter "x".

So, I guess I will have to chose between HD Mode 7 and upscaled modes 1-4.

[DerKoun]

Comparing the old and new images I think there may indeed be a bug here, maybe drawing pixels twice, horizontally. I'll look into that. Hopefully I just have to exclude some changes from 1x specifically. Thanks for the additional images and research.

[Azrael256]

On a tangentially related note, I am not sure I understand the difference in handling HD Mode 7 between BSNES-HD (both standalone and via libretro) and the "normal" BSnes' HD mode from 2019, that forces BSNES-HD to output everything at a higher scale?

Is my understanding correct that the "HD Mode" from e.g. the normal BSnes' libretro core is a backport from this project?

Yet, in the old BSnes libretro core, even if you set the HD Mode 7 scale to 8x, the actual amount of output/drawn pixels seem to always be the same (equal I think to HD Mode 7 set to 2x in bsnes-hd-beta), and seem to be interpreted as 1x the amount of pixels by the machine, allowing for upscaling shaders to work. Is there a reason why this cannot be achieved in BSNES-HD? Is there a significant difference between what the two projects refer to as "HD Mode 7"?

[DerKoun]

For Mode 7 bsnes and bsnes-hd use pretty much the same code.
However bsnes only applies the scale factor to Mode 7 scenes. Other ones have the default resolution, or double for hi-res or interlaced modes.
bsnes-hd applies the scale factor permanently. This is a design decision to allow other HD features, like smooth background gradients.

[Azrael256]

Aaaaah that explains it. I did notice that upscalers cease to work in Mode 7 scenes even in normal bsnes. So it's a trade-off between allowing for preprocessing features and allowing for resolution-dependant post-processing features. I guess the trade-off used by bsnes-hd makes sense, using pixel-art upscaling shaders is a bit of an edge-case and rendering in HD from the get go creates an opportunity for pretty cool features.

I guess a cool feature in the future would be to offer a downscaling feature in nearest-neighbor. At correct ratio, this should fix the problem, right? Although using high-res rendering+downscaling+re-upscaling is a bit of a convoluted method to do things...

[DerKoun]

Yes, some types of enhancements are mutually exclusive in nature.

There already is limited downscaling, for Mode 7 only. I have tried it for the completed HD image, but there are still issues with that code that I need to fix. (Also, technical note: It should not be "nearest neighbor", as that would eliminate the gained detail. It must blend n by n pixel squares into one pixel each, n being the scale factor.)