ClownMDEmu - The Greatest Mega Drive Emulator Ever (Someday)

(Cross-post from my blog)

v0.8
Try it in your web browser: clownmdemu.clownacy.com
Download: https://github.com/Clownacy/clownmdemu-frontend/releases/tag/v0.8.0.1

v0.8 is an update that was burdened by attempting a major change to the codebase. This change was meant to greatly improve the emulator's frontend, allowing for the elimination of numerous hacks within its codebase while making new features possible. Unfortunately, this did not go to plan, and all of the work on it cannot be released today. This change was the migration of the frontend from version 2 of the SDL library to version 3.

This would have been perfect for adding native multi-window support, but deficiencies in the pre-release SDL3 meant that the frontend would also be rendered extremely unstable and buggy. Instead, today's release is a refinement of the emulator based on its prior software stack. While it would be great to announce and show off native multi-window, that feature will have to wait until SDL3 matures. With that out of the way, here are the features that avoided disappointment.

Dark Title Bar on Windows 11
image-4.png


It is about time! Thanks to a tip from Ewan, the frontend now features a little WinAPI hack to set the title bar to the same colour as the Dear ImGui menu bar. This will only work on Windows 11 due to using a recent API.

image-7.png

Now Windows 11 users no longer have to put up with the ebony-and-ivory look that the emulator previously had.

Linux AppImage Build
Linux executables tend not to be anywhere near as universal as Windows executables: their heavy reliance on system libraries and lack of standalone installers make it very difficult to share precompiled software in a form that will work without any hassle on a variety of Linux distributions. Because of this, users either have to compile software from source code, or wait for their distro to provide it as a package. Unfortunately, clownmdemu is not provided by any distro, meaning that compiling from source is the only option. However, this is not a simple process, as it requires recursively checking-out the Git repo and then bootstrapping the build system with CMake - things which beginner programmers are apparently unfamiliar with. All of this combined makes it quite difficult for a Linux user to obtain a useable copy of my emulator.

This is where AppImage comes in - AppImage is a standard for Linux software which bundles all files (executables, libraries, and data) into a single executable, making it very convenient to share, download, and run. Additionally, software is encouraged to be compiled on the oldest still-supported version of Ubuntu LTS, allowing it to run on a huge range of Linux distributions without risk of glibc incompatibilities. With AppImage, I can produce a Linux build that is easy to download and run while also being compatible with the vast majority of current Linux distributions!

Supporting AppImage was not particularly difficult: it merely requires that the emulator's build system be capable of installing the software in a way that adheres to some freedesktop.org standards - particularly providing an icon and basic metadata. Likewise, compiling with an old version of Ubuntu was not very difficult, with it only requiring that I mark some CMake scripts as being compatible with older versions of CMake, and also needing to switch from C++20's format library to its backward-compatible middleware version.

With that done, I am now able to create a clownmdemu AppImage! It is provided alongside the Windows executable in the GitHub releases.

But why stop there? AppImage and its two competing standards - Flatpak and Snap - all provide catalogues for users to conveniently browse and download compatible software. It is one thing to make my software easy to run, but it would be even better to also make it easy to find! So, I have added clownmdemu to appimage.github.io!

Fix for Sprite Data Not Being Cached on Prior Scan-Lines
The ROM-hack 'Gametap Pro Gamer 6' had a couple of splash screens which were not rendering correctly in this emulator:

image-5.png
image-6.png

Hey, that's me!

The cause of this bug was surprisingly difficult to uncover, taking me several days. Eventually, I found that this issue was not the product of a bug, but rather a deliberate design choice: back when I had implemented sprite scan-line caching, I made it so that it would not cache the sprite data of scan-lines which had already been rendered. In hindsight, this makes absolutely no sense. I think this was a misguided optimisation, but, after all these years, I no longer remember why I implemented it in the first place. Regardless, by removing this anti-feature, this bug is now fixed:

image-7.png
image-8.png

I really am the best!

Approximation of the YM2612's 'Ladder Effect'
A feature which purists often mention in relation to the Mega Drive is a curious bug that is present in early models of the console: the 'low-volume distortion' effect which is present in the YM2612 sound synthesiser, but not the YM3438 that is used in later models.

The exact details of how this bug works do not yet appear to be understood, however the general idea of it is that an offset is applied to samples on one half of the sound wave. Exactly how large this offset is is unclear, but I have seen at least one mention of it being approximately 4 decrements of the YM2612's internal 9-bit mixer value.

This bug causes quiet sounds to be louder than normal. Many songs and sounds were designed with this behaviour in mind, causing the absence of this bug to make them sound incorrect. For instance, here is a song from Earthworm Jim 2:
Without the low-volume distortion, the grating whining in the background is absent. According to Tommy Tallarico, this whining is an intended part of the song, rendering its absence a flaw.

Another major example of this is the After Burner II soundtrack, with three songs bearing distinct differences:

YM2612:
YM3438:
  • Red Out has a descending note at the start which is inaudible without the low-volume distortion.
  • Super Stripe has an electric piano-sounding instrument playing some backing notes which are much quieter without the low-volume distortion.
  • After Burner has a lead melody whose notes decay much faster without the low-volume distortion.
These differences were egregious enough that they simply needed to be addressed, and so I have introduced a rough approximation of this bug into clownmdemu. Given its rough nature, and the fact that this bug does not exist (at least to the same extent) in later Mega Drives, an option to disable this is has also been added.

image-8.png

Have it your way!

Support for 'KDEBUG' Debug Messages
It is not often that standards manifest within Mega Drive emulation: there is no standard save state format, no standard layout for modern gamepads, and not even an agreed standard for how to convert Mega Drive colours to modern sRGB RGB888. KDEBUG, however, is an exception:

KDEBUG is an extension to the Mega Drive's API that provides a way for software to log text for the user to view. While useless for players, this is great for developers, in the same way that C's 'printf' function is: it can be used to make the game report things to the developer that are difficult to determine otherwise, such as how many of a pool's objects are allocated, or how many entries are currently populating a linked-list.

This API originated in Gens KMod, which is a modified version of the historical Mega Drive emulator Gens. KMod introduced a vast number of debugging utilities to Gens, with this API being one of them. Many years later, this API was adopted by another emulator, BlastEm. This API would go on to be used by Vladikcomper's debugging framework (which is widely used by Sonic the Hedgehog ROM-hacks) as well as SGDK (which is widely used by Mega Drive homebrew). In all, KDEBUG has become quite ubiquitous.

Indeed, it has become so ubiquitous that I have received multiple requests for KDEBUG support be added to my emulator too.

The API itself is pretty simple: it uses one of the VDP's unused registers as a character stream output, allowing strings to be logged by feeding each character into the register one after the other. After the last character is sent, a null character (literally the byte 0x00) is sent, which the emulator responds to by presenting the complete string to the user. On a real Mega Drive, writing data to this register does nothing whatsoever, making it a harmless no-op. This allows software which uses the KDEBUG API to remain compatible with platforms which do not support it.

As with other developer notifications, KDEBUG messages are recorded to the emulator's Log window. By enabling the 'Log to Console' option, the messages will be sent to 'stderr', allowing them to be viewed from the terminal just like a native PC program.

image.png

An example of Vladikcomper's debugger logging which chunk data is being loaded.

Accurate Motorola 68000 Instruction Durations
CPUs work by processing instructions. These instructions can do things like add two numbers together, perform a multiplication, and so on. Different instruction take a different amount of time to execute... and this is what my emulator has been failing to do for its entire existence.

The Z80 CPU emulator has had proper instruction durations for a long time now, but that was due to the necessity of it: games rely on the Z80's timing to be correct so that audio samples do not play at the wrong speed. However, for the Mega Drive's main CPU, the 68000, this necessity did not exist: many games worked perfectly with every instruction taking 10 cycles, and so my emulator did just that.

This remained the case until I overhauled how Z80 interrupts worked, which exposed an issue: in Sonic 2, with each 68000 instruction taking 10 cycles, the Z80 interrupt would consistently occur right in the middle of a long Z80 bus request operation, causing the interrupt to be lost, which meant that the music would play at an uneven, slow speed.

Rather than mitigate the consequences of this hack by adding another hack, I opted to eliminate the hack by finally implementing proper 68000 instruction durations. This was done by creating a test program which ran the 68000 CPU interpreter in isolation and compared the results to a sprawling series of tests whose data was derived from an extremely accurate microcode-level emulator. This way, I could compare my emulator's behaviour to that of the accurate emulator. This is something that I had already done previously back in late 2021, but I only verified instruction behaviour rather than instruction durations. With this new program, I could verify both!

All instructions now have accurate durations! With this, this music in Sonic 2 no longer plays at a wonky speed, and games should lag (or not lag) more consistently with how they do on a real Mega Drive.

True Multi-Window
Ever since debug menus were first added, they looked like this:

screenshot-debug.png


The menus mostly worked, but they had one glaring limitation:

image-3.png


They were all fake! It was not possible to see a 'fake' window outside of the 'real' window! Yes, you could maximise the emulator, but that meant obscuring all of the other programs that you were using, like your web browser!

The reason that the windows were this way was because they were made with Dear ImGui, which is a library for quickly and easily creating menus. This library works across many different operating systems, rendering its windows inside the program's main window by design, so that it works on video game consoles, which lack a proper windowing system like what a PC has. Unfortunately, this illustrates how Dear ImGui is better-suited for embedding in a video game than for developing a sprawling desktop application.

Despite this, I continued using Dear ImGui because I hoped that its developers would eventually address this shortcoming and allow for the creation of proper windows on platforms that support it. Indeed, such a feature has already been in development for quite some time!

97542423-fe8ffb80-19c6-11eb-9bf5-e26d86364e55.png


There was just one problem: it did not work on Linux. Not only was the X11 support extremely buggy, but it was fundamentally incompatible with Wayland, meaning that it would never support the protocol without first undergoing major modifications. This was a deal-breaker.

For years, I hoped that something would change, but nothing ever did. Even now, Dear ImGui's 'multi-viewport' support is completely unusable on modern Linux.

Finally, I had waited long enough, and decided that, if Dear ImGui was not going to support multiple windows for me, then I was going to do it for myself. This is the result:

image-1.png

It's finally here!

At last, all menus are now proper windows! This was achieved by bypassing Dear ImGui and manually creating windows using the underlying API, SDL2. Unlike Dear ImGui, SDL2 supports creating windows on all major operating systems, including Wayland-powered Linux.

Long ago, I had refactored the frontend so that a window was represented by a class, with which it was easy to create multiple windows. By inheriting from this class, I could create a class which represented a window that used Dear ImGui widgets, and then, by inheriting from that, I could create a class that represented a menu window. In this hierarchy, the main window is a 'window with Dear ImGui', while all other windows are 'menu windows'. Each menu can then inherit from the menu-window class and extend it with a unique interface. By structuring the code this way, all menus share a great deal of code, making it very easy to create new ones. Additionally, by centralising so much code into common classes, I made it easy to apply fundamental modifications to all menus at once! This would be very useful for solving one particular problem...

Unfortunately, not all platforms support using more than a single window. One platform in particular is one that my emulator only recently gained support for - Emscripten. By making my emulator use native windows, I had broken compatibility with platforms that cannot use them! To address this, I made it so that a menu can be either a real window or a fake window. Due to the aforementioned centralisation of the core menu code, this was very easy to do! As a result, the Emscripten version of the emulator will use Dear ImGui windows, while other platforms will use normal windows!

Allowing the menus to be used outside of the main window has been a feature that many users have requested over the years, so I am very happy to finally see this fulfilled!

There is just one problem: these menus are jank. On Windows, Linux, and presumably every other platform, these windows are not considered 'subwindows' of the main window. This means that each one gets a slot on the operating system's task bar, and that un-minimising one window will not cause the rest of them to be un-minimised along with it. These sound like small problems, but they really bring the whole experience down.

image-9.png

This is annoying.

SDL2, despite flaunting its ability to create multiple windows as one of its main improvements over SDL1, has no API to overcome this. There is one function for designating a 'modal' window, but it does not solve this particular problem. Not to mention, that function is only implemented on Linux; it does absolutely nothing on Windows and macOS, and was only introduced because the Linux port of the Unreal Engine's editor needed it. It is things like this which call SDL2's role as a platform-abstraction library into serious question.

Because of this, Dear ImGui windows remain the default for now, until something can be done to make native windows less clunky. For those feeling adventurous, native windows can be enabled by editing the frontend's configuration file, which can be found at '%APPDATA%/clownacy/clownmdemu-frontend/configuration.ini' on Windows and '~/.local/share/clownacy/clownmdemu-frontend/configuration.ini' on Linux. Just change the 'dear-imgui-windows' setting from 'on' to 'off', and you are good to go!

SDL2's successor, SDL3, provides a function for making a window the parent of other windows, which would solve all of these problems. One would think that the obvious thing to do is migrate this project to SDL3 so that it can make use of this feature, and that is exactly what I did! Unfortunately, SDL3 is still a work-in-progress, being far too unstable for regular use. While the day will come that the switch to SDL3 can be made, and the improved native windows can finally be released for everyone to use, that day is not coming any time soon.

Preliminary Migration to SDL3
Background
From the very beginning, this emulator has used the SDL2 library for handling the various differences between operating systems, allowing the emulator to work on any platform with minimal effort. There is one problem with it, however: it is quite old.

SDL2 as we know it began as SDL 1.3 in the late 2000s. Since its inception, it has maintained a rock-stable programming interface (both API and ABI), which allowed it to continue working with all software that had been built upon it, even after many years of releases, features, and heavy refactoring. This interface also had the unfortunate downside of cementing various poor design decisions into SDL2, forcing developers to deal with them for well over a decade.

At long last, SDL's developers have opted to shed these years' worth of flaws by developing SDL3, which boasts a refreshed API. Gone are quirks like functions returning 0 to indicate success while others would do it to indicate failure, the filtering of textures being specified by clunkily altering global state, and high-DPI support on Windows being enabled completely differently to how it is for other platforms.

In addition to a cleaned-up interface, SDL3 boasts new features like a 3D rendering API, file dialogs, and a replacement for the conventional 'main' function that makes software immediately compatible with Emscripten.

None of this is particularly beneficial to my emulator, since it does not need 3D rendering, it already uses its own custom file dialog logic, and it provides special-case code for Emscripten to work around the incompatibility with 'main' functions. However, by discarding this custom code and offloading the work to SDL3, not only would the codebase be simplified and lightened by several hundred lines of code, but improvements made to SDL3's implementation of these features would directly benefit the emulator, whereas it would remain completely unaffected if it continued using its own implementations.

What ended up pushing me over the edge, however, was the ability for a window to declare itself as the child of another window. By doing this, the child window is not allocated an icon on the task bar, and it is presented to the user whenever its parent is. This may seem minute, but it goes a long way towards making a multi-window program not feel like a mess to use.

Wanting to make the emulator's multi-window support the best that it can be, I decided to migrate my emulator from SDL2 to SDL3.

Migration
The process of migrating to SDL3 was very complex. Fortunately, the developers of SDL provide extensive documentation for how to port software to the new library, along with a few scripts to automate certain tasks such as renaming functions.

One of these scripts gave me a great deal of trouble, that being the Coccinelle script. Coccinelle is a program meant for applying patches to source code in a syntax-aware manner, but, on Windows and Arch Linux, it merely crashes when attempting to run the script. It lacks a Flatpak and neither MSYS2 nor Arch Linux provide a package for it, requiring that it be compiled from source, which makes it unclear whether the fault is with the program or merely how it was compiled. The only way that I got the script to run was by installing a recent version of Ubuntu in a virtual machine and installing and running its Coccinelle package from there, which is far more effort than it is worth. I do wish SDL's developers had chosen a more reliable tool.

After applying the Coccinelle script, I was able to run the accompanying Python scripts without any problems. From there, I consulted the migration documentation to address every compiler error that I encountered until the emulator successfully compiled. Unfortunately, a few bugs had sneaked in along the way, due to some API changes not producing compiler errors despite the issues they caused, such as the 'SDL_Event' struct's 'cdevice' ('controller device') member being renamed 'gbutton' ('gamepad device'), and then a new member being added that was also called 'cdevice' (short for 'camera device' instead). This problem could have been avoided by using a more verbose naming scheme.

With the emulator now running atop SDL3, I could replace its custom file dialog and 'main' function logic with usage of SDL3's new APIs. I also considered doing the same to replace the audio mixer, but I am uncertain about whether SDL3's mixer features dynamic rate control like mine does, so I have not made the switch.

Result
As of now, the SDL3 migration is complete. The codebase is benefitted by this in multiple ways: being free of many lines of code that were made redundant by SDL3, having clearer code due to targetting a better API, and being able to make use of features that did not exist in SDL2. With this, the emulator is prepared for the future!

...Unfortunately, it is not prepared for the present: SDL3 is still pre-release software, meaning that it is lacking in polish and stability. A glance at SDL's issue tracker will show many flaws in its current implementation. The worst problem occurs when closing a window. On Windows 11, this causes the desktop to behave erratically, as if the compositor or GPU driver had crashed, while on Linux and Windows XP, the emulator itself crashes. Strangely, this only seems to occur with the software and Vulkan renderers; OpenGL and DirectX appear to work just fine.

Because of all of this, the SDL3 migration was moved to its own dedicated branch. The main version of the emulator will remain with SDL2 until things improve. Those who are interested in trying the SDL3 port can find its source code here.

Closing
I think this is the first time in the project's history that I have implemented features which were either completely reverted or locked behind a hidden setting. Both native window support and the SDL3 migration held this update back for months, so it is a tremendous shame to not see all of that effort pay-off. This has me wondering if it would be worth abandoning SDL in favour of another library, such as GLFW. Unfortunately, SDL is so deep-rooted into the codebase that removing it would involve rewriting a great amount of code. While it is possible, it would not be easy. Still, refactoring is one of my favourite programming tasks, so maybe I will take the plunge. Only time will tell.
 
Last edited:
Back
Top