Config File Feature/Firmware Option: Testing, Using, and Feedback

[Ancyker]

In 3468703 I added support for using a config file to toggle some options without having to recompile the firmware.

Why?

The main motivation for this was the new firmware updater is almost finished. The firmware for it is compiled on-demand by my server and because of that, I need to limit the options people can select. I don't want people who are using the updater with HW5 to be stuck with whatever color I choose. Especially because the main target of the updater is people who get prebuilt units which often come with colored cases and such that I think most would want to match with the LCD color.

In addition, there are some things you may want to toggle without having to recompile the firmware, like the global log or the MD SRAM/save file format. In my testing, the config file option costs about 1,616 bytes (1.57kB) of PROGMEM and 73 bytes of DRAM. This does make it the third most expensive optional feature, behind RTC in 2nd using ~2,220 bytes (2.16kB) of PROGMEM/4 bytes of DRAM and the updater support in 1st using ~3,568 bytes (3.48kB) of PROGMEM/229 bytes of DRAM. However, the primary increase of enabling the updater is in the cost of the serial port code/library which does enable more than just the updater, such as setting the voltage (if using VSELECT) and the time (if using RTC).

Why disabled by default?

While the new config file option does open up some nice options, I didn't enable it by default because it offers the least value to people who are already manually flashing their firmware. I enabled the updater by default because I wanted to decrease the odds of people having to manually flash the firmware themselves before being able to use it. People who manually flash their firmware and don't need the additional serial commands can easily disable the updater.

That's in contrast with the intent behind the config file option, which is mainly aimed at people using the firmware updater application. With that, I can easily enable the config file by default when using it -- it doesn't need to be the default for everyone. However, if more options are added, especially ones with no compiler flags as alternatives, then enabling it by default might become reasonable.

What can it do now?

Right now it can be used to change the LCD color, turn off the global log, and change the save format of MD save dumps. The defaults/examples are in the sd/config.txt file. The options are similar to the compiler flags in Config.h, for the LCD color you must set lcd.confColor to 1 for the firmware to use the conf and then the individual color values to values between 0 - 255 (0x0 - 0xFF). The order of the key-value pairs in the config does not matter.

What can't it ever do?

It cannot be used to completely replace the compiler/preprocessor options. Many of these options exist to help reduce the size of the firmware by not including options that aren't needed/desired. For instance, enabling RTC costs 2.16kB of PROGMEM which if you don't have/want/need RTC is less PROGMEM available for other things, like cores. The config file cannot free this memory and would actually cost memory to implement.

For this same reason, the config file cannot be used to selectively enable cores. The AVR architecture (what the ATmega2560/Arduino Mega is) does not allow for executing programs from external memory/storage. It's not a limitation of the Arduino platform, it's a limit of the microcontroller's architecture and thus we have no way to do this. There is technically a hack that could make it work but it requires a custom-made bootloader that checks for a flag at boot and if found writes a program from extern memory to PROGMEM.

While this would technically work, a dedicated programmer is needed to burn a bootloader -- they can't be uploaded via USB. In addition, this would not only increase the time required to select a system it would wear out the PROGMEM flash fairly quickly. Due to how hacky it is, it would likely be somewhat unreliable and it would absolutely decrease the useful lifespan of the ATmega2560. As such, it's just not worth it.

The solution to not having enough PROGMEM is to change the architecture of the OSCR to one with more resources, preferably expandable resources. This is something I'm already working on and you can expect to start seeing prototypes late this year (Q3-Q4). The main holdups have been cost (everything has to be prototyped on custom PCBs -- a breadboard can't be used for much of it) and time (work has been busy lately leaving me with little time to step away from the computer, though I can do things that let me stay at my computer during the times I'm waiting on something which is how I've managed to still get some things done).

Testing/Feedback

I only have HW5, so if anyone can test this on other hardware versions I'd appreciate it. Specifically, I'd especially like to hear feedback regarding HW3 (standard and the STH version) as I don't have them to test with and the firmware updater targets them in addition to HW5.

I'm also looking for any (constructive) feedback anyone might have about the feature in general.

What can it be used for in the future?

For Users

It can be used for anything requiring configuration that may need to be changed often enough that recompiling the firmware would be inconvenient but not so often that a menu option is needed. A good example I can think of is to allow selecting the adapter for a system via a config option which removes the need to choose the adapter type. For example, someone could edit SMS.ino so that if you set sms.gg.adapter to a value it won't prompt for which Game Gear adapter you are using.

For Developers

There are 2 functions to start with, the one that will likely be the most used/useful for most is configGetLong(). This function accepts a config key value and default and returns either the value as a long (for AVRs, int and long are interchangeable/the same/synonyms, but this isn't always the case and the function specifically always returns a long due to the use of strtol() [string to long]) or the value/number passed as the default. If a default isn't passed the default will be 0.

The use of configGetLong() is preferred as it is the fastest/most efficient way to choose an option with code. However, if you absolutely need to allow the use of a string for some reason, you can use configGetStr() which accepts a config key value and returns a String. An empty string is returned if the key isn't present. I did some limited testing of this function and it works, but because it's not used anywhere any use of it will cause enabling the config to use more PROGMEM as it's currently being optimized out by the compiler. So, not only is it less efficient, but it will eat a lot more PROGMEM (and any implementation of it will use more DRAM than a similar implementation using configGetLong()) upon the first use. So, before using it, really consider if you need it and try to lock it behind another compiler flag (i.e. ENABLE_CONFIG_MYOPTIONNAME) rather than having it automatically included when the config is enabled.

Naming Convention

The naming convention of a flag depends on what the flag is for. Config keys are limited to a length of CONFIG_KEY_MAX (default 32), not including the = or string null terminator (\0). Config values are limited to a length of CONFIG_VALUE_MAX (default 32), also not including the = or string null terminator. You should avoid needing to increase these values.

Regardless of what it's for, the key name should be camelCase and use a dot/period to separate distinct parts. Unless there will be lots of subtypes, you shouldn't normally create a subtype. For example, I used lcd.red, lcd.blue, etc instead of something like lcd.color.red because there are not currently nor will there likely ever be any other fields that require a distinction to be made. If you are planning a lot of options for a single thing then it may become worth it. An exception to this is for sub-cores, such as the Game Gear is in the SMS core; for that you could use sms.gg.optionName instead of sms.ggOptionName. In the end, as long as it doesn't require increasing the size of CONFIG_KEY_MAX it will have minimal impact on performance, although it does increase PROGMEM usage by 1 byte per character used.

Options that are global and are not module-specific should be prefixed with oscr., like oscr.logging. Options that are specific to a hardware module, like the LCD or RTC, would use the module's short name as a prefix. An example of a config key in use is lcd.confColor, while some examples of potential option names include rtc.timezone, clockgen.useCalibration, etc.

For cores (aka systems), use the prefix of the system. An example of this in use is md.sramType. Some examples of potential option names include sms.gg.adapter, snes.cic.someOption or snes.someCicOption.

Examples of Usage

// Always wrap the names in F() so they aren't kept in DRAM.
uint8_t lcdRed = configGetLong(F("lcd.red"));

// Note: That actually should produce a compiler warning (converting from a [signed] long to uint8_t), but doesn't for some reason.

// The "!!" ensures we get a value that is type-compatible with a bool.
// The example also shows how to have a default (1 in this example) when the key is not set in the config.
bool loggingEnabled = !!configGetLong(F("oscr.logging"), 1);

if (loggingEnabled) {
   /* ... */
}

// Getting a string
String myString = configGetStr(F("oscr.myString"));

[SAVE-THE-HERO]

I was tested "https://oscr.tools/update".
Excellent work🎉

But there is one problem on HW3
After connecting with PC, PC does not recognize CR with the message "NO SD CARD".
When I reinsert the SD card, the PC recognizes the CR.
This is the same symptom when I select reset on HW3.
https://e.pcloud.link/publink/show?code=XZMO3nZOABqA6wzS90uG5OJLonRLQmDeYw7

This is reset error video
https://e.pcloud.link/publink/show?code=XZiO3nZxCFtOpiou24nc0EJ9YzpIRxxxueX

[SAVE-THE-HERO]

I tested it with 3 different types I have (MicroUSB/USB-C/HandmadeUSB-C).
All of them work fine as a result.

After much cross-checking, I found that the reset does not work correctly only when using the first yellow unit I used.
This is a hardware problem. （I don't know why.)

I consider the testing of the program a success.