Friday Facts #408 - Statistics improvements, Linux adventures

Posted by Klonan, raiguard on 2024-04-26

Hello,
welcome once again to the world of facts.


StatisticsKlonan

Do you love watching your production graphs grow as much as we do?

Accumulator graph

It is a little bit tricky when you transition from Steam power to Solar panels and Accumulators, to know if you have enough capacity in the system to survive the cold dark nights. Generally you might just wait until nighttime and see if your factory blacksout, if so, build more solar and accumulators.

It would be helpful and convenient to see the statistics of accumulator charge levels, so we added such information:

Accumulator graph

The main reason it was more critical, was that on Fulgora the production from lightning at nighttime is much less predictable, so it is much more important to see the timeline of the accumulator charge.

Fulgora graph

Science graph

You can track science pack consumption in the production GUI, but that does not account for things like productivity modules and the new research productivity technologies.

So we added a new special item in the production statistics, that shows the total final 'Science' that is produced.

Science graph

Per surface production

It was tolerable in the first days of playtesting with planets and platforms that all the production statistics were global. However when you want to get more and more precise with your gameplay and trying optimize each part, it becomes quite necessary.

For instance on platforms, we need to know if we are producing enough fuel and ammo to keep the ride going:

Platform graph

And its super helpful when checking if a specific planet producing enough when some items are crafted in many places.

Vulcanus graph

We also added a checkbox to switch to a 'Global statistics' view, so all the possibilities are available for the player.

Global graph

Quality graph

Going deeper still, we want to dissect our production by what quality the produced items are.

Quality graph

So what do you think? Are there any other statistics improvements you can think about for 2.0?


Linux adventuresraiguard

I have appeared in a few FFFs by now but I have never formally introduced myself. My name is raiguard. I have been playing Factorio since June 2017, making mods for the game since the 0.17 release in March 2019, and I finally joined Wube in March 2023. My primary roles at the company are expansion programming and Linux support, as well as being an advocate for the modding community. I have been daily-driving Linux for multiple years and have fallen ever deeper into the black hole of customization and minimalism.

"Why don't most games support macOS and Linux?" is a sentiment I often see echoed across the internet. Supporting a new platform is a lot more than just changing some flags and hitting compile. Windows, macOS, Linux, and the Nintendo Switch all use different compilers, different implementations of the C++ standard library, and have different implementation quirks, bugs, and features. You need to set up CI for the new platform, expand your build system to support the new compiler(s) and architecture(s), and have at least one person on the team that cares enough about the platform to actively maintain it. If you are a video game, you will likely need to add support for another graphics backend (Vulkan or OpenGL) as well, since DirectX is Windows-exclusive.

Many developers will take one look at the Windows market share and decide that it is not worth the trouble to support other platforms. Also, with the meteoric rise of the Steam Deck and Proton, it is easier than ever for game developers to ignore Linux support because Valve does some black magic that lets their game run anyway.

Factorio supports macOS and Linux so well because there has always been someone at Wube who actively uses these platforms and is willing to take on the burden of supporting it. Our native Apple Silicon support is a great example of this. Today, I will take you through some of the adventures I've had with maintaining Factorio's Linux support.

Wayland

My first self-appointed task after joining the team was to add Wayland support to the game. Wayland is a new display protocol that has been in development to replace the antiquated and insecure X11 system. Modern Linux distributions are beginning to switch to Wayland as default, so supporting it in Factorio is paramount.

We utilize the SDL library which neatly handles most low-level system interactions and abstracts them into a common interface. SDL has support for Wayland, so all that I theoretically needed to do was build SDL with Wayland enabled and it would "just work." However, it's not quite a simple plug-and-play. Wayland provides "protocols" in the form of XML files that you then use the wayland-scanner binary to convert into C program and header files.

Being relatively new to C++ at the time, my initial solution was convoluted and involved checking the generated Wayland protocols into our source tree, to be manually regenerated every time we updated SDL. A few months ago, armed with a years' worth of experience, I improved this workflow to automatically generate the files as a part of the build process, so they are always up-to-date with the protocol XML files that SDL ships with.

Factorio has supported Wayland since 1.1.77, but it needs to be explicitly enabled by setting SDL_VIDEODRIVER=wayland in your environment. For Factorio 2.0 I added a dropdown to select your preference in the GUI:

X11 Wayland

X11 (left) vs. Wayland (right) with the desktop display scale set to 125%.
Notice how the game renders at the display's native resolution when running under Wayland.

Client-side window decorations

Once Wayland support was implemented, I received a bug report that the window was missing a titlebar and close buttons (called "window decorations") when running on GNOME. Most desktop environments will allow windows to supply their own decorations if they wish but will provide a default implementation on the server side as an alternative. GNOME, in their infinite wisdom, have decided that all clients must provide their own decorations, and if a client does not, they will simply be missing. I disagree with this decision; Factorio does not need to provide decorations on any other platform, nay, on any other desktop environment, but GNOME can (ab)use its popularity to force programs to conform to its idiosyncrasies or be left behind.

To fix this, I had to bring in another dependency, libdecor. It functions, and SDL even has support for it, but a video game shouldn't have to supply window decorations in the first place.


The game has decorations now, but the theme doesn't match. Thanks GNOME!

Window resizing seizures

A video is worth more than a thousand words:

PHOTOSENSITIVITY WARNING: Rapid flashing images.

I use the Sway window manager, and a particularity of this window manager is that it will automatically resize floating windows to the size of their last submitted frame. This has unveiled an issue with our graphics stack: it takes the game three frames to properly respond to a window resize. The result is a rapid tug-of-war, with Sway sending a ton of resize events and Factorio responding with outdated framebuffer sizes, causing the chaos captured above.

I spent two full days staring at our graphics code but could not come up with an explanation as to why this is happening, so this work is still ongoing. Since this issue only happens when running the game on Wayland under Sway, it's not a large priority, but it was too entertaining not to share.

Dynamically linked libraries

In a C++ program there are three ways to load/include a library:

  • By including it in your source binary (static linking).
  • Having the system load it when your program starts (dynamic linking).
  • Your program loads it explicitly after startup ("dynamic loading" or what I call "runtime linking").
We have many libraries, such as SDL, FontStash, and Lua, that are statically linked, but Factorio 1.1 also has many dynamically linked libraries:

rai@tantal ~/games/factorio
$ ldd bin/x64/factorio
        linux-vdso.so.1 (0x00007ffc123b1000)
        libdl.so.2 => /lib64/libdl.so.2 (0x00007fc182f70000)
        librt.so.1 => /lib64/librt.so.1 (0x00007fc182f6b000)
        libresolv.so.2 => /lib64/libresolv.so.2 (0x00007fc182f5a000)
        libX11.so.6 => /lib64/libX11.so.6 (0x00007fc182e13000)
        libXext.so.6 => /lib64/libXext.so.6 (0x00007fc182dff000)
        libGL.so.1 => /lib64/libGL.so.1 (0x00007fc182d78000)
        libXinerama.so.1 => /lib64/libXinerama.so.1 (0x00007fc182d71000)
        libXrandr.so.2 => /lib64/libXrandr.so.2 (0x00007fc182d64000)
        libXcursor.so.1 => /lib64/libXcursor.so.1 (0x00007fc182d57000)
        libasound.so.2 => /lib64/libasound.so.2 (0x00007fc182c43000)
        libpulse.so.0 => /lib64/libpulse.so.0 (0x00007fc182bf1000)
        libpulse-simple.so.0 => /lib64/libpulse-simple.so.0 (0x00007fc182bea000)
        libm.so.6 => /lib64/libm.so.6 (0x00007fc182b07000)
        libpthread.so.0 => /lib64/libpthread.so.0 (0x00007fc182b02000)
        libc.so.6 => /lib64/libc.so.6 (0x00007fc182920000)
        /lib64/ld-linux-x86-64.so.2 (0x00007fc182f91000)
        libxcb.so.1 => /lib64/libxcb.so.1 (0x00007fc1828f5000)
        libGLX.so.0 => /lib64/libGLX.so.0 (0x00007fc1828c2000)
        libGLdispatch.so.0 => /lib64/libGLdispatch.so.0 (0x00007fc18280a000)
        libXrender.so.1 => /lib64/libXrender.so.1 (0x00007fc1827fc000)
        libXfixes.so.3 => /lib64/libXfixes.so.3 (0x00007fc1827f4000)
        libpulsecommon-16.1.so => /usr/lib64/pulseaudio/libpulsecommon-16.1.so (0x00007fc18276f000)
        libdbus-1.so.3 => /lib64/libdbus-1.so.3 (0x00007fc18271a000)
        libXau.so.6 => /lib64/libXau.so.6 (0x00007fc182714000)
        libsndfile.so.1 => /lib64/libsndfile.so.1 (0x00007fc182694000)
        libsystemd.so.0 => /lib64/libsystemd.so.0 (0x00007fc1825a1000)
        libasyncns.so.0 => /lib64/libasyncns.so.0 (0x00007fc182599000)
        libgsm.so.1 => /lib64/libgsm.so.1 (0x00007fc18258a000)
        libFLAC.so.12 => /lib64/libFLAC.so.12 (0x00007fc182524000)
        libvorbis.so.0 => /lib64/libvorbis.so.0 (0x00007fc1824f5000)
        libvorbisenc.so.2 => /lib64/libvorbisenc.so.2 (0x00007fc182448000)
        libopus.so.0 => /lib64/libopus.so.0 (0x00007fc1823ec000)
        libogg.so.0 => /lib64/libogg.so.0 (0x00007fc1823e2000)
        libmpg123.so.0 => /lib64/libmpg123.so.0 (0x00007fc182385000)
        libmp3lame.so.0 => /lib64/libmp3lame.so.0 (0x00007fc18230d000)
        libcap.so.2 => /lib64/libcap.so.2 (0x00007fc182303000)
        liblz4.so.1 => /lib64/liblz4.so.1 (0x00007fc1822df000)
        liblzma.so.5 => /lib64/liblzma.so.5 (0x00007fc1822ac000)
        libzstd.so.1 => /lib64/libzstd.so.1 (0x00007fc1821f0000)
        libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00007fc1821cc000)

Among these libraries are X11 and PulseAudio, which are being deprecated in favor of Wayland and PipeWire respectively. This causes a compatibility nightmare because if any dynamic dependencies are missing, the game will not launch. This obviously will not do!

The presence of these dependencies confused me because we utilize SDL for most of the low-level syscalls, audio, and video, and SDL relies entirely on runtime linking. An investigation revealed the source of most of these dependencies to be Allegro, the low-level library that we utilized for most of Factorio's alpha phase but we have since replaced with SDL. The only remaining use of Allegro in 2.0 was as a secondary audio backend in case a user experienced issues with the SDL audio backend, but the SDL backend has been stable for a very long time, so the time was ripe for its removal. This eliminated 123,024 lines of code from the game and drastically reduced the number of dynamic dependencies:

rai@tantal ~/dev/wube/factorio (master)
$ ldd bin/FinalReleasex64Clang/factorio
        linux-vdso.so.1 (0x00007fff96ff2000)
        libresolv.so.2 => /lib64/libresolv.so.2 (0x00007fd2df8a9000)
        libm.so.6 => /lib64/libm.so.6 (0x00007fd2df7c8000)
        libc.so.6 => /lib64/libc.so.6 (0x00007fd2df5e6000)
        /lib64/ld-linux-x86-64.so.2 (0x00007fd2df8d6000)

Clipboard woes

It turns out that Allegro was not the only thing requiring us to link against X11. Back in 2017, we received a bug report that a user could not paste large blueprint strings into the game, and Oxyd fixed this by adding support for X11 incremental clipboard transfers to our GUI backend's clipboard handler.

I was hoping to utilize SDL's built-in clipboard functionality, but unfortunately SDL does not support incremental transfers. This means there are three options:

  • Continue linking against X11, requiring users to install X11 on their system to be able to run the game (I don't want to mess with static linking).
  • Figure out how to do runtime linking and implement that.
  • Upstream our incremental transfers code into SDL so we can leverage SDL's clipboard functions and other SDL-based games can benefit from our work.

As you might guess, I chose the third option. The work to upstream our code is ongoing but should be done in time for Factorio 2.0's release.

Asynchronous saving

Many of you might not be aware that Factorio has support for saving your game in the background, without freezing while it does so. This feature is tucked away in the hidden settings and only works on macOS and Linux. This is one great example of taking advantage of a platform's features to benefit the game, which would not be available to us if we simply went through Proton.

Asynchronous saving works by using the fork syscall to essentially duplicate the game. The primary instance - the one you interact with - continues playing, but the newly forked child runs the saving process then exits on completion. I have used it for many years and have never had issues, but the setting remains hidden because there are a few unsolved problems with it and it requires a significant amount of RAM to work.

I would love to promote this feature away from its hidden status in 2.0. If you are playing on Linux or macOS, please enable asynchronous saving (ctrl+alt+click Settings -> "The rest" -> non-blocking-saving) and report any issues you find. I am particularly interested in reproducing a seemingly random freeze that occurs at the end of the process. Thank you in advance!

Continuing development

This has been but a glimpse of the work I have been doing to ensure Factorio on Linux is the best that it can be. There are still many open bug reports and other issues but I am generally happy with the state of things and can confidently say that Factorio has great Linux support.


As always, submit a text buffer with your feedback to the usual places.