Table of contents

I am coming back to mobile game development, but this time only for fun, as a pet project to relax at the end of the day. Let me introduce you to this new project.

Outline

The root idea comes from mixing keywords: “PvP”, “online”, “ECS”, and “Android”. Each has its own rationale.

I want to create a game that I could play with friends and family, it will thus be a Player versus Player one. Since we are quite far geographically it will be an online game, by necessity.

I was looking for an excuse to practice Entity Component System designs since a long time. This is an opportunity to do it, see what it’s worth, and learn new things.

Android because this is what I have on my phone and I want to be able to player spontaneously, without having to turn the computer on, and to play far from home.

Those are the ideas I want to gather. There is nothing here that I have not done before, when each idea is taken individually, but combining all of them is new for me, even if not far from previous experiences. This is an opportunity to do the best I can, to reuse what can be reused, to push a little bit further for the known parts and to discover the unknown. Taking small things and combining them together, I like it.

The Tech Stack

I was quite hesitant on the tools to use. Should I take something I know well and thus progress quickly? Should I pick an unknown tool and learn a lot? This is not an easy choice.

Godot

I gave a quick look at Godot but script languages are not my cup of tea. We break something in the programs, it pretends that everything is fine, goes to prod, and fails miserably.

“Heck! If you did not need three parameters for the call to this function, you could have said it on my computer, there’s no shame. Look at where we’re at now!”

I know there is more than GDScript in Godot, I can also use C#, C++, or C. Android support seems correct now (it was not the case when I started my project), which is a good thing. On the other hand I see that they still use SCons as a build system and this won’t work for me.

Additionally a Unity-like development environment is unappealing to me. I used Unity in the past, with its game setting in the UI, and what a pain! :’( There is no easy way to find the object and the field to modify in the interface, no way to do any kind of grep. Then you change the field and thanks to the combination of bindings and immediate mode it burns the computer to refresh everything. Pfff I could have compiled LLVM by the time Unity took my input into account. On top of that you can add the conflicts on 20'000 lines YAML project files when working concurrently on the game then you see how fast the fun-o-meter falls to zero.

In short, graphic tools for programming are not for me. Maybe you will say that Unity is not Godot, and you are right. My fears are maybe unfounded and maybe Godot is very reactive without having to buy a new laptop, and maybe we can grep the project and fix conflicts peacefully in the terminal?

That’s a lot of unknowns.

Bevy

I have obviously checked what was going on on the Rust side, as it would be a great opportunity to learn this attracting language. Moreover it would save me from being asked if I have considered rewriting my software in Rust. A good thing!

The most promising game engine on this side would be Bevy. A good argument for it would be that it uses ECS by default. Unfortunately Android was not supported when I began my game (it is supported since November, 2023).

Today it would be more interesting, with a small scare about dealing with teething problems.

SDL

I really love the SDL <3, its minimalist design, its simplicity. I know from experience that it it absolutely possible to create an Android application with it.

On the other hand, I also know from the same experience that if I pick the SDL I will have to handle everything: loading the textures, developing UI widgets, handling the audio, the inputs. It’s the inconvenient side effect of minimalism. It happens that I just want to write a game, not a game engine, so even though it is really tempting I will leave this option aside.

Axmol

The last time I made a mobile game that actually reached a market it was developed with Cocos2d-x, a game engine in C++. Because of this great experience it was a tempting solution. It’s a pragmatic choice, the tool is a bit of an oldie but it is battle-tested. Unfortunately Cocos2d-x is not maintained anymore and the project has been transformed into Cocos Engine, a Unity-like game engine/dev environment. Just like Godot (and Unity), programming in forms puts a damper on me. UIs for accountants are not for me.

Lucky me, the last version of Cocos2d-x has been forked under a new project named Axmol, and it seems to be very active. Ah-a! This is something I like :) Go Axmol.

About the development

Since I don’t have a lot of time to spend on this project, I try to focus on the essential parts. I had my share of never-ending projects, “only two years of work left” every year during ten years. This time I take care of avoiding pitfalls.

To put it in a single sentence: I unlock something in each dev session. Having a small success every time is essential to feel progress and stay in good spirits. And since I cannot have long dev sessions, I have to work with small targets, and thus to divide again and again until the task becomes feasible. For this, todo.txt is my friend.

Actually the real value of this file is not to keep track of the remaining tasks to do, it is more importantly a way to flush all ideas that come to mind as I progress. Quite naturally the important things end up at the top, and the end of the file is filled with nice-to-have or things that end up being done another way.

The very first day

In order to launch the project I set a first milestone: a UI in the terminal and an avatar (a simple character actually) moving in response to the player’s inputs. This should allow me to start thinking in terms of ECS.

But before this I need something even simpler: some files to compile. Starting a project from scratch is not an obvious thing; this is not something one does daily, and with experience things become even harder. This is not surprising, no one wants to start with something they know can be problematic on the long term. In my case I have spent quite some time to decide how to organize the code repository. Since the project is very small at the beginning, should I have a flat structure, with all files in the same folder? Yet I know that I will have to modularize thing one day, or it will be a mess. So should I create a first module as if there were many? And at some point I’ll have to add tests in many parts, should I handle this now?

Well, I’ll have to jump at some point, so I started with a small module and I expect to move it when the rest becomes real. The first commit does not even have what it needs to compile the project but it’s launched! There’s code, one commit, the project is real, there’s no looking back.

The next days

Once the project is launched the rest unrolls quite easily. The second commit contains what it needs to compile the project and a terminal application that doesn’t do much.

Well, since it doesn’t do anything I’ll have to make it do something. Third commit: we display the game area.

Fourth commit: time has come for a bit of automation. I add a script to download the dependencies and to launch the build.

Fifth commit: since there’s code I add automatic code formatting.

Then comes the addition of EnTT, some tweaks on the compilation flags, a build with AddressSanitizer, a bit of gameplay, some tests…

At this point the foundations are set. There’s what’s needed to compile the project, to test it, and to handle dependencies. From there I continue with the gameplay, a bit of clean-up, and I add a license. I obviously use the best one for free software: CC-BY-NC-ND AGPL3.

Bim! in a terminal

Online game mode

At this point I have approximately one calendar-month of work behind me. Considering one or two hours of work on some evenings, this sums up to something like 25 hours of development… Yeah, this is not a great ratio.

Since I now have a small application in the terminal to test a single-player gameplay, it’s time to switch to multiplayer. Writing the server and exchanging messages is not very difficult, and with the help of a couple of GoogleTests I managed to validate the bases and to detect the most obvious problems. During this time I have also added a build with ThreadSanitizer.

The most difficult part in a multiplayer game is synchronizing the players. In order to have a fair game, and to avoid the most trivial cheats, the server runs the simulation. Yet, since the the players cannot wait a round trip with the server on each tick, they also run the simulation locally, with the information they have. Then they adjust to catch up with the server when they receive a new state. I had a very, veeeeeryyyy, difficult time finding a satisfying implementation. For the inspiration I used GDC17 - Overwatch Gameplay Architecture and Netcode - Timothy Ford and GDC18 - 8 Frames in 16ms: Rollback Networking in Mortal Kombat and Injustice 2 - Michael Stallone. Two excellent talks on the topic, and the former also talks about ECS. Huge thanks to these speakers.

In the final implementation each client keeps a list of actions done since the last state sent by the server. When a new state is received the client removes from this list everything that happened before the new state, then it replays the remaining actions. Its seems easy actually. There’s obviously some complexity added on top of that, for example to handle the defeat of a player. Moreover, each client repeats the last known movement for the other players, supposing that they will continue in this direction. Aside from these complex parts, this is very simple :)

It seems to work well for now but I am far from real conditions and it is very difficult to test properly.

Online mode in a terminal.

Graphical interface

Writing the multiplayer code took approximately two calendar-months. Then I went to another huge part by implementing a graphical interface.

When i was working professionally on mobile games implementing the user interface was by far the largest part of the development. Programming the gameplay is one thing but integrating all the menus, the screens, the transitions between them, this is a very different kind of problem. If we want to additionally do it properly, avoiding the callback hell, the processes floating in the background, the interrupted animations in the transitions… We have to be meticulous.

It’s no surprise then that I have spent a lot more time working the foundations of this part than on the previous. And when I say “the foundations” it’s very basic: a screen with a sprite, a button, a music, and display settings based on some kind of configuration files.

First, when we say sprite we say asset management. On the one size we have static assets, such as the music of the title screen, and on the other hand we have assets produced during the build, like sprite sheets, for example. I have to wire all of this in the build scripts and to pass the corresponding directories to the application on launch.

Then, having buttons means handling inputs. I had a bad experience with the UI tools from Cocos2d-x, both in the asset management and the handling of user input, so I did not try again with Axmol. Consequently I programmed an independent module for the inputs, as well as other modules to write widgets and compose them into game screens. I’m not happy with the state of the module for inputs but it will be good enough for now.

Also, a large effort of this development was reduce the size of Axmol. See, due to its heritage from Cocos2d-x, Axmol contains many, many things. For example we’ll find Box2D, a run-time for Spine, code to load many image types and audio files… This is a nice thing but it comes with a price: many dependencies to handle and huge compilation times. Thankfully many dependencies can be disabled with a simple #define, but to get even shorter compilation time I go as far restricting the compilation of Axmol to the only files I need.

One last word about the integration of Axmol. Just like for many game engines the developers write their software as if the final game will be written in their engine. You’ll find many tools like programs to initialize a project, handle dependencies, compile everything, etc. Especially, there is no easy way to build an Axmol library to plug in another program. This is absolutely not what I need, I don’t want to write my game in Axmol, I want to use Axmol in my game (and it would be the same with other engines like Godot or Bevy, they have to be an addition to my game, not the other way around). This is why I compile Axmol by hand, such that it can be a brick in the software just like other dependencies.

The opening screen. Clearly an AAA game.

Android

There we are, six months later, with a graphical application running on Linux, running correctly and displaying an opening screen. This was not simple but now that it’s done the rest should be simpler. What I need to do now is the same thing on my phone.

I was not looking forward to coming back into Android dev, knowing that it would still be there, that it was patiently waiting until it could eat all the resources of my laptop. I am obviously talking about my old friend Gradle.

“Accelerate developer productivity” they sad. Lol. Look at it downloading the whole Internet; look at it running for an eternity before vomiting a stack trace. What did I do to deserve this?

Allow me to illustrate how Gradle is not fit for the world it lives in. When you have a loop in your shell that reads stdin and launches many Gradle commands then Gradle will consume the whole content of stdin and silently break your loop:

while read -r something
do
  ./gradlew someTask
done < input.txt

No matters how many lines input.txt contains, only a single execution of gradlew will happen, and it will read the remaining lines of input.txt.

Well, Gradle is no joy. Nevertheless I managed to have it output an Android application I can launch on my phone. All it took was one calendar-month.

Continuous integration

Now I have an application that must run on two platforms (Linux and Android), that needs fifteen dependencies for the build, and that compiles dozens of unit tests. I think it’s time to set up some continuous integration to ensure a minimal quality for this project.

Since the code is hosted on GitHub I thought that it was a great opportunity to test their Actions. I can say now that it a really good tool.

I target two platforms so I pick the suitable template and fill my matrix:

• system: the most recent Ubuntu, and let’s add the latest LTS to allow contributors with older systems.
• build type: release, debug, AddressSanitizer, ThreadSanitizer; obviously.
• compiler: G++, no way to ignore it, and Clang, of course.
• target platform: Linux, Android.

2 × 4 × 2 × 2 = 32 configurations tested in a couple of lines, not bad! Also, since I use unity builds for my day-to-day dev I will add an incremental build to ensure that it works too. Bingo! 64 configurations.

Well, as you could have expected, everything is red and it takes three hours to run… Time has already come for some pruning. First, there’s no need to run AddressSanitizer and ThreadSanitizer for all systems, compilers, and target platforms, nor for incremental builds. G++ with the latest Ubuntu, targeting Linux, and it’s enough (ideally a run on Android would be great too but it’s quite a pain to set up). Same for the Android target, there’s no need to test two compilers and two systems, especially as it only impacts the host.

At this point my exclusion list is 80% of my workflow’s configuration. I think it’s time to split it into multiple configurations. Once it’s done, and after many trial-and-error pushes to GitHub, I finally have a green pipeline. By the way, it is very difficult to debug a pipeline, especially when you have to deal with a Gradle that does dumb thinks. Thankfully we still have the best debugger in the world: printf.

In the end if I had to complain about anything in GitHub Actions it’s that they provide system images with too many tools by default. Compilers, tons of tools and libraries… This is not what I expect from my CI. IMHO the point of the CI is to check the full build chain, from the retrieval of the dependencies up to the publication. Obviously if some dependencies come pre-installed, we are at risk of missing a project configuration problem, then when we switch to another system, boom! it does not work. In order to avoid this problem I have simply used the possibility to launch jobs with specific Docker images, as provided by GitHub. As a result I can check that everything works on a minimal system, from the installation of the dependencies to the tests.

Dependency management

Since we are talking about dependencies…

If there’s one C++ trending topic in the last years it’s dependency management. Rust has Cargo, Node.js has NPM, Python has pip, and what about us, what do we have? We would love, too, to download the whole Internet at each and every build, to pull malicious dependencies, and to break production when a third party package is suddenly removed.

For C++ projects we have Conan, vcpkg, CPM.cmake, and probably others. I don’t know much about vcpkg. I keep an eye on Conan since a few years but I’m glad I did not use it seeing how users have suffered when they switched to version 2. Finally, I like the fact that CPM.cmake is integrated in CMake but Conan can also work quite well with CMake. Since I did not dig much into these topic I expect to learn from your comments.

There is quite a blocking limitation in CPM.cmake in my opinion: it does not cache the dependencies. This means that if you have many build folders (e.g. debug, release, sanitizers) then each folder will trigger the download and the compilation of each dependency. What a waste of resources :(

In the general case we distinguish the program dependencies and the build dependencies. Let’s take a concrete example in my great project. In the dependencies we’ll have:

• The app dependencies (i.e. required to run the app): Axmol, iscool::core, fmt, EnTT, Boost, GoogleTest. Those come with their own dependencies: moFileReader, JsonCpp, many other.
• The dev environment dependencies, which I use during the build but not in the program (Pack my Sprites, jsonlint), or that I use for the tests but not for the build (clang-format, ShellCheck, yamllint). Of course this includes essential tools like the compiler, ccache, curl, file, CMake, ninja, Python.

Since I develop an Android app I also need the SDK with some packages (the build tools, the NDK) but more difficult, pay attention, some of the libraries listed above are used differently depending on the target being Android or Linux. For example, I need Boost.Program_options for the Linux build but not for Android.

And to make things even more complex, I need a debug build for some dependencies when I build my app in debug. Moreover, I do patch some dependencies sometimes (e.g. Axmol) and I want to be able to use a branch from my local repository to provide the patched dependency for my app. Is there a package manager that can handle all these constraints?

In a professional context I would certainly use a Docker image to define the build environment, but this is a pet project so I will avoid. In my opinion the most problematic aspect of Docker, just before the incredibly huge storage and disk resources needs, is the lack of orthogonality. For example, if I have a step to install the Android SDK in my Dockerfile, then another step to install Pack my Sprites, then Dockers considers that the latter depends on the former, even though they are absolutely independent! Consequently when I update the SDK all the following steps are repeated. It’s terribly slow and resource consuming. As for using Docker for the app dependencies, with all the patches I do, it’s a lost cause; I would spent my time rebuilding the image.

Until someone releases my dream tool I have hacked a dependency management tool that does dumb archives depending on the build type and target platform, from a local or remote repository, and that can reuse the archives from one build to the next. Deployment is done in a project-dependent directory, with a separation between the host and the target, in such a way that it don’t mess with the host system. Moreover I can pick what is included in the archives and they can be installed and removed independently. Sure it is a very precarious tool that doesn’t handle transitivity very well, but it seems to me that this is the less inefficient solution for me over time. Anyway, it’s a pet project, so I’m allowed to use it :P

The game

“I’ve been reading his words since three hours and he still have not talked about the game.” Hey, I can hear you!

Well, yes, I do not talk about the game yet. There’s not even a link to the project. As you may have observed with the captures above there is not much to show. Not much yet, but be sure I’ll give you details as soon as possible.

Bonus

If you share my tiredness of websites that capture keyboard shortcuts, such as docker.com that puts the focus in its search field on Ctrl+K, you can open about:config and set permissions.default.shortcuts to 2. Firefox with then react as expected on keyboard shortcuts. You’re welcome.