Friday Facts #83 - Hide the latency

Posted by Tomas on 2015-04-24

Good afternoon,

the work on 0.12 are continuing quite well according to the updated roadmap presented in the previous facts. So this is a good opportunity to talk more in depth about one of the upcoming improvements for the Multiplayer - how we plan to deal with the latency.

The Latency problem

Let's start by defining the problem. You want to play a multiplayer game with your friend (that is not the problem yet). If you live far apart (i.e. different countries) or have flaky internet connection, it will take some time for packets to travel from one computer to another. This is what we call network latency. So depending on the distance and connection quality the network latency can get quite high. Also because the connection is unreliable we need to work with a round trip latency, basically doubling the number by two.

When the multiplayer game is started there is a latency slider which defines the expected roundtrip latency. At the moment this needs to be set manually. This value is the same for all the players. This number defines a lag between the time of user action occurence and the time of application of that user action. Say you click a mouse to open the machine gui. In single player this happens instantly. In multi player this happens only after defined latency time elapses (on average). The reason for this is to keep the determinism. All the players need to apply all the user actions in the same order. Therefore your game can apply your user action (and hence open the gui) only when it has applied all the previous (in game time) user actions from other players. So the result is that you click the entity and then wait, wait, wait aaaand only then the gui opens.

Latency Hiding

This problem is mitigated in client - server models (Quake, Doom, etc.). This is possible because the game state is rather small. As far as I know, the client there updates the state and now and then checks with the server whether his state is correct. If not it downloads the authoritative state from the server. However as mentioned quite a few times before this is not the way for us simply because Factorio game state is too big to transfer over network (and it is also changing too fast - think all the entities moving on transport belts).

Ok, this lag is annoying. Really annoying. So one of the things we are working on for 0.12 is a mechanism called latency hiding. In short this will pretend that the user action takes place immediately for some of the common tasks. It is neither aiming to be perfect nor to work for every action. We are after making the game experience as smooth as possible within given boundaries. At the moment most of the work is done and the following is working already:

  • player movement
  • entity selection
  • opening / closing guis
  • building / fast replacing entities
  • mining resources / buildings
  • picking things to cursor
We might add some details and maybe a car movement (but that is tricky because of the collisions - see below).

Latency State

Technically the latency hiding is done by having a special layer (we call it latency state) which duplicates the relevant part of the game state. Every tick this latency state is cleared and initialized from the regular game state. Then all the buffered local user actions that hasn't been applied yet in the game state are applied to the latency state. So back to our example with opening machine gui, the result would be that the gui is still closed in the game state, but there is an information in the latency state that the gui will probably open. The latency state is then used for drawing the screen (gui is displayed) and creating new user actions (player can take an item from the gui into cursor - even though in real game state the gui is still closed).

The nice thing about this approach is that it is self-correcting. This is because the latency state is reset every tick and reinitialized from the real game state. So what happens when there is user action collision? Let's say a player starts moving (in latency state) and then some other player builds an object in his way. Because of latency hiding the player starts running immediately. However when the tick, in which the object is built by other player, is processed, the player will seem to be "teleported" in front of the object and stay stuck there running into the object. There is no state correction or anything. Only the result of applying pending user actions (player running) is different after the information about new building has been received.

Future plans

In general the more state is involved/impacted by the user action the harder it is to perform latency hiding for this. Therefore we don't plan to do any latency hiding for interacting with entities (apart from basic operations like opening, rotating, etc.) or fighting. Shooting can result in cascade of game state changes which would have to be captured in the special layer (latency state). We find this just too hard to do for now.

The plan for us is to spend some time playing with high latency settings and make further changes / improvements based on this experience. Naturally the more players there are in the game the bigger the maximum peer latency gets (maximum latency between all the pairs of players). This is another weakness of our P2P model. However in the near future we plan to work on this one in 2 ways:

  • Reduce the size of multiplayer packets (heartbeats). Most of the time the packets contain no information, yet there is data transmitted because of serialization overhead for empty data structures. This will be improved for 0.12.
  • Allow relaying the communication via game hosting computer. This will be even better when the hosting computer can be a dedicated server (also something we will work on). This way players will need to communicate only with the server and not with each other. That means less traffic and avoiding potentially slow connection combinations. This probably won't make it into 0.12, but it is high on the list.

Today instead of a picture there is a short video which shows latency hiding in the game.

As usual, let us know what you think at our forums.