mino

The server owns every position: a client sends the direction it wants to walk in, and the server moves every player 20 times a second and sends snapshots. Several server processes, on one machine or several, share the place through Redis: each simulates its own players and publishes their snapshots, chat and presence, and the others relay them unchanged. The wire protocol is FlatBuffers, binary only. The browser client draws every player with WebGPU in one draw call, about 100 ms behind the newest snapshot, interpolated.

• A player whose server dies continues on another one: a resume ticket, the session and each shard’s last snapshot are in Redis. On AWS, through Cloudflare, with the first machine’s process killed, the player was welcomed on the second 436 ms after its socket closed and was back in place at 473 ms.
• A process at its limit refuses new players with “server busy” while its ticks start late because of its own work, instead of letting every tick slip.
• Guests for now; the first message has room for an identity-service access token.

Measured on AWS

Two c7a.large machines, one server process on each, nginx with TLS in front, and a load generator in the same subnet. Every player walks in a new direction every one to three seconds.

Show

GraphData

• two c7a.large
• one tick, 50 ms

snapshot gap p95, ms

250200150100500

05001,0001,5002,000

players

snapshot gap p95, ms

players	two c7a.large
100	51
250	51
500	52
1,000	58
1,500	104
2,000	237.5

At 1000 players the snapshots stay on time and each machine is about 30 % busy. At 1500 and 2000 the load generator was over 70 % busy, so those two steps are not clean; 2500 and 3000 were invalid.

Show

GraphData

• server
• nginx

1. 100 players
2. 250 players
3. 500 players
4. 1000 players
5. 1500 players
6. 2000 players

	server	nginx
100 players	8.9 µs	12.6 µs
250 players	6 µs	13.7 µs
500 players	5.9 µs	15.4 µs
1000 players	6.6 µs	19.6 µs
1500 players	8.6 µs	31.5 µs
2000 players	9 µs	32.1 µs

nginx encrypts and copies every frame, and its cost grows with the frame's size; the server builds a frame once and hands the same bytes to every socket. At 1000 players nginx used 0.39 cores on each machine, the server 0.13.

Measured on one machine

One desktop, server processes pinned to cores: where one process stops, and why.

Show

GraphData

• loopback TCP
• Unix socket
• the tick rate, 20 a second

ticks per second

2520151050

1,5002,0002,5003,000

players

ticks per second

players	loopback TCP	Unix socket
1,500	20.1	20
2,000	20.1	20.1
2,500	12.2	19.4
3,000	9.9	13.6

A frame costs the server about 30 % less over a Unix socket, and one process carries about 2500 players at 20 ticks a second instead of about 2000. This desktop's numbers, with the busy rule off.

• Arrivals were quadratic: every join read every shard and encoded the whole roster. With 2500 players arriving at about 280 a second, 671 joins timed out before the change and none after; memory went from 297 MB to 142 MB.
• The cost is per write, not per byte. With a client’s frames merged into one write a tick, two processes admitted all 2000 players instead of 1642, on 0.65 cores instead of 0.97.
• Half a million idle websockets on one process and one core cost about 1.3 KB each in the process and 2.7 to 4 KB in the kernel. Sending is the limit: one message to all of them takes 5.7 to 12 seconds of a core.
• The bytes grow with the square of the players in one place, since every player receives every other player’s position 20 times a second. Past a few thousand the limit is the network, and the way past it is to send each player only what it can see.