
It's been 6 months since [I last wrote about BattleTabs](/posts/battletabs-ships-and-fleets-update) and so much has happened and so many things have changed I felt it was time to do an update.

# 6 Months Ago..

.. We had [just released](/posts/battletabs-ships-and-fleets-update) the ships and fleets update which was by far our biggest update to that point. For the first time we introduced the concept of ships that have abilities and are grouped into fleets. This fundamentally changed the game from what was essentially just Battleships into its own unique experience.

[![](./select-a-fleet.png)](./select-a-fleet.png)

In the same update we added a proper tutorial and gave a UI a much needed overhaul.

[![](./new-sidebar.png)](./new-sidebar.png)

# Whats New

Since then we released a whole heap of new features and improvements over a total of 35 updates!

Some of the biggest new features include:

- New version for web (https://battletabs.io)
- Battle Club, our paid subscription feature giving the player access to exclusive ships fleets, avatars backgrounds and lots more.
- BattleTabs Live, a discord bot that announces major events in the game.
- A user profile modal for viewing states and auth data
- A new authentication system so you can signup and login with socials
- A new "Update Available" / auto update feature
- Private proposals so you can send invites to each other via links
- A bunch of new fleets and ships

[![](./fleets-list.jpg)](./fleets-list.jpg)

# Some Numbers

Players seem to enjoy the regular updates and improvements. The number of matches created per week has steadily increased to the point where almost 70,000 matches are created per week:

[![](./matches-created.png)](./matches-created.png)

Thats over 300,000 turns taken per day!

[![](./turns-taken.png)](./turns-taken.png)

# Discord

We also launched a [Discord Server](https://discord.com/invite/kvJKXsu) in the last 6 months which has seen surprising growth to over 8,000 members.

[![](./discord-memberships.png)](./discord-memberships.png)

At any one time we have hundreds of members online chatting away in the various channels. Its awesome to see such a fantastic community grow around our little game.

Brandon, my co-founder, has been busy running community tournaments which have proved incredibly popular garnering far more sign-ups than there is space for.

[![](./tourny.png)](./tourny.png)

# Under The Hood

<div style='position:relative; padding-bottom:calc(50.42% + 44px)'><iframe src='https://gfycat.com/ifr/redevenbedlingtonterrier' frameborder='0' scrolling='no' width='100%' height='100%' style='position:absolute;top:0;left:0;' allowfullscreen></iframe></div>

As you can imagine with all this growth comes some technical challenges.

## Scaling

In the past 6 months we have had our work cut our for us scaling the backend. Its a challenge as the game uses stateful sockets for communication which complicates the usual "stateless" servers mantra.

Our solution has been to use a combination of careful horizontal scaling of client facing nodes which communicate via a shared custom redis pub-sub mechanism.

Since launch we have run most of our infrastructure on Heroku which has been great for for rapidly getting us to were we are today but its scaling is a little lacking. For now it works but theres plenty room for improvement in the future.

## Client State

Recently we also changed the way state is managed in the client over from [MobX State Tree](https://github.com/mobxjs/mobx-state-tree) (MST) to just [MobX](https://mobx.js.org/README.html). Although [I am a fan of MST](https://mikecann.co.uk/posts/mst-libs) we have found that it unfortunately doesn't play very well with Typescript once your store reaches a certain size.

MST relies on a lot of type inference and union types and if you have been around Typescript for a while you will know that this specific combination of characteristics can quickly lead to a combinatorial explosion of types causing slow compile times and a poor IDE experience.

With MobX things are simpler as state is split between multiple stores defined as classes. This means Typescript doesn't have to work so hard calculating what the type of something should be at any given time.

We do give up some nice MST features such as time travel debugging, undo-redo and state managed purely in the background page but we either weren't using those features or we would have had to stop using them for [manifest v3](https://developer.chrome.com/docs/extensions/mv3/intro/) regardless.

## Monorepo

[![](./monorepo.png)](./monorepo.png)

We changed the way we code to move from a multi-repo approach to a single monorepo approach. This was done to simplify dependency management between the various parts of the system and simplify the versioning story.

This definitely makes things much simpler to understand when you have more than one person working on the codebase at once.

# Future

Theres so much more we have planned both from a feature standpoint and technically. I cant share too much right now but its super exciting so stay tuned!

If you want to play the game you can [download the extension](https://chrome.google.com/webstore/detail/battletabs/mjcklhnhfiepmofggcoegkmkokbljmjd) or play right now on http://battletabs.io/


We were suffering from slow integration tests on our server when I discovered something that sped up performance by 15 times!



# TLDR;

Set `fsync=off` in your postgres config.

# The Problem

For our game [BattleTabs](http://battletabs.com/) run a custom sockets based API server. To ensure that the server is working as intended I like to write integration tests that hit the database. This gives a good level of confidence that calling that endpoint from the game client is going to work as expected.

The issue is that the tests were taking quite a while to run:

[![](./before.png)](./before.png)

I narrowed the issue down to where most of the time was being spent:

[![](./trunc-before.png)](./trunc-before.png)

We must truncate the database before each test runs so that there is no state from the previous test that can affect the results. The problem is that this truncate seems to be taking an inordinate amount of time.

# The Solution

After some googling I came across [this post](https://dev.to/thejessleigh/speed-up-your-postgresql-unit-tests-with-one-weird-trick-364p#:~:text=fsync,are%20physically%20written%20to%20disk.) which seemed to suggest that there was a magical config value `fsync` that I could disable which would speed things up.

So I set `fsync=false` in my postgres config and restarted my docker container and..

[![](./trunc-after.png)](./trunc-after.png)

Wow! Thats a 15x speedup! Incredible.

So whats the downside? Well `fsync` ensures that changes are written (synced) to disk when Postgres makes a transaction. So by disabling it you are disabling protection from crashes power-cuts and whatnot so its [definitely not recommended in production](https://www.2ndquadrant.com/en/blog/postgresql-fsync-off-warning-in-config-file/) but when running tests we dont really care about data integrity so its safe to disable it.

# Github Actions

I wondered if it was possible to apply this sort of speedup to the CI which runs all the integration tests on each push.

To cut a long story short it appears you [cant supply config params to the standard postgres image while running in Github Actions](https://stackoverflow.com/questions/65438055/how-to-add-config-args-to-postgres-service-container-in-github-action/65444387?noredirect=1#comment115705280_65444387) but fortunately we can use another postgres image from Bitnami which does let us disable fsync.

```yaml
jobs:
  test-and-build-pr:
    runs-on: ubuntu-latest

    services:
      postgres:
        image: "bitnami/postgresql:latest"
        env:
          POSTGRESQL_DATABASE: postgres
          POSTGRESQL_USERNAME: postgres
          POSTGRESQL_PASSWORD: postgres
          POSTGRESQL_FSYNC: "off"
        options: >-
          --health-cmd "pg_isready -d postgres -U postgres -p 5432"
          --health-interval 10s
          --health-timeout 5s
          --health-retries 5
        ports:
          - 5432:5432
```

The result is a speedup of 3x, not quite as dramatic as the local speedup (probably because im running postgres in a linux docker container on windows locally and Github is running it on linux) but an impressive performance boost nonetheless!

We were having serious memory issues on the Heroku hosted NodeJS backend for our game [BattleTabs](https://battletabs.com) when we decided to apply "this one little trick"...



# TLDR;

Use [Jemalloc](https://github.com/jemalloc/jemalloc).

# The Problem

For those that aren't regular readers of this blog, for the past year my main focus has been a chrome extension game called [BattleTabs](https://battletabs.com). The game is based around the classic board-game BattleShips except you can play it in a Chrome Extension. This makes it rather unique as the game is immediately available every time you open a new tab in your browser. You can take a turn and move on with what you were doing or you can sit and play a longer session.

Because of the way the game works we keep a persistent socket open to our backend at all times. This allows us to keep the user's state in sync by pushing down updates from the server.

The game has grown considerably over the past 12 months and as a result the demands on the server have continued to grown. That coupled with the increased feature set has seen more and more open sockets and more required memory per socket.

# Vertical to Horizontal

We follow a pragmatic development strategy for the game, meaning that we only spend the time working on improving the infrastructure if its necessary. This meant that for 9 months we weren't really worried about server scaling or performance. We knew the issue was there in the background but whenever we needed more capacity we just bumped up the server size in Heroku:

[![](./heroku-dyno-sizes.png)](./heroku-dyno-sizes.png)

Well unfortunately the day game when we were up to the largest size dyno and we were starting to see some issues again, it was time to do something about it.

The issue wasn't really that of memory at this stage, instead it was the number of simultaneous open sockets to a dyno instance. As far as I am aware there is no docs on what the max number of sockets is per dyno but [this](https://stackoverflow.com/questions/25118381/websocket-concurrent-connections-limit-on-heroku) stack overflow answer seems to suggest 6000.

We were definitely right up against that but at the same time because we are on node we are single threaded so we were also up against a CPU bottleneck trying to handle all the requests when the dynos go through their daily restart.

The time had come to go from a horizontal scale strategy to a vertical one.

We had been putting this off because horizontally scaling a websocket server isn't trivial. Fortunately we didn't have that much local-state and we were using [Socket.io](https://socket.io/) which has some good plugins to help with scaling. I wont detail everything we did here, feel free to contact me if you want details.

[![](./socketio.jpg)](./socketio.jpg)

# Memory Issues

Now we could horizontally scale our server out to as many dynos as we wanted we should in theory be able to drop our dyno size back down to the smallest size (512mb RAM) and then just add more dynos as we grew.

The problem is, we seemed to still be using far more memory than we should have been given we were only running less than 1000 sockets per dyno. Worse still the memory seemed to be increasing over time indicating that there might be a memory leak somewhere.

After days of experiments and searching I decided to look a little closer at what socket-io was doing, thats when I came across this Github issue: https://github.com/socketio/socket.io/issues/3477

[![](./mem-leak.png)](./mem-leak.png)

Others on the issue had done the hard work for me and instead of a memory leak they suggest memory fragmentation was the root cause of the problem.

This was a big "Ahhh!" moment for me as memory fragmentation would look a lot like a memory leak as your available memory over time would decrease as the memory became more and more fragmented.

There were two suggested solutions on the thread. I tried the first which was to disable "perMessageDeflate" on SocketIO. This had some effect but still we seemed to be using far more memory than we should have been.

So I decided to try the second solution too which was something I would never have considered; replace the system memory allocator in Node.

# Jemalloc

The other users on the Github Issue suggested trying [Jemalloc](https://github.com/jemalloc/jemalloc). I personally had never heard of it before but apparently its well known and popular with 5,500+ stars on Github. It describes itself as:

> jemalloc is a general purpose malloc(3) implementation that emphasizes fragmentation avoidance and scalable concurrency support.

Sounds like its exactly what we need, but how do I get node to use that? As with most things Heroku, someone has thankfully trodden this path before me and written a "Buildpack" which I can just plug into my app: https://elements.heroku.com/buildpacks/gaffneyc/heroku-buildpack-jemalloc

And the effect?

[![](./half-mem.png)](./half-mem.png)

Wow! It halved our memory usage! Incredible!

# Conclusion

We were able to scale our heroku server out horizontally and overcome a memory fragmentation issue in socket-io by replacing the memory allocator with Jemalloc.

We have been running like this for a couple of weeks now and have yet to see any issues.

I haven't tested this on non-SocketIO servers but I suspect so I cant say for sure but I suspect it would have a similar magical effect. I would be very interested to hear your results if you tried this!

After months of hard work we are excited to announce that our latest and largest ever update to [BattleTabs](https://www.battletabs.com/) is finally live! Read on to find out about some of the gory details.



# TLDR;

The video below gives you a taste of whats changed.

<iframe width="100%" height="400" src="https://www.youtube.com/embed/JNCD3PhP08w" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen></iframe>

# Ships and Fleets

The most important change in this update is the addition of Ships and Fleets to the game.

Previously the game was pretty much just [standard battleships](<https://en.wikipedia.org/wiki/Battleship_(game)>) except perhaps with the addition of the distance indicators on the cells if you miss:

<iframe src='https://gfycat.com/ifr/DesertedHonoredFluke?hd=1' frameborder='0' scrolling='no' width='100%' height='450px'  allowfullscreen></iframe>

<iframe src='https://gfycat.com/ifr/BlackandwhiteLoathsomeDairycow?hd=1' frameborder='0' scrolling='no' height='450px' width='100%'   allowfullscreen></iframe>

This was enjoyable enough but we always felt it lacked a little depth so after a lot of brainstorming and hard work from our designers we came up with the concept of fleets.

[![](./select-a-fleet.png)](./select-a-fleet.png)

A fleet contains 4 ships which each have a [special ability](https://www.battletabs.com/how-to-play-battletabs-v2/) that they can use.

Some ships live in multiple fleets but they are carefully tuned so that each fleet has an play style.

For example the "Reveal & Snipe" fleet is high risk vs high reward fleet that revolves around discovering the location of the enemy ships with "Reveal" abilities then taking them out in one hit with the Coracle.

[![](./reveal-and-snipe.png)](./reveal-and-snipe.png)

This adds some much needed diversity to the battles as you learn how each fleet works and what its strengths and weaknesses are against other fleets.

[![](./example-battle.png)](./example-battle.png)

One other strong part of this design is that it is an obvious point of extension where we can add more fleets and ships as time goes on, keeping the game interesting to loyal players.

# Tutorial

Because the Ships and Fleets is now quite a bit more complex we also decided to take the time and build a proper tutorial into the game.

[![](./tutorial.png)](./tutorial.png)

Previously we simply popped up a modal with a couple of short gif animations that gave you the general gist of how to play but we always knew that was a temporary measure.

The tutorial now guides you step by step through how to play the game.

<div style='position:relative; padding-bottom:calc(50.42% + 44px)'><iframe src='https://gfycat.com/ifr/ValuableIncomparableAfricangoldencat' frameborder='0' scrolling='no' width='100%' height='100%' style='position:absolute;top:0;left:0;' allowfullscreen></iframe></div>

# Sidebar

We know from our analytics that a large percentage of our users are playing on chromebooks or small windows laptops both of which have very limited vertical height. This meant that the play area they had was small and made smaller thanks to a bunch of vertical space eaten by our UI at the bottom of the screen:

[![](./old-sidebar.png)](./old-sidebar.png)

So we decided as a part of this update to overhaul the UI and move all of this to a bar on the left hand side of the screen instead:

[![](./new-sidebar.png)](./new-sidebar.png)

This not only looks a whole lot better but it frees up vertical space so that the thing thats most important (the game) can take up as much space as possible and thus isnt scaled down.

# Other Stuff

We added a whole bunch of other stuff as part of this update that I wont detail but some of which includes:

- New avatars for players
- Larger maps
- New emote UI
- New Logo
- New info modal on pinning the extension
- Screen shake effects
- Better in-game UI
- Some nice CSS animations
- Faster load times
- Better ship placement and rotation
- Countdown sound effect and better UI

# Conclusion

We are really proud of this update as we think it brings some much needed depth and polish to the game. Hopefully it also sets us up nicely for a bunch of future features we have planned so stay tuned!

If you want to checkout the game head over to: [https://www.battletabs.com/](https://www.battletabs.com/)