Pulse Leaderboard and Rating System

by Stasi Vladimirov Aug 18 2020

First Stretch Goals

Do you wonder how the leaderboard works? What determines your current rank, and what can you do to improve it? In this article, we unveil the details and make it transparent for all competitive players and our community. This includes the original rating system, the replacement, and a new concept, Activity Decay.

The original system: Elo

When we launched TTA Pulse just one month ago, the goal was to release a minimum valuable product (MVP) as quickly as possible, so players can start using it and we can receive community feedback. Than we can start adding more features based on this feedback, and fine tune the platform based on demand and data. We knew that the rating system would be one of things to need adjustments, so we launched with a simple solution: the Elo rating system with a couple of extra bonus points per win. This works, but Elo has some disadvantages. The issues are well-documented online, and you can read about them by searching for ELO rating problems. We can do better.

Other rating systems

In our research for the best solution, we looked at a few options:

For details on each system, follow the links to the Wikipedia articles above. As usual, everything has advantages and disadvantages.

The considerations

When selecting a rating system, it's important to ask the right questions.

How many matches are needed for accurate ratings?

How does activity relate to rating?

How punishing are losses to newly-joining players?

We also want to be mindful of the season length, and what that means for late-joining players.

Other goals: prevent opportunities for abuse, use the ratings for the internal matchmaker, and minimize the complexity of the implementation.

The winner: Bayesian Interference

We decided on using a Bayesian Interference system for internal ratings. It allows us to track the degree of confidence in a player's rating, to reward those who have played enough to get a precise read on their skills. But this alone doesn't sufficiently address our goal of encouraging consistent activity through the season.

A second challenge: inactivity

We noticed a major theme while monitoring the leaderboard. This is a common issue on many competitive platforms: When a player reaches the #1 spot, they stop playing to preserve their ranking. The reasoning makes sense: Given the status, there is less to win and more to lose. Unfortunately, this leads to a stale environment, where it's harder for other players to catch up because they don't get the direct opportunity to defeat the leader. And when the rankings finally change, the problem often repeats itself.

The solution

It took us a lot of discussion and research to come up with the right solution. We studied other platforms, such as StarCraft2, Dota 2, and CounterStrike, and considered bonus pools, win bonuses, and activity bonuses. Finally, we decided on Activity Decay, a diminishing decay system.

Here is the concept:

Here is a specific example:

Hopefully this helps to illustrate this simple concept. Here are some other details:

Other Leaderboard changes

This release brings several new features.

At the end of the season (Sept 30), after we announce the winners, we will also publish a report with statistics on which players had the most consistent ranking throughout the season.

We've worked hard to design this system. As promised, we want to be transparent about how everything works. We hope you truly enjoy it.

Subscribe below if you are curious about how we designed the other major component of this system: the matchmaker.

As always, looking forward to match you Through the Ages!