Hello folks!

Though early testing, it’s clear the transition to ASP.NET will work. While Nancy would crash with a pitifully low number of requests, ASP.NET does not drop any requests at all with several thousand. However, as I mentioned before, we did not expect to see several thousand users attempt to register on day one. Admittedly, we got a bit ahead of ourselves; planning to implement many moderation features after the game had been established, due to a rather small expected userbase. This means a few things for us:

Our new projected user count is way larger than our original launch plans allow

Originally, we planned to launch the game with a large starter lot available for users to skill and earn money, until they could purchase their own lots. This is because we would not be able to provide starting funds for new users since registration was free. This plan was devised when I had projected a maximum of 200 initial users, and assumed we’d be able to fit most people on a super lot with a 128 avatar max count for an opening ceremony.

With over 1000 people, we would not even be able to get everyone in the building, and having all new users play at exactly the same time without lots to split everyone into would be a recipe for disaster. Picking a server configuration to allow this many users to play would be a complete shot in the dark – and we wouldn’t know the root cause or true capacity of a server configuration if we were way over it (it would just crash immediately)

Automated Moderation is Required

A large potential problem with launching a TSO-like game with free registration is multi-accounting to profit faster than other users. Unfortunately, it is currently very easy to do this – we do not have any automatic detection for botting and multi-accounting. As mentioned before, it is possible to do this by monitoring one-way transactions, suspicions large transactions, suspiciously predictable command submissions to the game (botting), and other checks. This would be enough to flag the few suspicious behaviours for investigation by a moderation team.

The plan was to implement this soon after launch – as it would take people a bit of time to rake in enough funds to transfer from a secondary account to their main. With over a thousand people, things can very quickly spiral out of control, so it’s important to get this right as early as possible.

Large demand from Brazilian fansites

On launch day, half of the traffic to the WordPress site was from Brazilian fansites. Our English discord even gained a Brazilian-Portuguese channel and voice chat for a short duration of time, since they were talking in Brazilian-Portuguese in the normal channels. This caused a small scale language divide, and only with a few members. With an equal number of Brazilian and English speaking users, the game’s city would experience a harsh language divide very quickly, and the quality of moderation and gameplay would be severely degraded, with half of the community expecting the other half to speak their language.

I believe to solve this problem, the Brazilian community need their own instance of Sunrise Crater. If this were on the same server as the English one, and I were able to take server donations, an imbalance in donations and game activity could mean that one side’s donations would be paying more for the other. I don’t think I would be able to deliver a satisfactory server to the Brazilian community, due to the language barrier. It would be ideal if the Brazilian community could find and elect their own server admin and moderation team to run their server in a way that works most for them. For this to work, I need to make the servers easier to run/maintain, and help a trusted server admin set up a separate server for Brazil. This would happen close to making the revised game server public, and ideally launch alongside our official servers.

I am still in full time education

I was banking on everything working fine for the launch, as the date was actually set as the latest possible before I had to go back to university and work on my final year project. Unfortunately, it doesn’t seem like it was ever actually meant to be – I was hoping I could get a lot more of the required features done, and sort of hoping that some of them wouldn’t be necessary with a rather small playerbase. It turns out that all of these are completely necessary, as manually moderating thousands of players without some kind of automated assistance (reports, transaction flags, botting flags) would cause some kind of mental breakdown.

I also need more server administrators to help keep the server up. Especially in early days, it’s likely that the server will go down, and I can’t be there to keep it up 24/7. If I am also making sure the servers are running all the time, I would have no time to develop the game itself. I’m a feature developer at heart, so it’s crucial to my own sanity that I don’t become a slave to my own server. Unfortunately, the servers are also quite hard to maintain right now – there is a confusing setup for running our watchdog, updating is complicated and sometimes requires a manual restart, and some things like the server nfs on linux need an easy way for casual systems administrators to set up. I will likely pick people I already know to do this, so please don’t suggest yourself. The problem is, I need to make it easy enough for them!

Because of my situation, things have to slow down again (as they have several times in the past few years). However, contributions to the project in GitHub are more than welcome. We can still launch while I am in full time education, it just needs to be well planned in advance.

A new plan for a “closed” beta

We have a few problems to cover. We have no idea how the city and lot servers will scale with a large number of real users, which is a function of how many lots we have open, how many players are on each lot, how many players are idle in city… We do not know at which points city or lot servers could become unreliable, or how they might behave. A large scale instant launch will break these servers so fast it will be impossible to diagnose where any of these issues start.

The solution is simple, we need to stagger the number of players we allow into the city. The current plan is a beta mailing list, where you submit your email to recieve a FreeSO beta invite code some time in the future. This way, we can gradually scale the server up, see how it behaves, and there will be no need to join a large “city lot” style lot to get started (this would only need to be available through the first few invite phases). I’m also predicting that many people will quickly stop playing. An immediate playerbase of say, 2000 users that gradually drops to 750, is a lot harder and more expensive to deal with than increments of 50 to get to the same 750 end users point.

This will be announced on this wordpress when it is available. The emails to recieve codes will probably be picked in order, but it’s likely some randomization will be employed. Please stay tuned.

Revised Roadmap

Here is a list of things that really need to be done so that we can try again.

  • Complete API transition to ASP.NET (require admin api, some bugfixes, testing)
  • More tools for admin UI, for viewing reports, transactions, flags and bans.
  • Transaction logging and automated flagging tasks.
  • Bot detection plugin (closed source).
  • Ingame moderator tools for bans and warnings, and user reports.
  • Ensure the distributed portions of the system work, including distributed shutdown, emergency shutdown of one component into reconnect and distributed update. Required for horizontal scaling.
  • Set up a system to gradually roll the server out to users via email invites.
  • Cleanup hosting the server to be less of an active chore. This will be necessary to start letting other language communities set up their own server, and let me get on with my life on the other side of things.

This large list means that we cannot launch soon. This isn’t a problem of money, it’s a problem of time (specifically mine). This is definitely possible, we just have to do it right. Sorry for getting you all excited! @_@

A note on private servers & instead helping develop the game!

Please remember, if I, the developer of the game for 3 years, could not successfully run a server, that you or anyone else definitely will not be able to run one successfully! Please also be careful of private servers hosted by non-developers. I know that at least one of the groups that may attempt this were involved with the DOS attack a few days ago, and due to their unpredictable nature I would not trust them with any of your user information.

If you have the ability to host a basic server, your time will be better spent helping me implement these features, so we can launch the game for everyone. The MMO server stuff should be rather easy to jump into developing (especially compared to simantics), so if you have experience with C#, give it a shot! We will be accepting pull requests on our GitHub repo.

I’d like to end this post by saying that we will not be accepting any donations until at least a server is open for players to see the game running for themselves, so be wary of any requests for money posted on social media. The TSO community has been burned once before, and I don’t want it to happen again in case for some reason we fail!

Thanks,

Rhys

Ouch! When we tried to launch the game tonight, the servers literally couldn’t take the insane load that we underwent. Here’s everything that happened:

  • One day before launch, DOS attacks began hitting the wordpress and forum servers. These were mitigated as they were pretty small scale, and against normal web servers which can deal with a reduced load much better.
  • When setting up the API server on our large scale servers, DOS attacks began coming through to the API server recently added to the DNS. These were rate limited, but somehow were still enough to take down the server. It became clear something was wrong with the Nancy driven API server from this.
  • By rate limiting to 10 requests a minute (yikes!) we were able to reduce the load on the registration enough to keep it alive, before it was even announced to anyone. During this period, I was able to log in with what seemed like no issues. In this period, 100 users registered.
  • I released the registration link on the discord channel. The whole thing blew up, immediately 500 concurrent requests were made to the api server, taking it down immediatly. Hours later, these have still not simmered down – there are a few hundred requests being made and simultaneously active. The large delay each of these requests is incurring is causing the majority of them to fail to get through.

Throughout this process, it became clear that there was a serious issue with the amount of load the API server was able to take. I would say that it should have been able to take at least 25x as much load, and it certainly should not have repeatedly crashed. After some quick experimentation tonight, it seems that the HTTP serving library used to create the API server, Nancy, was serving even plain non-sql requests way slower than other libraries.

We also did not expect this many registrations. Of all of the requests, 885 users from unique IPs signed up. This makes things really difficult. I’m only one developer, and the flash interest in a game that basically died of inactivity years ago is very surprising. It’s also pretty satisfying; it is fantastic to see this much interest in FreeSO, it’s just also very hard to manage.

Could this have been mitigated by opening registrations earlier? Could users whose registrations got through get to play the game?

No, the API server does not only drive registration. If we were to somehow mitigate impact on registration, then the impact caused by users logging into the game would bring it down instead. This would be a lot more certain, as authentication makes 4 times the requests as registration. Registration was also not ready until close to the deadline. This is mainly due to the DOS attack to the wordpress mentioned before.

So what is the plan?

We will perform further load and api testing, look into the root causes of load issues and the server stability. The current plan is to rewrite the API server using ASP.NET instead of Nancy, as it is a lot more “battle-tested”. This could take some amount of time.

We cannot announce a retry date. We might even need a new launch plan altogther, to avoid the load of a few thousand users wanting to play the game on day one. Whatever the plan is, it will be posted on here eventually.