Though early testing, it’s clear the transition to ASP.NET will work. While Nancy would crash with a pitifully low number of requests, ASP.NET does not drop any requests at all with several thousand. However, as I mentioned before, we did not expect to see several thousand users attempt to register on day one. Admittedly, we got a bit ahead of ourselves; planning to implement many moderation features after the game had been established, due to a rather small expected userbase.
Continue reading “Plans for the future – Additional Preparation and Invite Beta”
Ouch! When we tried to launch the game tonight, the servers literally couldn’t take the insane load that we underwent. Here’s everything that happened:
- One day before launch, DOS attacks began hitting the wordpress and forum servers. These were mitigated as they were pretty small scale, and against normal web servers which can deal with a reduced load much better.
- When setting up the API server on our large scale servers, DOS attacks began coming through to the API server recently added to the DNS. These were rate limited, but somehow were still enough to take down the server. It became clear something was wrong with the Nancy driven API server from this.
- By rate limiting to 10 requests a minute (yikes!) we were able to reduce the load on the registration enough to keep it alive, before it was even announced to anyone. During this period, I was able to log in with what seemed like no issues. In this period, 100 users registered.
- I released the registration link on the discord channel. The whole thing blew up, immediately 500 concurrent requests were made to the api server, taking it down immediatly. Hours later, these have still not simmered down – there are a few hundred requests being made and simultaneously active. The large delay each of these requests is incurring is causing the majority of them to fail to get through.
Throughout this process, it became clear that there was a serious issue with the amount of load the API server was able to take. I would say that it should have been able to take at least 25x as much load, and it certainly should not have repeatedly crashed. After some quick experimentation tonight, it seems that the HTTP serving library used to create the API server, Nancy, was serving even plain non-sql requests way slower than other libraries.
We also did not expect this many registrations. Of all of the requests, 885 users from unique IPs signed up. This makes things really difficult. I’m only one developer, and the flash interest in a game that basically died of inactivity years ago is very surprising. It’s also pretty satisfying; it is fantastic to see this much interest in FreeSO, it’s just also very hard to manage.
Could this have been mitigated by opening registrations earlier? Could users whose registrations got through get to play the game?
No, the API server does not only drive registration. If we were to somehow mitigate impact on registration, then the impact caused by users logging into the game would bring it down instead. This would be a lot more certain, as authentication makes 4 times the requests as registration. Registration was also not ready until close to the deadline. This is mainly due to the DOS attack to the wordpress mentioned before.
So what is the plan?
We will perform further load and api testing, look into the root causes of load issues and the server stability. The current plan is to rewrite the API server using ASP.NET instead of Nancy, as it is a lot more “battle-tested”. This could take some amount of time.
We cannot announce a retry date. We might even need a new launch plan altogther, to avoid the load of a few thousand users wanting to play the game on day one. Whatever the plan is, it will be posted on here eventually.
The first city the official FreeSO server will feature is the evolution of Test Center, Sunrise Crater!
I originally planned to reveal this on December 25th, but I got a little carried away with the changes and town layouts. The city is now full of small towns and interesting features for communities to spring up around. Each part of the city has been intricately detailed by hand, from within the client, rather than just whipping up some graphics in a standard editor. Thanks to this, the difference is so staggering you might be surprised that this is still using the same landscaping system as the original.
Continue reading “Introducing Sunrise Crater!”
It has been a very long journey. Started on February 16th 2014, the SimAntics & World Development thread was the first attempt at re-implementing the complex Virtual Machine behind both The Sims and The Sims Online. 17 months later on July 28th 2015, we ran our first online playtest, where everyone appeared as Bob Newbie, the server crashed every 5 minutes and many tables were stood within. Now, 17 months later again, for New Year 2017, we’re finally launching the game in the large scale MMO context that we’ve been working towards this whole time.
Continue reading “FreeSO Open Beta kicks off on January 6th!”
Hello again! It’s been a while since our last post, and a lot has changed. First of all, we have completely ditched Bass.NET for a custom solution using only Monogame (like the rest of our audio) and a decoding library called Mp3Sharp. This lets us play mp3 music through the same channels as our sound effects, meaning, you guessed it, speakers ingame (and robot job) can now play game music! No longer are you stuck dancing to silence.
Last month, the simulator underwent one of the largest and most important changes it’s faced recently – the addition of Permissions and checking them.
Continue reading “Mama-Mia!”