Around a month ago, the owners of the longstanding legacy server freeso.ml and I decided to collaboratively set up a test server for people to play in, which would inevitably be wiped as game breaking economy and skilling bugs were worked out. This was purely so people could play the game in its current, mostly working state, rather than waiting for me to deal with my RL business before they could play at all. Our hosts and moderators have proven an absolute blessing in that I’ve been able to put the right amount of time into everything, with them handling support and keeping the server alive.
The server was advertised on the discord and partially the forums, so that the most dedicated users could have a chance at playing first. These aren’t our only users though – we’ve received a lot of attention from various streamers over the past few weeks, which has set off a few communities, top 100 lot competition. At max, we’ve sustained 350 concurrent players in around 60 lots, on one server box. There are just over 2000 lots on the map, and 11,521 avatars in existence. Cumulatively, sims have §75,528,319 in the bank, and have bought §92,393,467 worth of objects (191,956 objects)!
Continue reading “The Sunrise Crater Test Server” →
Hello folks!
Though early testing, it’s clear the transition to ASP.NET will work. While Nancy would crash with a pitifully low number of requests, ASP.NET does not drop any requests at all with several thousand. However, as I mentioned before, we did not expect to see several thousand users attempt to register on day one. Admittedly, we got a bit ahead of ourselves; planning to implement many moderation features after the game had been established, due to a rather small expected userbase.
Continue reading “Plans for the future – Additional Preparation and Invite Beta” →
Ouch! When we tried to launch the game tonight, the servers literally couldn’t take the insane load that we underwent. Here’s everything that happened:
- One day before launch, DOS attacks began hitting the wordpress and forum servers. These were mitigated as they were pretty small scale, and against normal web servers which can deal with a reduced load much better.
- When setting up the API server on our large scale servers, DOS attacks began coming through to the API server recently added to the DNS. These were rate limited, but somehow were still enough to take down the server. It became clear something was wrong with the Nancy driven API server from this.
- By rate limiting to 10 requests a minute (yikes!) we were able to reduce the load on the registration enough to keep it alive, before it was even announced to anyone. During this period, I was able to log in with what seemed like no issues. In this period, 100 users registered.
- I released the registration link on the discord channel. The whole thing blew up, immediately 500 concurrent requests were made to the api server, taking it down immediatly. Hours later, these have still not simmered down – there are a few hundred requests being made and simultaneously active. The large delay each of these requests is incurring is causing the majority of them to fail to get through.
Throughout this process, it became clear that there was a serious issue with the amount of load the API server was able to take. I would say that it should have been able to take at least 25x as much load, and it certainly should not have repeatedly crashed. After some quick experimentation tonight, it seems that the HTTP serving library used to create the API server, Nancy, was serving even plain non-sql requests way slower than other libraries.
We also did not expect this many registrations. Of all of the requests, 885 users from unique IPs signed up. This makes things really difficult. I’m only one developer, and the flash interest in a game that basically died of inactivity years ago is very surprising. It’s also pretty satisfying; it is fantastic to see this much interest in FreeSO, it’s just also very hard to manage.
Could this have been mitigated by opening registrations earlier? Could users whose registrations got through get to play the game?
No, the API server does not only drive registration. If we were to somehow mitigate impact on registration, then the impact caused by users logging into the game would bring it down instead. This would be a lot more certain, as authentication makes 4 times the requests as registration. Registration was also not ready until close to the deadline. This is mainly due to the DOS attack to the wordpress mentioned before.
So what is the plan?
We will perform further load and api testing, look into the root causes of load issues and the server stability. The current plan is to rewrite the API server using ASP.NET instead of Nancy, as it is a lot more “battle-tested”. This could take some amount of time.
We cannot announce a retry date. We might even need a new launch plan altogther, to avoid the load of a few thousand users wanting to play the game on day one. Whatever the plan is, it will be posted on here eventually.
The first city the official FreeSO server will feature is the evolution of Test Center, Sunrise Crater!
I originally planned to reveal this on December 25th, but I got a little carried away with the changes and town layouts. The city is now full of small towns and interesting features for communities to spring up around. Each part of the city has been intricately detailed by hand, from within the client, rather than just whipping up some graphics in a standard editor. Thanks to this, the difference is so staggering you might be surprised that this is still using the same landscaping system as the original.
Continue reading “Introducing Sunrise Crater!” →
Hello folks!
It has been a very long journey. Started on February 16th 2014, the SimAntics & World Development thread was the first attempt at re-implementing the complex Virtual Machine behind both The Sims and The Sims Online. 17 months later on July 28th 2015, we ran our first online playtest, where everyone appeared as Bob Newbie, the server crashed every 5 minutes and many tables were stood within. Now, 17 months later again, for New Year 2017, we’re finally launching the game in the large scale MMO context that we’ve been working towards this whole time.
Continue reading “FreeSO Open Beta kicks off on January 6th!” →
