Surprising Results! Notes from the ashes of my servers.

Uncategorized

Ouch! When we tried to launch the game tonight, the servers literally couldn’t take the insane load that we underwent. Here’s everything that happened:

  • One day before launch, DOS attacks began hitting the wordpress and forum servers. These were mitigated as they were pretty small scale, and against normal web servers which can deal with a reduced load much better.
  • When setting up the API server on our large scale servers, DOS attacks began coming through to the API server recently added to the DNS. These were rate limited, but somehow were still enough to take down the server. It became clear something was wrong with the Nancy driven API server from this.
  • By rate limiting to 10 requests a minute (yikes!) we were able to reduce the load on the registration enough to keep it alive, before it was even announced to anyone. During this period, I was able to log in with what seemed like no issues. In this period, 100 users registered.
  • I released the registration link on the discord channel. The whole thing blew up, immediately 500 concurrent requests were made to the api server, taking it down immediatly. Hours later, these have still not simmered down – there are a few hundred requests being made and simultaneously active. The large delay each of these requests is incurring is causing the majority of them to fail to get through.

Throughout this process, it became clear that there was a serious issue with the amount of load the API server was able to take. I would say that it should have been able to take at least 25x as much load, and it certainly should not have repeatedly crashed. After some quick experimentation tonight, it seems that the HTTP serving library used to create the API server, Nancy, was serving even plain non-sql requests way slower than other libraries.

We also did not expect this many registrations. Of all of the requests, 885 users from unique IPs signed up. This makes things really difficult. I’m only one developer, and the flash interest in a game that basically died of inactivity years ago is very surprising. It’s also pretty satisfying; it is fantastic to see this much interest in FreeSO, it’s just also very hard to manage.

Could this have been mitigated by opening registrations earlier? Could users whose registrations got through get to play the game?

No, the API server does not only drive registration. If we were to somehow mitigate impact on registration, then the impact caused by users logging into the game would bring it down instead. This would be a lot more certain, as authentication makes 4 times the requests as registration. Registration was also not ready until close to the deadline. This is mainly due to the DOS attack to the wordpress mentioned before.

So what is the plan?

We will perform further load and api testing, look into the root causes of load issues and the server stability. The current plan is to rewrite the API server using ASP.NET instead of Nancy, as it is a lot more “battle-tested”. This could take some amount of time.

We cannot announce a retry date. We might even need a new launch plan altogther, to avoid the load of a few thousand users wanting to play the game on day one. Whatever the plan is, it will be posted on here eventually.

42 thoughts on “Surprising Results! Notes from the ashes of my servers.

  1. I think you guys did a fantastic job to find out the issue and take all the negative comments that were thrown at you. I’ve been a game developer myself and I know how much negative feedback you get if something goes wrong. People are immature and not understanding that the hard work you guys have done for this game out of your own living lifes is amazing. I thank you all for trying and will be looking forward for further updates.

  2. RIP Nancy! Lol. Thank you for this great update. It was crazy to see over 400 people on the forums at a single time. Take whatever time you need to rewrite the API. We’ll be here ready xD

  3. I understand the problem, but unfortunately this game many frustrated players waiting for the promised launch for today!

    Please create an official Facebook page of the game (FreeSo) so you can inform the news of the official launch in real time for all the people who are waiting. Good luck, I’ll be anxiously awaiting the game!!

  4. It’s funny how people somehow don’t think the devs are just as upset about this happening, some things are totally out of their control. Give them time to adjust and fix the issues at hand. Even if we lost some people over this – so what? We don’t want people like that, who give up to be in the community anyways..

  5. I want to thank you the Devs for all your hard work. i imagine it has got to be so frustrating. I applaud you guys and gals for the countless hours to bring such an amazing community back together. Rake all the time you need to fix the issue. We will be here!!

  6. While yes I’m disappointed that I won’t get to play….I also understand when things go crazy. Good luck in fixing it. I’ll be here when it comes up for play again! Can’t wait!!

  7. Take all the time you need. I’m just happy that in time, I’ll be able to revisit the place that brought me such joy and tremendous friendships. I’m happy to wait… *Settles down with some tea and memories and waits patiently*

  8. Wow, fantastic interest, and thank you SO MUCH for trying to get this whole deal up and running!!! You’re amazing!!!!

  9. It’s understandable, nobody could’ve forseen that so many people would log in at the same time 😛 Good luck in fixing everything! 😀

  10. I will be here….to be there!!!I figured there would be an outpouring of people wanting to play..I feared the server being able to handle the load. I sure hope and pray the best in your continued endeavors to get this thing going!!!! I know there are thousands upon thousands of DIE NEVER SIMMERS out here, and Im One of them. Thank You.

  11. With all the games out there its weird that so many people want to play this. To this day the sims series is my fav but fore some weird reason i enjoyed playng the 1st part a little bit more than the others. Thank you for all your hard work! Some of us will wait paciently , because its worth it!

  12. The fact that this is taking off the way it is is enough for me! It really excites me to see that when this is final, there will be a player base! I’ve been following all of the TSO projects, and finally one is making progress! Keep it up devs! I’ll be here and waiting for it! 😉

    1. Node.js would be an even worse disaster, and I like programming with it. The rest of our code is in C#, and we already have an ASP.NET prototype that is very promising.

  13. Well at least the registrations are open, right?
    So, more people will have the time to register, download the game and be prepared for the opening.

  14. I’m 16 and was always too young to play! I’m so excited to finally be apart of The Sims Online! These “Rewritten” games have brought so much joy and happiness once again, and they make games that really should have never left return. Thank you so much!

  15. Hi everyone! I am not really familiar with this website or knew that such wonderful people and information was out there. With that being said. I am proud of all that have made the sims vip site, the freeso.net download, and being so kind. I would also like to think KIt (simvip staff) who helped me so fast. I love playing the sims and I have both the sims 3 and sims 4 and am so excited to the future with you all. Like I said earlier, I am very new to this site and new to the freeso.net download. I mean I didn’t know any of this existed until yesterday. I knew that when I searched online for help with my games; I would always see the simsvip website with comments from the members. It has helped so much. I think you all are awesome and I appreciate you working so hard to bring us all so much joy. Thanks again and I had rather you take your time to fix what ever you need than to rush and it not work the way you wanted. Good Luck!

  16. I really appreciate you trying to get this game going again. I’ve been missing it, and it looks like a lot of people have been missing it too! You will definitely see me sign up once the game gets up and running.

  17. OMG we are excited about the launch. Take your time and I know the new world will be fun to play-looking forward to chillin and skillin lol

    1. I think it’s only a text, it doesn’t connect you to EA’s server.

      Maybe in the game’s files there’s file that contains every single text of the game, even the launcher’s texts.
      So, in the future can be changed from “Connecting to EA.COM server” to “Connecting…”

  18. “We might even need a new launch plan altogther, to avoid the load of a few thousand users wanting to play the game on day one.”
    Did you mean a MILLION? Soo excited!!!

  19. Hey, just having it running for even a minute is a blessing on its own! Take as long as you need, man! We sure ain’t gonna lose patience if we’ve been waiting for over 10 years, since the payout would be the return of a masterpiece. 😉

Leave a Reply

Your email address will not be published. Required fields are marked *