Hello folks!

Though early testing, it’s clear the transition to ASP.NET will work. While Nancy would crash with a pitifully low number of requests, ASP.NET does not drop any requests at all with several thousand. However, as I mentioned before, we did not expect to see several thousand users attempt to register on day one. Admittedly, we got a bit ahead of ourselves; planning to implement many moderation features after the game had been established, due to a rather small expected userbase. This means a few things for us:

Our new projected user count is way larger than our original launch plans allow

Originally, we planned to launch the game with a large starter lot available for users to skill and earn money, until they could purchase their own lots. This is because we would not be able to provide starting funds for new users since registration was free. This plan was devised when I had projected a maximum of 200 initial users, and assumed we’d be able to fit most people on a super lot with a 128 avatar max count for an opening ceremony.

With over 1000 people, we would not even be able to get everyone in the building, and having all new users play at exactly the same time without lots to split everyone into would be a recipe for disaster. Picking a server configuration to allow this many users to play would be a complete shot in the dark – and we wouldn’t know the root cause or true capacity of a server configuration if we were way over it (it would just crash immediately)

Automated Moderation is Required

A large potential problem with launching a TSO-like game with free registration is multi-accounting to profit faster than other users. Unfortunately, it is currently very easy to do this – we do not have any automatic detection for botting and multi-accounting. As mentioned before, it is possible to do this by monitoring one-way transactions, suspicions large transactions, suspiciously predictable command submissions to the game (botting), and other checks. This would be enough to flag the few suspicious behaviours for investigation by a moderation team.

The plan was to implement this soon after launch – as it would take people a bit of time to rake in enough funds to transfer from a secondary account to their main. With over a thousand people, things can very quickly spiral out of control, so it’s important to get this right as early as possible.

Large demand from Brazilian fansites

On launch day, half of the traffic to the WordPress site was from Brazilian fansites. Our English discord even gained a Brazilian-Portuguese channel and voice chat for a short duration of time, since they were talking in Brazilian-Portuguese in the normal channels. This caused a small scale language divide, and only with a few members. With an equal number of Brazilian and English speaking users, the game’s city would experience a harsh language divide very quickly, and the quality of moderation and gameplay would be severely degraded, with half of the community expecting the other half to speak their language.

I believe to solve this problem, the Brazilian community need their own instance of Sunrise Crater. If this were on the same server as the English one, and I were able to take server donations, an imbalance in donations and game activity could mean that one side’s donations would be paying more for the other. I don’t think I would be able to deliver a satisfactory server to the Brazilian community, due to the language barrier. It would be ideal if the Brazilian community could find and elect their own server admin and moderation team to run their server in a way that works most for them. For this to work, I need to make the servers easier to run/maintain, and help a trusted server admin set up a separate server for Brazil. This would happen close to making the revised game server public, and ideally launch alongside our official servers.

I am still in full time education

I was banking on everything working fine for the launch, as the date was actually set as the latest possible before I had to go back to university and work on my final year project. Unfortunately, it doesn’t seem like it was ever actually meant to be – I was hoping I could get a lot more of the required features done, and sort of hoping that some of them wouldn’t be necessary with a rather small playerbase. It turns out that all of these are completely necessary, as manually moderating thousands of players without some kind of automated assistance (reports, transaction flags, botting flags) would cause some kind of mental breakdown.

I also need more server administrators to help keep the server up. Especially in early days, it’s likely that the server will go down, and I can’t be there to keep it up 24/7. If I am also making sure the servers are running all the time, I would have no time to develop the game itself. I’m a feature developer at heart, so it’s crucial to my own sanity that I don’t become a slave to my own server. Unfortunately, the servers are also quite hard to maintain right now – there is a confusing setup for running our watchdog, updating is complicated and sometimes requires a manual restart, and some things like the server nfs on linux need an easy way for casual systems administrators to set up. I will likely pick people I already know to do this, so please don’t suggest yourself. The problem is, I need to make it easy enough for them!

Because of my situation, things have to slow down again (as they have several times in the past few years). However, contributions to the project in GitHub are more than welcome. We can still launch while I am in full time education, it just needs to be well planned in advance.

A new plan for a “closed” beta

We have a few problems to cover. We have no idea how the city and lot servers will scale with a large number of real users, which is a function of how many lots we have open, how many players are on each lot, how many players are idle in city… We do not know at which points city or lot servers could become unreliable, or how they might behave. A large scale instant launch will break these servers so fast it will be impossible to diagnose where any of these issues start.

The solution is simple, we need to stagger the number of players we allow into the city. The current plan is a beta mailing list, where you submit your email to recieve a FreeSO beta invite code some time in the future. This way, we can gradually scale the server up, see how it behaves, and there will be no need to join a large “city lot” style lot to get started (this would only need to be available through the first few invite phases). I’m also predicting that many people will quickly stop playing. An immediate playerbase of say, 2000 users that gradually drops to 750, is a lot harder and more expensive to deal with than increments of 50 to get to the same 750 end users point.

This will be announced on this wordpress when it is available. The emails to recieve codes will probably be picked in order, but it’s likely some randomization will be employed. Please stay tuned.

Revised Roadmap

Here is a list of things that really need to be done so that we can try again.

  • Complete API transition to ASP.NET (require admin api, some bugfixes, testing)
  • More tools for admin UI, for viewing reports, transactions, flags and bans.
  • Transaction logging and automated flagging tasks.
  • Bot detection plugin (closed source).
  • Ingame moderator tools for bans and warnings, and user reports.
  • Ensure the distributed portions of the system work, including distributed shutdown, emergency shutdown of one component into reconnect and distributed update. Required for horizontal scaling.
  • Set up a system to gradually roll the server out to users via email invites.
  • Cleanup hosting the server to be less of an active chore. This will be necessary to start letting other language communities set up their own server, and let me get on with my life on the other side of things.

This large list means that we cannot launch soon. This isn’t a problem of money, it’s a problem of time (specifically mine). This is definitely possible, we just have to do it right. Sorry for getting you all excited! @_@

A note on private servers & instead helping develop the game!

Please remember, if I, the developer of the game for 3 years, could not successfully run a server, that you or anyone else definitely will not be able to run one successfully! Please also be careful of private servers hosted by non-developers. I know that at least one of the groups that may attempt this were involved with the DOS attack a few days ago, and due to their unpredictable nature I would not trust them with any of your user information.

If you have the ability to host a basic server, your time will be better spent helping me implement these features, so we can launch the game for everyone. The MMO server stuff should be rather easy to jump into developing (especially compared to simantics), so if you have experience with C#, give it a shot! We will be accepting pull requests on our GitHub repo.

I’d like to end this post by saying that we will not be accepting any donations until at least a server is open for players to see the game running for themselves, so be wary of any requests for money posted on social media. The TSO community has been burned once before, and I don’t want it to happen again in case for some reason we fail!

Thanks,

Rhys

Ouch! When we tried to launch the game tonight, the servers literally couldn’t take the insane load that we underwent. Here’s everything that happened:

  • One day before launch, DOS attacks began hitting the wordpress and forum servers. These were mitigated as they were pretty small scale, and against normal web servers which can deal with a reduced load much better.
  • When setting up the API server on our large scale servers, DOS attacks began coming through to the API server recently added to the DNS. These were rate limited, but somehow were still enough to take down the server. It became clear something was wrong with the Nancy driven API server from this.
  • By rate limiting to 10 requests a minute (yikes!) we were able to reduce the load on the registration enough to keep it alive, before it was even announced to anyone. During this period, I was able to log in with what seemed like no issues. In this period, 100 users registered.
  • I released the registration link on the discord channel. The whole thing blew up, immediately 500 concurrent requests were made to the api server, taking it down immediatly. Hours later, these have still not simmered down – there are a few hundred requests being made and simultaneously active. The large delay each of these requests is incurring is causing the majority of them to fail to get through.

Throughout this process, it became clear that there was a serious issue with the amount of load the API server was able to take. I would say that it should have been able to take at least 25x as much load, and it certainly should not have repeatedly crashed. After some quick experimentation tonight, it seems that the HTTP serving library used to create the API server, Nancy, was serving even plain non-sql requests way slower than other libraries.

We also did not expect this many registrations. Of all of the requests, 885 users from unique IPs signed up. This makes things really difficult. I’m only one developer, and the flash interest in a game that basically died of inactivity years ago is very surprising. It’s also pretty satisfying; it is fantastic to see this much interest in FreeSO, it’s just also very hard to manage.

Could this have been mitigated by opening registrations earlier? Could users whose registrations got through get to play the game?

No, the API server does not only drive registration. If we were to somehow mitigate impact on registration, then the impact caused by users logging into the game would bring it down instead. This would be a lot more certain, as authentication makes 4 times the requests as registration. Registration was also not ready until close to the deadline. This is mainly due to the DOS attack to the wordpress mentioned before.

So what is the plan?

We will perform further load and api testing, look into the root causes of load issues and the server stability. The current plan is to rewrite the API server using ASP.NET instead of Nancy, as it is a lot more “battle-tested”. This could take some amount of time.

We cannot announce a retry date. We might even need a new launch plan altogther, to avoid the load of a few thousand users wanting to play the game on day one. Whatever the plan is, it will be posted on here eventually.

The first city the official FreeSO server will feature is the evolution of Test Center, Sunrise Crater!

citynew

I originally planned to reveal this on December 25th, but I got a little carried away with the changes and town layouts. The city is now full of small towns and interesting features for communities to spring up around. Each part of the city has been intricately detailed by hand, from within the client, rather than just whipping up some graphics in a standard editor. Thanks to this, the difference is so staggering you might be surprised that this is still using the same landscaping system as the original.

sizzle

Thanks to the changes to lot background rendering, road behaviour and smooth zoom, your lot is really a part of the world now. All of your buildings will populate the small town areas within the city, and define their character. Eventually, these towns will become formally defined as neighbourhoods, and you’ll be able to define the way your district is headed by being elected by your peers. This is all far future though! Got to get those core features in first. 😉

A few incremental saves I made of the city building process. The vertex colour has been changed to the final in advance, though.

That’s all for now. On January 6th, you’ll get to see it in person, and set up shop wherever you please. When you join on the first day, make sure to stop by the City Hall on the top of the mountain, for our launch ceremony!

Thanks,
Rhys.

⚽

Hello folks!

It has been a very long journey. Started on February 16th 2014, the SimAntics & World Development thread was the first attempt at re-implementing the complex Virtual Machine behind both The Sims and The Sims Online. 17 months later on July 28th 2015, we ran our first online playtest, where everyone appeared as Bob Newbie, the server crashed every 5 minutes and many tables were stood within. Now, 17 months later again, for New Year 2017, we’re finally launching the game in the large scale MMO context that we’ve been working towards this whole time.

city
You will be able to own a house in the city, run a store, skills lot, money lot, service lot, nightclub or whatever you want to create! Breach the top 100 with a fantastic lot, sell your in-game skills and real ones, work your way to the top of the career track with friends or just have fun, there will be a lot to do off the bat.

With new features and objects from the start, the future of the game is looking brighter than ever. The max lot size is now a huge 64×64 with 5 floors, and you can now see your neighbours around your lot. Compliment each other’s views or go to war with your neighbours, or just make a 5 floor castle in the middle of nowhere. As the past few years have proved, I’ll be there to listen to any feedback, quickly fix bugs that pop up, and lead us through the beta period to a full launch. Of course, the launch will fix a ton of bugs and introduce many welcome changes too!

job

For more information, check out the FAQ here. You can ask more questions in the forums and I’ll put the answers on there, so check back frequently!

In October, members of the forum voted for which city they wanted to live in first. After 3 rounds, you chose Test Center… however, the city has changed a lot over the past few years, so who knows what will be different! All will be revealed on December 25th, so stay tuned.

secretcity

This has been a long time coming and I’m glad I can finally do this. There’s a small chance that this date might be missed – if that’s the case then you’ll be able to find out here first. There will be more information on the exact time the server will go live and what you need to download closer to the date.

That’s all for now. See you ingame!
Rhys.

🐸
🐸

Hello again! It’s been a while since our last post, and a lot has changed. First of all, we have completely ditched Bass.NET for a custom solution using only Monogame (like the rest of our audio) and a decoding library called Mp3Sharp. This lets us play mp3 music through the same channels as our sound effects, meaning, you guessed it, speakers ingame (and robot job) can now play game music! No longer are you stuck dancing to silence.

Last month, the simulator underwent one of the largest and most important changes it’s faced recently – the addition of Permissions and checking them.

Giving Maxis our spare key.

But of course, that’s not all. Alongside many quality of life changes, such as the property list and two person object interaction fixes, we’ve also fully implemented sim budgets in a way that will transfer nicely to the global server.

TSO specific game features have always been a priority, since they have not been replicated anywhere else. One such feature, “plugins/EODs”, allows an object to display an interface on the sim’s local machine, and allows it to connect to an “EOD server”, hosted within the object in question. The biggest TSO objects, signs, the nightclub job and pizza, use this functionality extensively to provide interfaces for gameplay simply not feasible within SimAntics alone. The challenge was to implement this “server in a server” system in a way that could easily be extended for all ~20 plugins in the game. As of a few days ago, we were successful!

By far the most useful plugin, the dance floor plugin.

Signs and the Dance Floor plugin, but that’s not all…

Playing Pizza on the night of release!

However, implementing these three does not mean an easy ride for the remaining ones! Each EOD has a custom server which can be complicated (it especially was for pizza! 8 different simantics events, and ~7 freeso ones) and requires a backing implementation for its UI on the live mode panel. Most of the work is done by the UIScripts, however all functionality has to be implemented by hand… unlike SimAntics, where the simulator drives the object functionality from the game scripts!

dancer2
The Signs Interface. Features 4 different modes, one of much enables a tab to set permissions.
dancer
The Dance Floor interface, featuring a cool 25 buttons to queue custom interactions via the Dance Floor Controller.
The pizza interface, which must closely follow the state of the object, who is connected to it and the decisions they make.
The pizza interface, which must closely follow the state of the object, who is connected to it and the decisions they make.

We’ve come pretty far in the past year. It’s still hard to believe that the first online playtest was less than a year ago! You’ll also notice that our Road To Live Release thread only has a few select items left… Crikey, I wonder what this means? 😉

Thank you everyone for following development, and playing what we have so far has definitely been a lot of fun. Great things are to come!

Rhys.

Hello folks, it’s been a while since I posted an update. A number of things have happened since the last post, so I’ll go over them quite quickly.

Firstly, a lot of work has been put into our resource editing tool, Volcanic. It is now possible to make, test and debug new objects entirely from scratch within FreeSO! This is the culmination of 6 months work; you are now able to fully modify and create most resource types that TSO uses.

Rocket in Volcanic's DGRP editor.

Many object interactions have been fixed – specifically the food counters, beds, campfire and bubble blower. There are a lot of small changes, but the gist of it is that now there are very few objects that do not work correctly. The only remaining main SimAntics VM features are permissions, budget, and the “plugins” system that drove things like “Pizza” and the sign posts. After these, we can get started on the MMO server.

…and of course (due to popular demand), grass and other terrain types are now visible. Here are some screenshots of objects users created with Volcanic as part of its user evaluation, grass included:

We also have a new text logo! This was designed with guidance from some of the members of the community, and resembles more of a modern sims feel than an old one. This was really just to make it go with the icon I already created.

freeso logo

You may have noticed that the forums are currently down. Don’t worry, we haven’t gone anywhere! They should be back up soon, but while you’re waiting here’s a little teaser for what I’ve been working on. 😉

For all you anti-forums folk, a lot has changed since the last update. The change suit primitive now works correctly, TV sounds are half working, and most importantly the game can now serialize and deserialize entire game states, avoiding the huge wait times people were seeing when connecting to servers.

Chat bubbles and chat history have been added, a ton of bugs have been fixed, and work has begun on a full Independent Development Environment for SimAntics to let users create their own objects! (and us modify existing ones 😉 ) There will be more on this soon, once it’s in a stable state.

hats

Rather interestingly, a Brazilian community has recently started developing for the game, and they’re dedicated enough to have already translated parts of the game with the rather shoddy multilingual string support! Full translation of all objects is a little while away though (the IDE will provide this), and sprite font text rendering makes it rather hard to translate into certain languages. We’ll be working on this in the coming months, stay tuned!

br

To better support the objects and environments present in the original game, we had to switch routing algorithm to one which supported more accurate routing without incurring a huge cost. This combination of a rectangular cover algorithm and A* is performant, fit for purpose and seems to be quite similar to the way the original game does things.

As well as this, a large number of related changes have been made to routing, as well as smaller changes to SimAntics which fix objects such as the Whacketball set, computers and job carpool. Things should generally look smoother and work a little better, but I encourage users to look for regressions and bugs in the new code that I haven’t found on my own!

Of course, the branch contains a lot of extra changes too. For more information check the development roadmap thread on the forums and the pull request on GitHub.

 

Hopefully this was worth the wait!

Rhys

Hey all, the forums are currently down temporarily. Should be back up soon, it’s just the same database problem we had during the first playtest. You can keep track of all immediate developments on GitHub and the build bot primarily, but the new routing is nearing completion, at which point I’ll write a full blog about it. We’ll probably do a similar thing all the way to release, with detailed blog posts about all the important merges into the master branch detailing what specifically has been achieved, usually accompanied by a youtube video.

The new routing is looking phenomenal, but you might need a comparison video to show just how much better. This will come in a few days, directly after the merge. The roadmap is available on the forums, which I’m still going to be following semi-strictly, so next we’ll be clearing up the suit primitive and handling. Hope you guys are OK with barbie doll style naked sims, since the censor effect will not be immediately available. 😉

EDIT: Things are back to normal now. The roadmap I was talking about is here: http://forum.freeso.org/threads/road-to-live-release.801/