Updates, Error 23 Obituary, and Current Roadmap
Since the neighbourhoods update has been out for over a month and people have become settled in their new town halls, I figured it was about time for an update post on what we’ve been doing… and what we will be! Yes, this post will contain a full roadmap on what we are focusing on bringing to the game before leaving beta. Yes, that is still a thing. Feel free to skim any of the sections and just read headings, as many of them get quite technical.
(If you are still concerned about the contents of the last post, I suggest looking at the post date.)
Automatic Reconnection (anti-error 23 tactics)
Back in the olden days, the internet was a very unreliable medium. Transfer speed was low, cross country signal drops were frequent, and everything was just poor in general. Nowadays everything “just works”, and you can guarantee that most connections will stay connected for large periods of time. Most.
Assuming we were past the “extended suffering” period of the internet, FreeSO’s core networking was built with the assumption that when a connection is created, it is only destroyed when the other user quits out of the property/city or their connection drops entirely. It’s predecessor from the original TSO, “Aries”, had an additional feature for handling a third situation – disconnecting due to a temporary line fault, but being able to re-establish connection shortly afterwards via a slightly different connection handshake.
The internet is essentially a huge network with many different routes connecting points around the world. Essentially, depending on your global position, you would have a different route to our server. There’s no guarantee that all of these routes have 100% uptime, or behave nicely with long-lasting connections, so occasionally connections ended up dropping/terminating to some users more than others. This was what was causing “Error 23”, where the user lost connection to the City Server, or simply being booted from the property without warning. This disconnection was instant, and immediately destroyed the sim’s session, booting them from the property they were on.
To solve these issues, I first made this session live on for 30 seconds after the initial disconnection. Given the user has the right authentication available, they can then retry to re-establish connection and regain control of that session, without closing it. This lets users who drop connections or switch networks re-establish connection to the game without dropping out of the property or city! The networking on top of all of this is on another layer of abstraction, so it should all work the same – handling disconnects is just deferred until we know that we definitely cannot reconnect.
An interesting side effect of this is that you can now switch between different internet connections without dropping out of a property, such as a WiFi connection to a 4G one. This is only becoming more important as more devices are able to connect to cellular networks.
This is still work in progress, of course. In some cases the game attempts to reconnect in the wrong situations (a mistake that doesn’t do much) and there are very rare cases where re-connection could still fail entirely. It’s definitely many steps above the instant guaranteed drop we had before, which is useful when you’re doing something long term like a game of Band, Pizza or deep into a… future EOD.
Incremental Updates and New Updater
Since the start of beta, FreeSO updates have been distributed in a really simple way – when you log on, the server tells you the version it’s on and sends you a link to the zipped client with that version. If the version mismatches your client, then you are prompted to download it. When it was downloaded, a simple patcher application simply extracted all of the files in the zip, and reported any hiccups it ran into along the way. Unfortunately, when it hit these hiccups (usually due to access being denied to Program Files or multiple open clients) it usually ended up breaking the client, requiring a redownload via the launcher.
The process of creating this client zip was entirely manual. We had to download the build from our buildbot (servo) manually, add in extra files such as the splash screen, objects and catalog xml, re-zip it all and then upload it to a webserver. This whole process took a lot of manual time, and not all of the moderation team knew how to do it. A more recent problem came from the fact that users updating to the latest version were downloading the full update zip, which recently has been exceeding 120MB.
Updating the server has also been a lot of trouble – to minimize impact on players, we usually end up updating when the server restarts at its usual non-peak time at 9:30 GMT. This involves stopping the server during its restart, uploading a preprepared server zip like the client one mentioned above (this takes time to make) and then extracting it over the server files, making sure some config files are not overwritten. The server files are usually backed up in case a config file is wiped, the database is handled separately. The update download URL had to be updated in the config, and some more things had to be performed manually to make sure clients didn’t log in with the wrong version (before it auto-updated).
All of these things have been discouraging small bugfix updates, as the cost for us and the users is too high and I always have to guide the process, essentially. It’s due to this that I decided to rethink the entire update process, to make it easier to develop the game and cause less trouble when we fix issues or introduce new features.
If you felt sorry for us reading that previous paragraph, be assured that things are definitely much better now. For a start, ALL of these steps are now automated. The generation and upload of client and server zips from servo are automated via a “branch” configuration, and queued to apply after any given date on a server restart. This already saves a lot of work (and dependence on me to push the update), as now you can simply click a button to get the process started. This has greatly sped up generating updates for internal testing too, which helps catch bugs before they affect Sunrise Crater.
The real magic behind this automated generation is the ability to generate Incremental Updates. If you’ll remember, the regular client update is essentially the entire FreeSO client zipped up and downloaded every time, regardless of whether you had the game installed already. Our incremental updates are zips that only contain files that have changed from the last update, reducing the size of the update dramatically. For reference, a recent full update was about 126MB, and its incremental update from the previous version was 6MB. These also have a manifest with hashes of the old/new versions of files, and any files that are deleted.
This huge reduction in filesize is good for a lot of things – the bandwidth of client and server, the time taken to update and the number of files where something can go wrong. When you are a few versions behind, we can still use these incremental updates to save space. Say you are 3 versions behind – the full redownload may be around 128MB, but you can get to the current version by downloading 3 incrementals instead (6MBx3 = 18MB). Given that most things in the FreeSO client do not change between updates (existing objects, patches), this incremental update size will always be very small.
On the user side (what you’ll see), you’ll probably already have seen the new incremental updater that has been used for the most recent updates. It’s a lot more fleshed out than our last updater, which simply displayed the file it was currently writing and alerted on any errors. Now it can extract multiple files (eg. multiple incremental updates), shows progress visually, continue to extract files if one or two cannot extract, and rollback the entire process if something goes wrong. This prevents a ton of cases where the game would end up corrupted and unbootable.
If something goes terribly wrong, you switch between cities that use different versions, or you reinstall the game, you can still download the full update at any time. Note that the patcher is now supported on osx (rather than our hacky script solution), though we’re still working out some issues there.
Thanks to these changes, we’re able to be much more confident about pushing updates without worries that people will be locked out of the game, removing some anxiety from development. We look to further this more in future with runtime object/patch updates for live development and emergency fixes, which could lead into a CC platform in the future.
(note: we have also added an auto patcher for TSO version to the game, in case the user installs the wrong version of TSO when attempting to use FSO)
Phew! With all of that out of the way, It’s time to spill all the beans. We’re closer to release than ever before – it’s time to stop the feature creep and get down our core set of features. Here is what will come to FreeSO before it leaves Beta:
2D Mode Refactor (Currently in Progress)
If you have a powerful computer, have been on a few object heavy lots (sparkling’s is an easy choice) and have played in 2D AND 3D, you’ll notice that 2D mode actually ends up running a lot worse than 3D, with more stutters and slowdown than you’d expect from a 2D game. This is because 3D mode is intensive on your Graphics Card, and 2D is intensive on your CPU. 3D mode makes enough sense because it has more complex shaders and more triangles, but 2D mode’s shortcomings need a bit more explanation.
The way 2D mode works right now, it has two sprite modes: “STATIC” and “DYNAMIC”. Objects that don’t change often are lumped in “STATIC”, and objects that do are kept in “DYNAMIC”. When a “STATIC” object changes its graphic or moves, it hops into the “DYNAMIC” buffer, which causes the static buffer to be regenerated as the object is no longer there. Buffer regenerations like this are also caused when a wall or floor is changed, or even when you mouse over a wall to cause it to cut away. When changes do not occur, however, the static buffer does not need to be regenerated (or even redrawn in most cases), saving a lot of cpu and gpu power over 3D mode.
The problem is, right now any single change will regenerate the entire buffer – even elements which are far away from the change, are the same or use a different set of geometry. You’ll find that attempting to place flooring can lead to extreme slowdown, lots with many objects and walls stutter heavily when these buffers are regenerated, and the experience simply isn’t great on older PCs that can’t regenerate them quite as fast. The “DYNAMIC” buffer is no better – it could be caching sprite vertices for individual sprites in the buffer… instead it regenerates all of them every frame.
The goal is to get a lot maxed out running well on a laptop CPU, mostly through changes preventing data from being regenerated when it will be the same anyways. This may include other changes such as simplifying the grass shader, optimizing lighting etc – some of which have sliced 11.5% off of cpu time in a current dev build for 2D AND 3D. Similarly, optimizations to things like roofs, terrain and floors will apply to 3D as well, as these elements are shared between both modes. When the update drops, the goal is for the changes to be big enough that you can really feel the difference.
3D Hybrid Mode, 3D shader changes and Dynamic Textures
I’m also looking into adding an additional mode called “3D Hybrid” mode, which will allow us to display 3d-only objects within 2D mode. This will allow us to do a few things with custom objects that sprites typically limit us from doing, such as applying dynamic textures (eg. cc images on a poster), having objects with a very large number of states, or even animations. With correct pre-baked AO (and anti aliasing enabled from options), these should blend in pretty well with the other 2D mode objects, but should still only be used when required.
The focus on 3D hybrid should also work the other way – allowing for using 3D mode graphics to still play in 2D isometric, but with smooth rotation and better zooming.
With these changes I will also be looking into including a few more features for our 3D remeshes – such as specular lighting/cubemaps and perhaps normal maps. These would help us better represent objects with really shiny 2D sprites – which right now we simply fake by baking specular into the diffuse texture. I’m not saying we’ll have a full PBR solution, but any improvement will be an improvement.
Upgrade System for Objects and Revised Tuning
Tired of the same Moroccan chair in front of every table? Televisions which are good for nothing except decoration? We realise that one of the biggest problems with The Sims Online is not the number of objects available, but the number of which are viable. You shouldn’t be punished for wanting to have a rustic themed property, so what if you could upgrade your crappy furniture to the same stats as the best? This will essentially be done via a huge tuning replacement table, and the goal is to have similar furniture (chairs, beds) share “upgrade levels”, where objects may differ on which level they start from, but generally cost about the same to get to the best. Obviously crafted objects and rares will always be a cut above, but this should make the catalog a lot more interesting.
These won’t just be assigning the existing tuning values to upgrade levels – we will likely change how each object performs at its best, and add upgrades for objects where there is only one version (eg. the bubble blower fun object). These upgrade levels will be shown when purchasing the object, viewing it in buy mode etc. so that they are easy to identify when trading and selling objects.
Improve Skill Gameplay
Recently we introduced a concept called “group skill” with the maze object and wargame. This is a faster way to gain a particular skill that involves more user activity, via the Maze game and the increased motive drain while playing. We will be considering tweaking both of these objects to make their benefits more obvious, and adding more objects of this kind to cover more skills. We will also be looking into the speeds and behaviours of the idle skill objects and decay, to see what we can do about the game becoming boring for users not used to the slowed down pace of TSO gameplay.
Note that past beta, we will have a skill lock system that rewards active gameplay over sim lifetime. This will be important for the long lifetime we expect the server to have – as as things are right now (many users with max locks) the game stops making much sense.
Improve Server Infrastructure for Distributed Scale
Right now, the FreeSO server runs on the assumption that you’ll only be running one city at once, with the lots and api server running on the same FSO.Server instance. The server has been designed to support scaling in a few ways: multiple lot instance servers for each city server, multiple city servers for each database server, a potentially separate API server… but none of these things have been necessary yet, and do not work as seamlessly as we might like them to. A larger focus on this will future proof any future extensions to the server we might make, such as a second city or a city running a different game version! (similar to Test Center in the original)
Obviously scaling would not be good for our wallets, but it could be inevitable if the game gains a considerably larger number of players. The best approach is to prepare for it regardless!
Dynamic Distribution of New and Modified Objects Without Updates (Stretch)
This is more of a pipe dream, but over the course of beta there have been a few times where we have had to update the game just to add a few objects/patches, or fix a game-breaking bug with one. These updates didn’t change any client code or other binaries – just files that were loaded by the game on startup. But what if they could be dynamically loaded and unloaded AFTER startup, and potentially served from the server on demand, without an update?
Essentially, we would be able to push out “temporary” objects and patches on the server side which would be loaded into the server’s content system immediately. When a lot encounters one of these modified or new objects, it would then tell all of the clients about it, and force them to download any missing or updated files for the new object, then load it into their game. This is obviously a simplification, but it would allow live development and testing of new objects in a test server, and deployment of critical fixes or event objects without forcing an update.
This could also lead into a Custom Content platform in the future, where CC objects would be uploaded to the server and served out only to users who encounter them in properties. We’d have different rulesets for visual only changes (on top of the upgrades system mentioned above… you see? it all ties together :angel:), patched script changes and full new objects that would allow placing and using these objects without downloading ALL of them, but while minimizing any hitches that would come from doing so.
Over the course of these updates, we will be doing several bug rounds where we will squash some remaining bugs and small issues that have been around for a long time. The priority list is as follows:
- Pets (free will misbehaving, rules around running away + email for it, occasionally teleporting out of world or becoming unusable)
- Restaurant Bugs (Alignment of certain job lots, serving large groups of people breaking, round activity tracking)
- Top 100 Avatars, Relationship Decay (top 100 needs relationship decay to stop its query locking the db for 100 seconds)
- Chat Channels (fix bugs, make tab behaviour/hiding more obvious)
- Various UI bugs and crashes
- Minor simulator bugs, eg. erratic playback of simulation ticks in some cases.
Don’t worry if I missed any – during these bug rounds I will actively find out what player’s biggest issues are and target them directly. I’ll let you know when I’m doing one!
All the features listed here are just what I’ll be working on – you might still be surprised by some additional features, objects and events from the rest of our team in the coming months! Who knows? 👀
To keep better track of this roadmap, I’ll probably set up issues for it and a milestone on Github, alongside the split between master/develop/feature branches I’ve been planning.
FreeSO has always been a hobby project, so it has always been hard to predict how much time would be spent implementing a feature – and it has been especially hard predicting when we were going to leave beta. As always, thank you for being patient – hopefully you won’t have to be for much longer!