The History of Hypergrid Directories: A Single Point of Failure

This article is not intended to be overly critical of those various people who have generously put their time and effort into providing directories of Hypergrid locations, which must have been very considerable. The phrase “single point of failure” is a technical description and not a criticism or judgement. It is no small feat to put the server infrastructure and programming time into making this work and maintaining it over a period of time. As one who has operated servers, I understand that the personal circumstances of such volunteers change and resources such as time and finance for servers can be limited.

By way of comparison, a number of the original OpenSim developers have moved onto new projects, yet OpenSim is still releasing new versions. Why? Because there is not, in computing terms, a single point of failure that will cause things to fall over if those individuals or servers can no longer contribute. All of these directories depend on a single, privately maintained database even where the code itself has been open sourced, e.g. with code on GitHub developed by kidd piko (Zvi Ben Yaakov). This has recently had more downtime than ever before and appears to be unmaintained at present, for whatever reasons. [Ed.: in fact, I discover from kidd that the service needs a database sync but would then otherwise be functional, although the web site is not currently available.] Unfortunately, since Kidd put vast amounts of time into this development, all that fantastic effort now seems to be wasted [Ed.: but the situation is now under discussion since this was written.] because nobody else has put a system in place using this open source codebase. It was the most famous and widely used Hypergrid teleport network with its iconic Stargate styled gates.

Stargate out of order

Stargate out of order in Ocean Grid, January 2016. Will it work again? With existing or new code?

It was not the first, which was, defunct since about early 2013, if my records are correct. It seemed to survive a while but it was never clear how the database was being maintained or by whom. Most systems rely on the manual addition of regions either by an administrator or by grid owners, which may in some cases, e.g. Hyperica, be vetted by the grid owner.

Lately, iDreamsNet ceased to function, which was an easy to use system based on rezzing an item and then proceeding to a web page to register the region. Like Hyperica, it did not have an easy-to-use in-world directory, only one on the Web, but it was a great effort all the same.

Defunct iDreamsNet beacon

Defunct iDreamsNet beacon

The defunct iDreamsNet seems to have inspired the latest open source effort, OpenSimWorld, which is a brilliant combination of simple registration on the same principle as iDreamsNet with an easy-to-use in-world and out-of-world directory, like TheHyperGates except for the lack of the iconic Stargates – although there is nothing stopping you putting the HUD code in a Stargate and it will work in a similar way as its predecessor. (Maybe I will do this if TheHyperGates doesn’t come back on line soon. [Ed.: I have now used a modified HUD script in a telephone box because I wanted to keep the Stargate for its original purpose if at all possible.]) Being a HUD, the intention is that you wear it, which is another useful addition to the previous approach taken by TheHyperGates; however, you can just as easily rezz the script in an in-world object, or use the script both in a HUD and an in-world object too. You can only have one beacon script per region but the HUD script is not limited in the same way.

OpenSimWorld also shows you how many people are currently on line and orders the results accordingly: great for finding parties! I used the code from GitHub (which produces a prim like the one below, which I have framed in a sign with an added banner above) but you can also get a different design of beacon from the OpenSimWorld region on OSGrid. Finding people has hitherto been a major problem in OpenSim (and often in SL). It reported 30-40 people at any one time at LBSA Plaza on OSGrid (probably the most famous location in the OpenSim Hypergrid) over New Year 2016. Happy New Year, everybody!

OpenSimWorld directory on Ocean Grid

OpenSimWorld directory on Ocean Grid, Saturday 2 January 2016, 14:30 GMT

Clearly, these systems have been built on each other’s successes and evolved productively. The difficulty is that the database remains the single point of failure. If a single site goes down, the directory goes out of action entirely. I suggest that the next development would be to build a federated system with copies of the database that update each other, so if one breaks there can be a seamless (or at least simple) switchover to an alternative principle database, where perhaps the system records, e.g. in a notecard, the locations of active databases that are communicating with each other, i.e. that the administrator has registered with others, whether or not they are working, and then falls back to a different one and alerts the administrator if the main database stops working temporarily or permanently. One similar federated system in the system of PGP key servers, although that is only a general comparison of course.

It should be possible to re-use the open source code of either TheHyperGates or OpenSimWorld and add a federated database system underneath. It might even be possible to create a database standard that any and all systems could query with only relatively small changes to the code, making the user interface and underlying server code independent of each other, with a modular approach.

In the meantime, OpenSimWorld looks like a major success and I really hope it keeps going in the future.


Security in OpenSim

There are, confusingly, working options for using secure connections to OpenSim but these are related to remote admin and other web interfaces, not to the more fundamental issue of securely logging into the simulator in the first place. (I will avoid the usual misuse of the term “SSL” here, since this has long ago been obsoleted by the more modern TLS, but in practice these terms are used interchangeably.)

OpenSim logins are done over an insecure HTTP-only connection. There is no reason that they could not be done over a secure connection but the code has not been written made functional. It would be quite a simple job. Grid administrators would need TLS (“SSL”) certificates in the same way as they mostly also use for the front-end web sites representing their grid online or whatever else they run securely on their server. It’s not a big deal to do.

The viewer sends an MD5 hash of the password, not the clear-text password, to the server. This, however, is not secure. The reason that MD5 is still used rather than changing to the much more secure SHA2 is that the entire password table for a every OpenSim grid would have to be changed. I wonder if the table could be modified by adding a column noting whether MD5 were used or, if available, the password had been migrated to SHA2? This however, would need coordination between viewer developers and OpenSim core developers, as the viewer would need to transmit the information to OpenSim grids. In practice, this is unlikely to happen. It would also require front-end software i.e. web GUIs to be changed.

Admittedly, MD5 uses salt so that rainbow tables can’t be used, but MD5 is still susceptible to collision attacks and passwords can be sniffed and recovered trivially in a matter of seconds. What other server technology, other than OpenSim, transmits MD5 hashed passwords in clear text? Answer: nobody! It’s not at all safe and HTTPS is a basic requirement for all server platforms. If this was in place, the MD5 hash would be transmitted using strong encryption anyway, so it wouldn’t matter any more that we were still using MD5 internally because only the sender and the server would see it. Ok, SHA2 would be ideal but there we are – at least things would be basically secure.

Here is the config section in OpenSim.ini that you need to look at:

; ssl config: Experimental! The auto https config only really works definately on windows XP now
; you need a Cert Request/Signed pair installed in the MY store with the CN specified below
; you can use https on other platforms, but you'll need to configure the httpapi yourself for now
http_listener_ssl = false ; Also create a SSL server
http_listener_cn = "localhost" ; Use the cert with the common name
http_listener_sslport = 9001 ; Use this port for SSL connections
http_listener_ssl_cert = "" ; Currently unused, but will be used for OSHttpServer

; HTTPS for "Out of band" management applications such as the remote
; admin module
; Create https_listener = "True" will create a listener on the port
; specified. Provide the path to your server certificate along with it's
; password
; https_listener = False
; Set our listener to this port
; https_port = 0
; Path to X509 certificate
; cert_path = "path/to/cert.p12"
; Password for cert
; cert_pass = "password"

The confusion here is that there are two separate “SSL” configurations here. One is for “out of band” connections like web interfaces and remote admin, i.e. https_listener, which comes second here. This does work (also see Robust{.HG}.ini etc). But it’s the first, http_listener_ssl, that is relevant here. This does not work. It’s for a code stub that hasn’t been completed and, if you try to use it, the region will never start successfully. The code in question is: [INSTALL_FOLDER]/opensim/OpenSim/Server/Base/HttpServerBase.cs

What is needed is for this code to be completed. It also needs the addition of an option to force logins over secure HTTP connections, as the existing options would only add HTTPS logins rather than replace them over HTTP if it worked as intended. For security, you need to actually prevent people from sending passwords in clear text only, i.e. by adding force_https_login = true in order to redirect all login requests to HTTPS, not simply enable optional HTTPS for the login URI. It might be worth also having a setting for force_http_traffic = false as default but allowing a setting of force_https_traffic = true for better security where sufficiently fast connections allow all traffic to be sent over HTTPS  – this would obviously affect only HTTP(S) and not UDP traffic to and from the viewer. Note that Second Life™ does use HTTPS for its login URI, and this has been true for many years. What is stopping OpenSim from developing this very simple, industry-standard security?

Editor’s note 2014-12-30: despite Melanie’s comment “… one really should not trust grids that don’t offer a https:// loginuri”, the only grids currently appearing to offer such URIs are Second Life™ (and test grids) and Avination. These have the wherewithal to develop and deploy solutions beyond what is available in the current OpenSim release code. While Melanie’s position is ideal, it is also currently impossible, either because the code simply doesn’t work or else because how to deploy it is undocumented and obscure to the point that nobody does it in practice. It seems likely that the former is true, given that the present writer and no doubt many other grid administrators have tried and failed to make it work. I also doubt Melanie’s assertion (from 2012) that MD5 is sufficiently difficult to break in practice: by 2014, it appears now to be trivial to crack MD5 hashes – corrections welcome from anybody with greater expertise in encryption hashes etc.