Configuring secure HTTPS ports with TLS (“SSL”) on OpenSim

Background

It has already been remarked in an earlier post that it is now possible to configure the simulator to use secure ports on OpenSim with TLS (still erroneously but commonly known by the name of its defunct predecessor SSL), i.e. that you can run it on HTTPS instead of HTTP. However, there is no native ability within OpenSim to do this for the ROBUST services, including most critically the login service. This means that your password details are broadcast in plain text at the time of login. Obviously this is far from ideal.

In order to provide secure ports it is necessary to configure a reverse proxy, of which the most obvious and commonly suggested by OpenSim administrators is Nginx, which also happens to be the best all-round modern HTTP(S) server and an improvement (in my opinion) on the venerable but battle-tested and dependable Apache. (See here for a discussion, in which it is stated that OSGrid uses Nginx.) For all this, there appears to be no published advice on how to achieve any of this. It seems no surprise that few – if any? – publicly available OpenSim servers on the Hypergrid seem to be configured to use HTTPS ports with TLS. It is fiddly, so this is worth providing here.

Most other server software in live installations is configured to use secure ports as an industry standard – why should OpenSim not follow the same standards? Otherwise I would suggest that you can never store personal data in OpenSim and you can never be sure of not being hacked. It’s fairly unlikely that an OpenSim grid would be an attractive target for a hacker, being somewhat niche, but it is entirely possible and no doubt has happened. (Famously, SL was hacked with flying penises.)

Secure ports 9002 and 9003 on Ocean Grid

Ocean Grid is now an example of just such a grid. In fact, for reasons that we shall come to later, it remains possible to log into Ocean Grid using the insecure HTTP port 8002. However, as of 2 May 2020 it is now also possible to use the secure HTTPS port 9002. Furthermore, the traffic to the private port is also configured on 9003 instead of 8003 for reasons specific to Ocean Grid that you may not need to bother with: if your internal port does not need to be open to the Internet then only your subnet will be able to see that traffic anyway, so why add the (fairly small) overhead of encryption?

First of all I began by configuring a second instance of ROBUST. Ocean Grid now runs duplicate copies of all the services. I have never separated, for example, the asset server from the others and I have not done so now either, but this can be done too. This would use Nginx as a reverse proxy for its other purpose, namely to load balance between duplicate copies of services, whether the whole lot of them in one ROBUST instance as I have done or on a service-by-service basis. The obvious one to do this with is the asset server. It depends on the ancient Tiny Web Server, written in C#, which is very slow and probably ought to be replaced – I’d suggest with h2o because it is fast and can do HTTP/2 and (experimentally so far) the new HTTP/3 (formerly QUIC) over UDP. So far I have configured no load balancing for Ocean Grid. One large 1280×1280 test region/simulator (Rheged) communicates by HTTPS while the other four regions (again, all on separate simulators to prevent out-of-memory crashes) all still use the original HTTP port. This of course may change in future.

In case you are wondering how I have different regions on different simulators, I have adapted Gwyn Llewelyn’s brilliant instructions on how to run multiple instances concurrently. Thanks, Gwyn ❤ (Be still, my virtual heart!) I also use some scripts that use GNU Screen to automatically switch between the console interfaces of the regions/simulators and ROBUST.

The upshot of all this is that all the services are available by either port. You can log in locally or via Hypergrid by either and reach the same simulators with no apparent difference in performance between the two. In fact, removing a region/simulator that is 5×5 larger than a normal region (i.e. 25 times as big) to a separate asset service has considerably improved performance. Otherwise there are only four 256×256 traditional regions in Ocean Grid to date.

It would of course be possible to attach other people’s regions on other servers to Ocean Grid, although this has not ever been done to date because it is a small private grid with mainly just myself, Starflower Bracken, as the principal user. I get a small number of Hypergrid visitors and a tiny number of visits from old friends of 2007 vintage who I met years ago in SL and have registered accounts locally on my grid. This would add a little more impetus to use HTTPS for security reasons, as the internal services then need to be open to the public Internet and there are more targets to attack. In any case, the same principles apply about best security practices as outlined above.

Before moving on to the mechanics of the installation, it suffices to add that there will be further tweaks as time goes on, probably including the load balancing described above in order to provide a much-needed boost to performance, especially as regards load on the asset server.

Problem with https:// prefix in the viewer

One particular tweak, migrating entirely to HTTPS and removing the insecure ports, will have to be has been postponed for an unknown period of some time. The grid selector in the viewer has always previously contained a bug [FIRE-24068 fixed 2020-11-07, SV-2392 fixed at unknown date], since it was first added to the original viewer for OpenSim purposes, that adds the http:// prefix to any loginURI and makes it impossible to add https:// instead. The only workaround to this annoying problem is to manually edit grids.user.xml (or grids.xml in some viewers). On doing this, HTTPS logins work perfectly, including using free Let’s Encrypt certificates as are installed on Ocean Grid. But less experienced users may not know how to edit this XML file, so any total migration would shut them out of logging in locally to the grid. Now that the major viewers allow this, it may make sense to migrate to the secure port, which is currently under review. You may ask, who cares since only I log in locally for the most part, while other visitors come via Hypergrid. In my case I have one reasonably frequent visitor who logs in with a locally registered account, who happens to be a very welcome one. I don’t really want to put a barrier in the way of the few local visits that I do still receive or force people to come via Hypergrid. If you do visit me via Hypergrid then I urge you to use port 9002 and thus to do so securely!

NOTE 2021-09-15, edited 2021-09-26: It is now possible to use Firestorm and Singularity with secure ports. (Singularity has a recent OpenSim version from 2020 despite being listed as an inactive viewer or in an “unknown” state – what is the status of this viewer? I was unable to run Dayturn 1.8.10 or 1.8.9 on Windows 10 as it simply failed to start for me, so I have not been able to test it. The VL Cool Viewer also supports HTTPS (although it will let you add the grid with the secure port OR the insecure port but not both under different names!) Radegast (text-only viewer) supports HTTPS. SceneGate (viewer with reduced functionality) supports HTTPS and for bonus points gives a security warning when connecting to insecure HTTP ports. This seems like a much more healthy position than described above. If Dayturn allows secure ports, there appears to be little good reason not to migrate entirely to secure ports, which I am seriously considering.

Setting up the ROBUST server with HTTPS

First of all you need to configure and run a second copy of ROBUST. I copied Robust.HG.ini to a new file Robust.HG.tls.ini and then edited that. You will use -inifile=”$DIR/Robust.HG.tls.ini” where you can replace $DIR with the path to wherever you keep the file, if you don’t use the method that I use.

Then you need to edit the following in blue, adapting it as appropriate for your purposes:

[Const]
BaseURL = “https://oceangrid.net”
PublicPort = “8102″
; The proxied external public port of the Robust server
PublicPort2 = “9002”
PrivatePort = “8103″

[ServiceList]
VoiceConnector = “9004/OpenSim.Server.Handlers.dll:FreeswitchServerConnector”

[Hypergrid]
HomeURI = “${Const|BaseURL}:${Const|PublicPort2}”
GatekeeperURI = “${Const|BaseURL}:${Const|PublicPort2}”

[LoginService]
MapTileURL = “${Const|BaseURL}:${Const|PublicPort2}/”;
SearchURL = “${Const|BaseURL}:${Const|PublicPort2}/”;
SRV_HomeURI = “${Const|BaseURL}:${Const|PublicPort2}”
SRV_InventoryServerURI = “${Const|BaseURL}:${Const|PublicPort2}”
SRV_AssetServerURI = “${Const|BaseURL}:${Const|PublicPort2}”
SRV_ProfileServerURI = “${Const|BaseURL}:${Const|PublicPort2}”
SRV_FriendsServerURI = “${Const|BaseURL}:${Const|PublicPort2}”
SRV_IMServerURI = “${Const|BaseURL}:${Const|PublicPort2}”
SRV_GroupsServerURI = “${Const|BaseURL}:${Const|PublicPort2}”

[GridInfoService]
login = ${Const|BaseURL}:${Const|PublicPort2}/
gridname = “Ocean Grid TLS
gridnick = “Ocean Grid TLS
gatekeeper = ${Const|BaseURL}:${Const|PublicPort2}/
uas = ${Const|BaseURL}:${Const|PublicPort2}/

In the case of OpenSim, there are cascading copies of OpenSim.ini in order to override certain settings for each region/simulator and inherit the rest of the settings from the main file. You may not have that setup, of course. However, this is what I did for the copy for my region Rheged:

[Const]
BaseHostname = “oceangrid.net”
BaseURL = https://${Const|BaseHostname}
PublicPort = “9002″
PrivURL = https://localhost
PrivatePort = “9003″
;; the following line was for testing – ignore
;PrivatePort = “8103”

Step required only if you are using port 9003 HTTPS on localhost

There is also another step required for making the certificates for localhost (or 127.0.0.1) work if, like me, you are making the internal services function over HTTPS. There are better ways to do this and this is a bit of a hack but bear in mind that the loss in security is fairly inconsequential since it is only on the internal subnet with only trusted devices connecting. As noted above, you don’t even need to encrypt these if they are never accessed from the public Internet but in my case they are because I have an external proxy server as well, which provides a roundabout solution to the lack of NAT loopback (NAT hairpinning) on the rather rubbish router that my ISP has provided. A better solution would be to get a decent router (and preferably a fixed IP address, as I used to have, rather than Dynamic DNS) and then I could just keep port 8003 internal and dispense with 9003. (I could alternatively use the /etc/hosts file on any other machine on the network running a viewer to connect with the local grid.) As also noted, another possible reason to expose these ports to the Internet is if external simulators are connecting.

Anyway, here is what I did:

[Startup]
; #
; # SSL certificates validation options
; #

; SSL certificate validation options
; you can allow selfsigned certificates or no official CA with next option set to true
; NoVerifyCertChain = true
NoVerifyCertChain = true
; you can also bypass the hostname or domain verification
; NoVerifyCertHostname = true
NoVerifyCertHostname = true
; having both options true does provide encryption but with low security
; set both true if you don’t care to use SSL, they are needed to contact regions or grids that do use it.

Setting up the simulator to use HTTPS (separate, optional step)

Again, for completeness, another step that is strictly not necessary is to give it a secure port. I actually did this some time ago quite, separately from all of this, for just the simulator rather than ROBUST:

[Network]
http_listener_port = 9106 ; if http_listener_ssl = true then this will be ignored

;; THIS IS EXPERIMENTAL IN OPENSIM 0.9.1.0
; ssl config: Experimental!
http_listener_ssl = true ; if set to true main server is replaced by a ssl one
http_listener_sslport = 9206 ; Use this port for SSL connections
; currently if using ssl, regions ExternalHostName must the the same and equal to http_listener_cn
; this will change is future
http_listener_cn = “oceangrid.net”
; if the cert doesnt have a official CA or is selfsigned viewers option NoVerifySSLCert need to be set true
http_listener_cert_path = “/opt/opensim/bin/certs/oceangrid.p12” ; path for the cert file that is valid for the ExternalHostName
http_listener_cert_pass = “your_password_here” ; the cert password

; the handler is in [opensim]/OpenSim/Server/Base/HttpServerBase.cs
http_listener_ssl_cert = “” ; as of 2019-10-11 this is not fetched by [opensim]/OpenSim/Framework/NetworkServersInfo.cs
; but the next undocumented lines are fetched, though who knows what, if anything, is done with them since it doesn’t
; work and you get the error “Error – certificate Parameter name: ‘certificate’ cannot be null.”

Nginx reverse proxy configuration

The next step is to provide the Nginx reverse proxy configuration as follows (tidied up a bit!) in a file called something like /etc/nginx/sites-available/oceangrid.net-9002-9003 as follows:

server {
listen 9002 ssl http2;
listen [::]:9002 ssl http2;

server_name oceangrid.net http://www.oceangrid.net;

ssl_session_cache shared:SSL:20m;
ssl_session_timeout 10m;

ssl_protocols TLSv1 TLSv1.1 TLSv1.2 TLSv1.3;
#very secure, very compatible
ssl_ciphers ECDH+AESGCM:ECDH+AES256:ECDH+AES128:DH+3DES:!ADH:!AECDH:!MD5;
#highly secure, less compatible
#ssl_ciphers ECDH+AESGCM:ECDH+AES256:ECDH+AES128:!ADH:!AECDH:!MD5;
ssl_prefer_server_ciphers on;

location / {
proxy_pass http://127.0.0.1:8102$1;
#resolver 8.8.8.8; #public Google IPv4 DNS resolver (insecure)
proxy_redirect off;
proxy_buffering off;
proxy_set_header Host $host:$server_port;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
}

location ~ \.php$ {
proxy_pass http://127.0.0.1:8102$1;
#resolver 8.8.8.8; #public Google IPv4 DNS resolver (insecure)
proxy_redirect off;
proxy_buffering off;;
proxy_set_header Host $host:$server_port;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
}

access_log /var/log/nginx/grid.oceangrid.access.log;
error_log /var/log/nginx/grid.oceangrid.error.log;

ssl_certificate /etc/letsencrypt/live/www.oceangrid.net/fullchain.pem; # managed by Certbot
ssl_certificate_key /etc/letsencrypt/live/www.oceangrid.net/privkey.pem; # managed by Certbot

}

server {
listen 9003 ssl http2;
listen [::]:9003 ssl http2;

server_name oceangrid.net http://www.oceangrid.net;

ssl_session_cache shared:SSL:20m;
ssl_session_timeout 10m;

ssl_protocols TLSv1 TLSv1.1 TLSv1.2 TLSv1.3;
#very secure, very compatible
ssl_ciphers ECDH+AESGCM:ECDH+AES256:ECDH+AES128:DH+3DES:!ADH:!AECDH:!MD5;
#highly secure, less compatible
#ssl_ciphers ECDH+AESGCM:ECDH+AES256:ECDH+AES128:!ADH:!AECDH:!MD5;
ssl_prefer_server_ciphers on;

location / {
proxy_pass http://127.0.0.1:8103$1;
#resolver 8.8.8.8; #public Google IPv4 DNS resolver (insecure)
proxy_redirect off;
proxy_buffering off;
proxy_set_header Host $host:$server_port;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
}

location ~ \.php$ {
proxy_pass http://127.0.0.1:8103$1;
#resolver 8.8.8.8; #public Google IPv4 DNS resolver (insecure)
proxy_redirect off;
proxy_buffering off;
proxy_set_header Host $host:$server_port;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
}

access_log /var/log/nginx/grid.oceangrid.access.log;
error_log /var/log/nginx/grid.oceangrid.error.log;

ssl_certificate /etc/letsencrypt/live/www.oceangrid.net/fullchain.pem; # managed by Certbot
ssl_certificate_key /etc/letsencrypt/live/www.oceangrid.net/privkey.pem; # managed by Certbot

}

Finally make another copy of this file, for example called /etc/nginx/sites-available/localhost-9002-9002 but comment out (#) the lines that are your equivalent of server_name oceangrid.net; near the top. These will then work for localhost (127.0.0.1) on those ports. You will now need to type:

sudo ln -s /etc/nginx/sites-available/localhost-9002-9002 /etc/nginx/sites-enabled/localhost-9002-9003
sudo ln -s /etc/nginx/sites-available/oceangrid.net-9002-9003 /etc/nginx/sites-enabled/oceangrid.net-9002-9003
sudo systemctl restart nginx
OR
sudo service nginx restart

Here you have it. As discussed above, you won’t need to do the localhost part if 8003 is firewalled on your local network, in which case you will not be using 9003 and those sections can be ignored here in both files.

Bear in mind that the resolver setting may be different on different servers and I must leave you to discover the best and safest way to do this on your own configuration – don’t use an easily guessable public DNS resolver like the one commented out above because of the risk of interception.

The future of HTTPS in the open metaverse

You could adapt and simply these instructions easily enough for a standalone installation of OpenSim but, since I don’t use one, unless somebody contacts me for advice then I shall leave it to them to do so because I don’t see it should be too horribly difficult based on the instructions for a grid. Most people’s requirements will be simpler than mine but my grid has evolved over almost 10 years.

I hope that people use these instructions. I am asking people to adopt 9002 and 9003 for these secure ports, i.e. to establish a convention that is easily guessable like the present 8002 and 8003 convention. Please also add your voice to the call to fix the grid selector in the various viewers [FIRE-24068SV-2392]. I am looking forward to a more secure open metaverse in the future and hope I have contributed to that.

In-world web browsing in OpenSim #1

Prototype in-world browser in OpenSim grid

Prototype in-world browser in OpenSim grid

Web browsing in OpenSim

Web browsing inside OpenSim (or even Second Life™, for those still left there) is never going to be as seamless as through a browser. You might ask, “What’s the point?” but, in fact, there is a point in certain limited circumstances, i.e. when a number of people want to look at the same resources together in a virtual environment. The 3D metaverse isn’t about replicating real life at all but rather being able to meet people in a simulated physical environment in order to share the same space. It is an immersive discussion forum and therefore one can easily imagine wanting to show people things on the Web and discuss them within that immersive environment.

Media On A Prim doesn’t do the job

Why don’t I just use Media On A Prim (MOAP)? The reason that I am not using Media On A Prim (MOAP) is because (a) it seems to be broken in OpenSim at present, so every time you click on a link it reverts to the previous page straight away so you cannot navigate; (b) the texture does not immediately render without user intervention and it does not look the same for every viewer. Despite it being technically better in supporting Flash, audio, video and native scrolling (if it wasn’t apparently broken), it isn’t ideal for collaborative use. My method uses the OsDynamicTextureURL* functions, which ensure that all viewers see the same texture.

My image-based solution

Aside from MOAP, this has been done before but there is apparently no open source code available nor any advice on how it was done. Well, realising this made me want to find a way to replicate it and provide exactly that. I have now made a mostly functional prototype web browser inside OpenSim. It relies ultimately on WebKit through PhantomJS, itself described as a “headless WebKit scriptable with a JavaScript API”, i.e. a set of tools that can test web pages without a visible browser and, amongst other things, render the output into image files. However, manipulating the relatively arcane directives of PhantomJS directly proved difficult, so I naturally turned to CasperJS as suggested, which is “an open source navigation scripting & testing utility written in Javascript” in order to make PhantomJS easier to deploy. Although it’s not altogether different and there is still a level of complexity, especially for someone like me who only crudely hacks JavaScript, it does live up to its aim: it makes it considerably easier to use PhantomJS for its intended purpose.

The LSL/OSSL script communicates by HTTP POST with a PHP script on our server, which requests an image of the web site from the JavaScript, which processes clicks on the surface coordinates that it was sent to the links in the page and renders the resulting page as an image.

The JavaScript

Let’s call this imagecapture.js (see the code below) and put it on a server somewhere. The location does not have to be public once it’s out of testing – in fact, if you want the secret key to remain secret, it had better not be so. We should make this HTTP request from OpenSim using TLS i.e HTTPS, still often incorrectly known as SSL, it’s predecessor in cryptography. That way, we are not transmitting the secret key in clear text. Effectively, we are producing a secure image-based proxy.

There are instructions elsewhere on the Web for how to install PhantomJS and CasperJS, which I won’t repeat – not least because it will depend upon your *nix distribution. I found it reasonably easy to install PhantomJS using apt-get or aptitude on Ubuntu. It was mildly harder to install CasperJS on Ubuntu 14.04 but a little research should accomplish it easily.

There are some things that you will not be able to do, like plugins showing live video or audio of any kind. But you can do this in other ways in OpenSim already. Some JavaScript links will not work, although those using onClick="location.href='http://example.org/'", for example, will work fine. Normal anchor links with or without images will work fine too. In order to make anything work at all, I must tell you now that you will need access to a web server and know how to use it. Here is the JavaScript, to start with:

var casper = require('casper').create({
  pageSettings: {
    javascriptEnabled: true,
    loadImages: true,
    loadPlugins: true
  }
});

var webPage = require('webpage');
var page = webPage.create();

var args = require('system').args;
var startURL = args[4];

var width = 1024; // 1024;
var height = 768;
var zoom = 1; // DO NOT CHANGE THIS

casper.options.viewportSize = {width: width, height: height};

x = parseFloat(args[5]);
y = parseFloat(args[6]);

x = Math.round(x * width);
y = Math.round(y * height);

casper.start().zoom(zoom).thenOpen(startURL);

// code to click x, y
casper.then(function() {
  this.mouse.click(x, y); // clicks at coordinates x, y

  documentHeight = this.evaluate(function() {
    return __utils__.getDocumentHeight();
  });

});

var address = startURL;

var clipHeight = casper.evaluate(function(){
  return document.querySelector('body').offsetHeight;
});

casper.then(function() {
  address = this.getCurrentUrl();
  console.log(address);
  this.capture('cache/image.png', {
    top: 0,
    left: 0,
    width: width, // clipWidth,
    height: documentHeight // clipHeight
  });
});

casper.run();

This code is basically my own and is released under the GNU General Public Licence (GPL) 3.0 for public use. It has been created with the considerable help of the CasperJS and PhantomJS documentation. While not complete, this is pretty helpful and I have to take off my hat to the developers, who are clearly very clever and able people.

The PHP script

It won’t run without some PHP, which I have also written and release under the same licence. Change the “secret key”: may I suggest using SHA2 for a secret hash? Mine is just a single letter for testing, so far – terribly secure! Let’s call this imagecapture.php and put it on the same server.

<?php
$secretKey="l"; // change this after testing
if ( isset($_POST['secretKey']) && $_POST['secretKey'] == $secretKey ) {
  if ( isset($_POST['searchURL']) ) {
  $searchURL = stripslashes($_POST['searchURL']);
  if ( isset($_POST['x']) ) {
    $searchURL .= ' ' . $_POST['x'];
    if ( isset($_POST['y']) ) { $searchURL .= ' ' . $_POST['y']; }
  }
  $execString = '/usr/bin/casperjs --ignore-ssl-errors=yes ./imagecapture.js ' . $searchURL;
  //$execString = '/usr/bin/phantomjs --ssl-protocol=any ./imagecreate.js ' . $searchURL; // Old version before I used CasperJS
  exec($execString, $output, $return_var);
  echo addslashes('imageURL').'='.addslashes('http://oceangrid.net/imagecapture/cache/image.png'); //$imageURL;
  echo '&'.addslashes('returnURL').'='.addslashes($output[0]);
  list($width, $height, $type, $attr) = getimagesize("cache/image.png");
  echo '&'.addslashes('width').'='.addslashes($width);
  echo '&'.addslashes('height').'='.addslashes($height);
  }
}
?>

Put it all together

Once this is out of testing, adjust the location of the above imagecapture.js file and change the location in this script to reflect this. You ought to make sure that these are owned by the Apache or Nginx user (usually www-data), which will mean they will not write the file image.png when you run them from the command line. When we call them later via HTTP request to the web server from within OpenSim, however, they will do so properly.

This is all obviously quite snazzy, though it does have limitations like lack of audio or video. It has no Flash.  Audio can be done in other ways but MOAP isn’t optimal as described, but perhaps at some point these could be somehow resolved in one neat package. Until then, this fills a gap. Simple JavaScript links will work but most fancier bits won’t, so expect some broken links etc.

Now all we have to do is create a web browser in OpenSim. As we speak, I am about 75% through this process. I have to add scroll bars, back and forward functionality and do a fair bit of checking. However, the basic thing works. If you want to see it in its incomplete state, please get in touch with me personally and I will tell you where in my grid Ocean Grid you can find it. Once it is complete, I will then release a full version in IAR format as well as the various LSL/OSSL scripts that are required.

This is the end of part one. So far, we have created a way by which a set of HTTP POST variables can be sent to a PHP script, which in turn calls some JavaScript that writes an image of the web page and returns the URL (which may have changed e.g. via 301 redirects on the web server). The PHP then returns all of this stuff in POST data to the requesting script inside OpenSim.