Recently, my private grid (running OpenSim 0.7.0.2-post-fixes) has been having some major problems with the standard physics engine ODE (the Open Dynamics Engine) throwing exceptions due to segment faults when running on Mono 2.6.7. This happened after Mono was upgraded when I incautiously decided to upgrade Ubuntu 10.04 to 10.10 without first checking whether the bundled new Mono version would interfere with OpenSim. It’s not easy to downgrade since various bundled parts of Ubuntu 10.10 rely on this version of Mono. Currently, I’m not quite sure if I have managed to solve it and, if so, how, but the problems are currently levelling off have now inexplicably disappeared again. It largely happens with hollow or cut-path prims, usually large ones over 10m, and is was entirely reproducible. I tried using ulimit -s 262144 but I’m wondering if this was set in the same child shell as the GNU Screen session in which OpenSim runs, as it seemed to have little effect until I did it manually. The crashes have temporarily abated, but I am no closer to finding out whether it is anything to do with ulimit and the stack size.

It did focus my mind on the ODE physics engine, however. Readers may know that Second Life ™ uses the free binary distribution of Havok 4, noting that the proprietary source code however is not freely available as open source. It can’t be distributed for commercial purposes (likely to be a problem for some commercial grids based on OpenSim) and the OpenSim developers can’t modify it for use with OpenSim without access to the source code. There are several other physics engines packaged with OpenSim. These are: Basic Physics, BulletDotNET BulletX, ODE, Modifed BulletX, POS, RealPhysX. [2011-10-04] only ODE and BulletSPlugin are now functional. Since at one point I could barely keep an avatar moving near my build for a few seconds at a time without crashing the region server, I decided to try these out. The initial results were not at all encouraging:

Basic Physics – this works fine but, as it sounds, is basic: in fact, so much so that it does not support collisions. Flight is jerky and all prims are phantom, so buildings or any prim structures are basically pointless. It may have some use for debugging, although the jerky flight made it awkward to use for building.

BulletDotNET – as my grid runs Linux, it wasn’t possible to test this, as it only runs on Windows and .NET rather than Linux and Mono. [defunct]

BulletX – it has long been recommended that Modified BulletX is used to replace this under OpenSim, so I did not test it directly and don’t know if it is still directly supported as a separate option. [defunct]

ODE – largely fine, minus proper vehicle physics and one or two minor glitches. But if my experience and that of several others is typical, use caution for the time being if you intend to upgrade to Mono 2.6.7 in the near future, which appears to the most likely source of the recent spate of crashes. [2011-10-04] Note that Mono 2.6.x and 2.10.x but NOT 2.8.x work with the OpenSim 0.7.2 Release Candidate according to the release notes, but I suggest that you do not use above 2.4.3 with OpenSim 0.7.1.x versions. According to this Mantis issue, the crash problems should have been sorted out, so please comment here or re-open the issue there if you experience problems similar to those I described.

Modifed BulletX – disappointing, this is no longer functional in OpenSim. Although it is possible to start a simulator using this physics engine, it appears to process every prim in every region before it is possible to connect, which takes a long time. At the end of this process, the avatar is unable to move from around 100m in flight without being bounced slowly back into that position, and other movement is likewise impossible. It’s thus difficult for the average user to test whether physics works for other objects and collisions. [defunct]

POS – billed as Basic Physics with collisions, at least it is possible to start a simulator and connect to it, although avatar movement does not work at all, unlike in Basic Physics. It appears to be obsolete. It was not possible to test it any further; in any case, it never supported much functionality in the first place. [defunct]

RealPhysX – while this may be a useful physics engine too, it is no longer functional with OpenSim, which simply crashes on start up. [defunct]

[2011-10-04] BulletSPlugin was added in June by Robert Adams, as noted at the top of this post.

So, whether or not the more advanced physics engines listed above (e.g. BulletDotNET, BulletX, Modified BulletX, RealPhysX) are better or worse than ODE is currently impossible to tell inside OpenSim running on Linux, and I’ll leave it to others running Windows to discuss BulletDotNET.

However, I did discover that there are various other physics engines available for gaming applications, of which ODE is only one. The problem is that it would require a new physics API to be written for OpenSim in C#, since none of these are currently supported. Although OpenSim is technically modular enough to allow this, it has never yet been done. There are a number of other physics engines available, as I gleaned from this discussion, from which this is the relevant excerpt:

Box2D <http://www.adrianboeing.com/pal/engines.html#box2d>
>> (experimental)
>> – Bullet <http://www.adrianboeing.com/pal/engines.html#bullet>
>> – Dynamechs<http://www.adrianboeing.com/pal/engines.html#dynamechs>
>> (deprecated)
>> – Havok <http://www.adrianboeing.com/pal/engines.html#havok>
>> (experimental)
>> – IBDS <http://www.adrianboeing.com/pal/engines.html#ibds>
>> (experimental)
>> – JigLib <http://www.adrianboeing.com/pal/engines.html#jig>
>> – Meqon <http://www.adrianboeing.com/pal/engines.html#meqon>
>> (deprecated)
>> – Newton <http://www.adrianboeing.com/pal/engines.html#newton>
>> – ODE <http://www.adrianboeing.com/pal/engines.html#ode>
>> – OpenTissue <http://www.adrianboeing.com/pal/engines.html#ot>
>> – PhysX (a.k.a Novodex, Ageia PhysX, nVidia PhysX)<http://www.adrianboeing.com/pal/engines.html#novodex>
>> – Simple Physics Engine<http://www.adrianboeing.com/pal/engines.html#spe>
>>
>> – Tokamak <http://www.adrianboeing.com/pal/engines.html#tokamak>
>> – TrueAxis <http://www.adrianboeing.com/pal/engines.html#trueaxis>

I’m not entirely sure for how many of these the source code is available as fully open source software, or whether there are commercial restrictions on some of them, so I’ll leave it to the reader to follow up on this point. There are instructions for using Newton physics with OpenSim dating from 2008, and these may still work although I have not tested them. This is an open source N-body gravity physics engine for astrophysics that is not aimed primarily at gaming physics and collisions and is apparently NOT the same as Newton Game Dynamics, a freely available proprietary, closed source physics engine for gaming. Obviously, OpenSim and SL employ gaming physics.

The really interesting possibility for physics is the Physics Abstraction Layer (PAL), which OpenSim developers such as Adam Frisby and Teravus Ovares have been keeping an eye on for some years. If suitable developers could be found and the logistical issues overcome, it would enable more or less any advanced physics engine to be used with OpenSim. I wonder if any developer types with more knowledge than me would like to comment on the feasibility of this? It does seem that ODE physics development for OpenSim hasn’t moved on much lately, so perhaps it is not a current priority?

The new Aurora distribution of OpenSim is working on a modified version of ODE that apparently includes vehicle and underwater physics. I did notice that, when testing this (in early alpha, to be fair), once or twice I ended up propelled upwards and left standing on a vertical block of air about 20 feet tall. But no doubt such glitches will be ironed out: it is impressive that they are taking on the fearsome work of improving the physics engine, which I doubt is work for the faint-hearted. [2011-10-04] It seems from their web site that many improvements have been made to Aurora, including ODE fixes and enabling HyperGrid.

Since physics is so basic to the experience of 3D worlds, surely it should be more of a priority in order to improve the user experience? At the same time, I know it is a difficult area of technical development. I hope to see advances in this area, given occasional glitches, and my recent bad experiences with ODE crashes. To some users, though not me in particular, vehicle physics in particular is an important missing area of functionality. At present, it seems that we have little alternative to continuing to use ODE with OpenSim for the foreseeable future. [2011-10-04] Although in testing, BulletSPlugin apparently now works, and is said to outperform ODE in some respects. I have not tried it out myself, however.

This is, of course, a demonstration that the world of OpenSim grids, so dependent on the Linden codebase, is still very much semi-detached from development in SL, and interoperability will always remain a core issue.

One reason for this is mesh, coming soon (but we’re still not sure when) to a simulator near you. Look out, OpenSim and ModRex! Despite the latter’s support for mesh being far older than the Linden effort, ported back from RealXtend‘s version of the OpenSim codebase to a region module for OpenSim, it has never caught on. Although OpenSim developers seem to be working on mesh, it’s no longer clear if ModRex is the main plank of this effort. It still only works in standalone mode, not in grid mode. The pace of development on ModRex seems still to be incredibly slow, after an initial burst of activity, and it’s blog and web presence remains embryonic and dated.

The 1.x codebase can support some newer backported features such as Display Names (which is not likely to be complex code) but it will be increasingly difficult, and most likely impossible, to continue developing viewers that work for both OpenSim and SL without embracing the viewer 2.0 codebase. Incidentally, Display Names will not work on OpenSim, but apparently will on Aurora based-grids (see below). Hopefully, however, the terrible viewer  design will be completely ignored. Even the developer of Kirsten’s viewer, which rather slavishly avoids any affront to Linden Lab by providing direct support for OpenSim (which is a simple matter of using the open source grid manager code from the Hippo viewer), has been critical of the Viewer 2.0 design. I should say, in Kirsten’s defence, that it is not completely impossible to use the present version of the viewer with OpenSim.

Meanwhile, new forks of the OpenSim server codebase are appearing, notably Aurora, which provides a great deal of core functionality that users have been crying out for. The new Kokua Viewer (a separate project loosely associated with Aurora, previously known as Imprudence) will support some of these extra features. At present, things like profiles, groups, search and web interfaces have to be hacked together once per upgrade, and database changes leave all of these side projects struggling to keep up with the OpenSim codebase.

OpenSim developers, after their huge success with Hypergrid, have managed to undo their own work by fracturing the community into no less than three mutually incompatible and often unreliable versions of what is the most fundamental part of the open metaverse. At present, Hypergrid is barely working at all, and it is a major victory to teleport off one’s own servers. Yesterday I finally managed to reach OSGrid (though it’s misconfigured locally, so that one cannot leave) using a test grid running Aurora, although I cannot do so with any revision of OpenSim 0.7.1-dev on which it is based. Admittedly the latter is development code, but many grids are already running it, including OSGrid. People were astonished to see someone arrive from the outside: one said it had been a year since they had known it to be working! Obviously, some of the unreliability is down to local server configurations, which is an operational problem. But why keep breaking Hypergrid with every new release? Why does it have to be so hard? This is no way to help grow the open source metaverse.

It seems that the OpenSim developers do not seem to see the hypergrid as a priority even though it is what makes people compare OpenSim grids to the Web and its rich competitor SL to the once-mighty AOL. At present, documentation and communication about OpenSim remain amateur and patchy. Of course, the developers make the blinkered ideological claim that they are NOT a competitor to SL, but such claims are often made by those who are manifestly failing to capitalise on their obvious strategic advantages. However talented the OpenSim developers are, they are terrible salesmen. And they are convincing nobody. Their user base certainly is competing with SL, even if they personally, as developers, are not. Remember, the user is queen – or even king!

Get it together again! All this fine work needs a bit more coordination, no?

This can only be a good thing. With ModRex waiting in the wings, you can bet that OpenSim will be on the case: in fact, they were ahead of the game on this one. LL realised this, as I said they would, and do not want to be seen to be lagging behind their competitor and spin-off imitator, which already has other significant functionality that SL lacks. To be behind in something so fundamental and obvious could kill SL and send people running for OpenSim. But SL has the advantage of being first at the party, and LL naturally wants to keep it that way rather than surrender to the many new grid providers running the OpenSim software on their servers across the Web.

By the way, OpenSim 0.7 now fully supports the Linden Viewer 2.0 codebase. If, as expected, meshes are only available in the 2.x branch and not the 1.x branch, the third party viewer developers will have to play catch up. But this is something that they have done before with Snowglobe in double quick time, and I would put very little past their ingenuity and determination where it concerns keeping up and surpassing the Linden viewer code. There is also the alternative of adapting the existing GPL code used in the RealXtend viewer that can already display meshes, although this was written for the 1.x codebase.

As I said in the previous post, it is meshes that will allow both SL and OpenSim to continue to compete and outperform the other virtual worlds, to remain modern in terms of attractive, industry standard graphics capability, to provide ever improving content creation, and to attract new users.

(1) Allow objects to be any shape natively, avoiding the halfway house that is sculpted prims. Sculpties appear first as blobs and then take their shape according to an additional texture which acts as a texture map. Instead, mesh objects could be imported from external 3D modelling software and other sources already on the Web.

(2) Allow avatars to be any shape, no longer necessarily relying on the crude controls created by Linden Lab, which survive from its early days. This allows shapes to be imported from external sources, and would allow far more realistic avatars, as well as all sorts of other shapes for avatars, including non-human ones. This is possible using ModRex and RealXtend with OpenSim.

At present, it is possible to use ModRex with OpenSim, but the majority of grids running OpenSim do not. Why? The main reason is that most third-party viewers, e.g. Phoenix (formerly Emerald), Imprudence, Hippo, Ascent, do not support meshes. You would have to download the RealXtend Viewer to do this. There is also an experimental viewer called Naali, which has been designed to avoid using any code from the Linden Viewer and thus avoid the GPL licence. However, Naali does not yet support multiple regions, which means that you can’t teleport out from the sim that you log into. So the best solution for most people remains the RealXtend Viewer at present, which is a heavily modified third-party viewer, based on the standard Linden code.

The problem with RealXtend is that it is only used by a handful of devoted developers and has not achieved any market penetration. There is little point using it with SL, which does not support meshes. Few people use it with OpenSim. Would it not be better to capitalise on the market penetration of popular viewers like Phoenix and Imprudence by enabling meshes in grids that support it? Naturally, this should not affect their performance in SL, or in OpenSim grids that do not enable ModRex. As the RealXtend code is under the GPL like the other third-party viewers based on Linden code, the necessary modifications to the code are already available to and reusable by the Phoenix and Imprudence developers.

Although the third-party viewers support OpenSim grids, it has not by and large been their focus. I’d suggest that they need to wise up to the increasing flow of users to OpenSim grids, which are now viable, stable alternatives to SL, some of which have currency and functioning economies. If the major viewers support ModRex, its development will become more of a priority for the OpenSim developers, and their eventual plan to make it a standard module for OpenSim will be realised quicker. The metaverse would, at a stroke, become far more attractive because of the ability to support modern, games-style graphics that meshes would deliver. For a long time now, SL and OpenSim have been visually old-fashioned, and thus they lose users to otherwise less sophisticated virtual worlds. If OpenSim supported a functionality that SL doesn’t, and yet the viewers that are used to connect to SL can support, there would be a strong impetus to implement it in SL too, in order to remain competitive.

In short, mesh support in Phoenix, Imprudence and other viewers, taken from the GPL RealXtend code, would dramatically increase the attractiveness of OpenSim, and probably SL as soon as the risk of being outdone by the competition drove them to follow suit. One ModRex becomes standard in OpenSim, content will be better, cheaper, and the metaverse will move into a new, more modern era. The revolution in content quality will be greater than what was offered by flexible and sculpted prims put together. While meshes are not available, other platforms will increasingly steal users away from OpenSim and SL. It must therefore be the biggest priority for OpenSim, if our way of life in the 3D metaverse is to develop, thrive and survive.