Introduction
With the rapid development of Augmented / Virtual and Mixed reality technologies, Metaverse software architectures are becoming widely available and artist friendly nowadays. Here are some of the noticeable XR applications for making sound art running on desktop OS: Patch XR, Korg Gadget VR, The Music Room. Meta is developing a web version of its Horizon Worlds. Open worlds for Web XR, like Mozilla Hubs have become suitable for collaborative listening of music materials and holding live coding sessions. Being an open source project, Mozilla Hubs can be extended by using A-Frame component framework, which contains plenty of ready to use components.
Recursive portal in Krestianstvo SDK using SolidJS / Fiber
But today, almost all well known creative software for making web-based sound art are using the well studied application-centric approach. A noticeable examples are: P5.js with it’s outstanding library, Msx/MSP RNBO and WebPD with functionality of exporting patches to Web Audio, Soul audio language and infrastructure for writing and deploying audio code. Sonic PI is also planned to run on the web using the Elixir programming language. Much of the projects are developed by using vanilla JS and ToneJS library are also examples of application-centric approach. Thus in most cases an artist uses some kind of domain specific livecoding programming language on top of JS with the set of Web Audio / Visual libraries. And the final project is generally distributed in the form of an application or a website hosted on the Internet.
On the contrary, the Metaverse software approach is based on using fully immersive self explorative environments and virtual worlds foundations (not necessarily 3D) with the spirit of “Matrix” and “Cyberpunk” artworks. And as Web browser is becoming more stronger today, glad to low-level Web Assembly and WebXR standards, we can expect of appearing lot of web sound art projects based on these foundations. For sure, all existed web sound frameworks and applications will be heavily used behind the scenes and can get a new vision being integrated into virtual environments.
The Foundations of Virtual Worlds
Several devices are running one shared Virtual World with physics
Generally, virtual worlds software development is associated with computer games and simulations. In that context the term Virtual is superficially pointing to some sort of a predefined scenario with limited action rules, that developers & users are allowed to operate on. Thus, considering this as reducing freedom in the development process, a lot of projects prefer a classical tools based approach. Especially since the current Web standards provide a variety of frameworks for audio, visual, network programming, developers individually form their own stack of technologies. That classical approach works quite well for a single user mode applications. But when there is a need to make collaborative or networked multi-user applications, the use of a classical approach leads a developer to operate in terms of web sockets, realtime databases, streams etc. implementing network codes. And this is not always a trivial task, lying too far from the web sound art.
On the contrary, Virtual Worlds software architectures isolate low-level network stuff from the development process, in a such way that an artist never touch net code at all. The term Virtual starts shining with the notion of embodiment. So, that every user is representated with an Avatar, operating in a united shared Time & Space spreading across a mesh network. This feature of Virtual Worlds turns into a noticeable advantage compared to the classic approach. It encapsulates all the internals and difficulties of working with networks in web applications. It reminds us something, that is encapsulated in the Web browser using Virtual approach already. If we compare JavaScript to the general purpose programming languages like C++, we will find out that JavaScript Virtual Machine encapsulates the low level Memory out of the programming scope. That allows developers to build Web applications using just prototypes, objects and thier properties. In that context, the term Virtual points to the programming language itself or any other hosted domain-specific language, applications can be coded on.
Reminding the history of the JavaScript language, we should take note of its roots. There are Lisp, Smalltalk and Self programming languages lying behind the scenes. But what JavaScript VM dismissed from those is an image based distribution (Virtual Memory) and a message passing based Virtual Machine. Both features are still crucial for implementation of the Virtual World software architectures inside Web Browser. Gload to the functional programming paradigm and modern ES6 standard, JavaScript VM allows to implement these things natively in JavaScript or transpile them from other programming languages like Haskell, Clojure, Scala or Smalltalk.
Virtual Time & Space
Virtual Time is the crucial concept of Virtual Worlds architecture. Croquet project have came closer to the implementiion of the virtual time in the Metaverse then anyone else. There are several implementations of Croquet from open source versions based on Squeak Smalltalk and JavaScript (Virtual World Framework / Krestianstvo) to closed sourced Croquet OS - the Operating System for the Metaverse. Generally, all versions provide a synchronization system for multi-user Metaverse experiences. It allows multiple users to work or play together within a single shared distributed environment, and it guarantees that this distributed environment will remain bit-identical for every user. It defines Virtual Time by looking at objects as a stream of messages that leads to deterministic computations on every connected node in a decentralised network.
Recently Electric Clojure has made a unification of continuous-time and discrete-time primitives under the common Flow protocol. It uses correct incremental maintenance of dynamic DAGs (directed acyclic graph) without inconsistent states (aka glitches in functional-reactive programming). Electric is a reactive and network-aware Clojure/Script DSL that fully abstracts over client/server state sync at the programming language layer, in order to achieve strong composition across the frontend/backend boundary in dynamic web apps. There is no client/server dichotomy from the programmer’s perspective. Croquet and Electric provides a notion of shared VM, that is distributed across the network.
So, having such correctly defined Virtual Time & Space, we can operate with the paradigm of Portals instead of Hyperkinks on a web application. In simple words Portals are the Web Hyperlinks, in a such way that they should be. Here is an portal prototype, where every client entering this collaborative World is interacting with a shared World in a Multiverse way. Being represented with 100 Yourselves (Avatars) in 10 World replicas running in 10 isolated shared Time & Spaces, reflecting on each other.
It is like, as you open 10 new Web Browser windows or connect 10 new Devices with QR-code, and is manipulating them all simultaneously. In this example, it is running on one Device and in one Browser, just as one pure function.
Here is one more example of recursive portals, where the Avatar is observing itself.
These worlds are built with Krestianstvo SDK - open source implementation of the Croquet application architecture in Functional Reactive Paradigm withing SolidJS. Using just Signals, Effects and reactive computations, a JS application gets the original scene of Croquet’s Objects with Message passing.
You can play with portals by yourself in the playground Krestianstvo Toolbox
Real world examples with Web sound
Example 1
Here is an example tutorial of using Croquet OS with React JS and WebSound. Even though the Croquet OS is closed sourced, its Microverse is open sourced. It allows users to create rich web applications and use it for free for small-scale projects.
React Music Box A simple multiplayer music box with realtime view side update. The app has a number of “balls”, each of which represents the timing and pitch of a note. A participant can manipulate them to compose a loop. The timing for the bar to wrap is synchronized by the model side logic, but the view interpolates the position of the bar and plays a sound when the bar passes a ball.
Croquet OS tutorial: React Music Box
Example 2
Here is another example of using now LiveCoding.space as an open source implementation of Croquet application architecture in WebXR with A-Frame components and ToneJS library.
Musical world with ToneJS
Example 3
| The project This is not a concert by Delia Ramos Rodríguez and Nikolai Suslov was developed and presented during the Festival of Web-based Music | BROWSER 2021. |
Several devices are running art project
The art work was presented in the form of an interactive, multi-user, collaborative p2p web application. The application can be run on any desktop or mobile Web Browser. The audience collaboratively explores the artwork inside a virtual canvas space within multi-contextual / conceptual creative layers by touching the virtual objects. These layers are not visible by default. Interaction is based on applying or viewing through some sort of “filters” (augmenting reality in virtual reality). Several participants can personally or collaboratively explore the hidden layers, as well as experimenting with the artwork through these layers without breaking the original artwork.
Example 4
In this example Virtual World shows an orchestration of an external not native web applications, like Sonic Pi. The control of an applications here is carried out by sending OSC messages, sinchronised within shared Virtual World.
Conclusion and Discussions
While developing web sound art projects using web frameworks as is, a programmer or an artist still thinks in terms of libraries, tools, operation system, network etc. This repeats the same life-cycle of an application creation for a desktop OS. Considering Virtual Worlds as an alternative paradigm for developing modern Web applications will hugely simplify things. Self explorative live coding environment can replace a debugger, virtual time can encapsulate network related stuff, Avatar can mediate human-computer interaction, Meta-languages negate the knowledge of the programming language for the end-users. Wether a Web application being built with the Virtual World paradigm will fully satisfy the needs of an artist on live performing and interacting with an audience is an open question for further discussions.
To Top ↑