Unified Three-Dimensional Windowing System (Strata System)

This document details the architectural foundation of our proprietary Strata System, a Unified Three-Dimensional Windowing System (UTDWS). The Strata System successfully addresses the fundamental limitations that previously hindered the development of general-purpose 3D user interfaces: the necessary lack of robust hardware abstraction and the architectural inability to support multiple, disparate 3D interface contexts simultaneously within a single immersive spatial environment.

Our Strata System achieves these goals by extending the proven architectural paradigm of the 2D windowing system into the immersive, three-dimensional realm, utilizing the Wayland display server protocol as its foundational layer.

This visual demonstrates the Strata System’s unified 3D windowing capability. The scene contains multiple windows, each showcasing different content types:

• The first window displays a flat image, illustrating standard media embedding.
• The second window contains a fully interactive 3D cube, demonstrating spatially anchored, volumetric UI content.
• The third window acts as a portal - inside its bounded “aperture” you see another 3D region, within which floats another 3D cube. This highlights Strata’s support for advanced window types, such as portals into nested or remote spatial contexts.

Together, these elements show how the Strata architecture enables arbitrary spatial arrangement, occlusion, and composition of heterogeneous 2D/3D content - going far beyond traditional desktop windowing by supporting true 3D interfaces in a unified immersive environment.

---

1. The Core Problem: Architectural Fragmentation

Historically, immersive 3D interface development has faced two substantial challenges that prevent modularity:

1. Device Abstraction Deficit: High-quality 3D interface devices (including 3D input like 6DOF sensors and immersive displays like HMDs) demand custom integration with every application. This requires developers to maintain dedicated code for each hardware API, seriously limiting application support.
2. Lack of Coherent Space Sharing: Sharing the 3D interface space between multiple applications is challenging because current graphics infrastructure and 2D display technology underlying immersive displays are designed for a single application to handle the custom projection (e.g., stereo viewports and distortion correction). Meaningful division of the 3D interface space requires dividing it in a 3D way.

2. The Architectural Solution: The Strata Compositor on Wayland

The Strata System abstracts 3D interface hardware and manages the composite 3D scene at the operating system level, creating a unified 3D interface space.

The core capability of the Strata System is the extension of the traditional 2D window concept to a 3D window (or 3D interface context), which represents a 3D region within the immersive interface space.

2.1 Wayland Selection and Advantages

The UTDWS leverages the Wayland display server protocol because its architecture unifies the display server and the compositor into the same entity. This is vital for spatial compositing because it grants the compositor centralized control over:

1. The arbitrary embedding and transformation of window output in 3D space.
2. The transformation of 3D input events back into the correct window’s coordinate system.

2.2 Spatial Interface Context Definition (Strata Volumes)

Clients define their interactive region within the global 3D space through specialized interface contexts. This interface context is referred to as a Strata Volume:

• Strata Enclosure: Defines a box-shaped, spatially bounded region where the client draws its 3D content.
• Strata Aperture: Represents an unbounded client space whose content is only visible through a defined bounded 2D opening, functioning as a portal connecting two 3D spaces.

3. Hardware Abstraction and Custom Input Processing

The Strata System is designed to abstract diverse 3D hardware behind standardized input primitives.

3.1 6DOF and Sensor Abstraction

Input hardware (including proprietary 6DOF sensors, accelerometers, and gyroscopes) is abstracted behind standardized input primitives. This provides continuous spatial tracking throughout the 3D space, forming a 3D analog to traditional mouse input. New hardware only requires a driver for the windowing system abstraction to achieve support from all applications using the system.

The system provides architectural support for high-level interaction metaphors:

• Pinch-to-Grab and Move: This action maps onto the system’s abstract 3D input events derived from continuous changes in the 6DOF transform.
• Finger Tap to Click: This symbolic input is delivered as a 3D analog of a button event, attached to the current 6DOF spatial transform.

4. Implementation Details: Strata Protocol Extensions

The Strata System defines specific extensions to the standard Wayland protocol. These extensions provide the 3D windowing services to clients.

Interface	Wayland Protocol Extension Role	Core Functionality
Strata Fabric	Advertises 3D windowing service support to clients.	Allows clients to acquire a Strata Volume from an existing wl_surface.
Strata Volume	Represents the client’s 3D interface context (the 3D Window).	Allows the client to request a specific context type (Enclosure or Aperture) and dictates the buffer size for necessary projections.
Strata Projection	Represents a virtual camera/eye in the 3D space.	Sends synchronized View Matrix and Projection Matrix updates to the client before rendering the frame.

4.1 View and Projection Matrix Synchronization

The Strata Projection interface manages the View Matrix (camera position/orientation) and the Projection Matrix (optical properties/field of view). These matrices control essential depth cues like Motion Parallax and Stereopsis.

To maintain a consistent 3D environment, the compositor sends up-to-date matrices to all 3D clients before each frame is drawn. For stereo displays, the compositor provides separate matrices for each viewpoint (eye), ensuring every client application renders its 3D content from the exact perspective the user is currently occupying.

5. Advanced Compositing Techniques for Unified 3D Space

The Strata System must compose 3D content from multiple sources, even though clients only submit 2D images (buffers).

5.1 Depth Compositing and Occlusion (Strata Layering)

To preserve the correct occlusion order (which geometry is in front), the system coordinates depth information.

1. Client Depth Submission: Clients submit their 2D rendered image and their corresponding Depth Buffer contents to the compositor.
2. Per-Pixel Depth Test: The compositor performs a traditional Depth Test on a per-pixel basis. This uses the clients’ depth data to guarantee that geometry from separate 3D contexts correctly overlaps and integrates into a single, cohesive 3D scene.

5.2 Strata Volume Clipping

The Strata System enforces the spatial boundaries of each Strata Volume. Fragments outside the client’s designated 3D region must be discarded.

• The compositor defines the precise 3D volume (the “full clipping surface”) for the client’s requested context type (e.g., Cuboid or Portal).
• This surface is written into the stencil buffer to enable the stencil test, which dictates that fragments (pixels) outside the bounds are discarded during the compositing pass.

5.3 EGL Depth Buffer Extraction Mechanism (Critical Workaround)

A significant implementation challenge arises because the Wayland EGL interface currently does not allow the compositor direct access to the client’s default depth buffer.

• Client Side Maneuver: Clients must pack their 32-bit depth data into a 4-vector using a technique taken from and write this packed data into a double-height color buffer.
• Compositor Side Extraction: The compositor accesses this single color buffer, extracts (unpacks) the depth data, and writes it into its own scratch depth buffer for composition.

This procedure, while computationally intensive and introducing significant rendering overhead (two extra passes), is mandatory for enabling the required depth compositing needed for a spatially coherent environment. Resolving this efficiently requires EGL Depth Buffer Extensions to the Wayland EGL implementation itself.

Key Innovations & Impact

The Strata System demonstrates that spatial computing doesn’t require throwing away decades of windowing system wisdom. By extending proven 2D paradigms into 3D space, we created a foundation for:

• Universal 3D Application Support: Hardware abstraction enables any app to become spatially aware
• Multi-Application 3D Environments: True sharing of immersive spaces between applications
• Natural 3D Interaction: Gesture and 6DOF input as first-class citizens
• Future-Proof Architecture: New hardware integrates seamlessly through standardized primitives

This work established fundamental patterns for spatial computing interfaces, proving that immersive 3D environments can be as modular and extensible as traditional desktop systems.