XRC AI

XRC AI is package used for prototyping with foundation models in XR applications, enabling the use generative AI and large language models.

XRC AI is a Unity (UPM) package, part of XR Collaboratory’s XRC Toolkit. XRC AI enables the integration of LLMs from multiple providers, such as OpenAI, as well as locally hosted models via Ollama. XRC AI uses the Microsoft Agent Framework (.NET) and abstractions from the Microsoft.Extensions.AI libraries.

View the XRC AI package: xrcollaboratory .github.io/edu.cornell.xrc.toolkit.ai

Features

XRC AI allows a simple setup, where a client holds the LLM and an agent is defined independently of the client. The package includes UI Toolkit integration for visualizing messages. Tool calls can be visualized in the UI.

Multi-agent workflows and orchestrations are implemented via the Microsoft Agent Framework, with patterns such Sequential, Concurrent, and Handoff already implemented and other patterns to be implemented. Image data can be handled in parallel with text input. Middleware is used extensively to wire in custom behavior, such as updating UI Toolkit on callbacks.

Experiments

The XRC AI package is in active development, and will be embedded in several projects at the XR Collaboratory throughout 2026. Below are glimpses of current XRC AI experiments and work-in-progress

Adding a multimodal agentic interface to Axis.
Work-in-progress 2026.

Visualizing agent orchestration patterns with physical objects, using XRC AI package and ROS 2.
Work-in-progress 2026.

XRC AI

Credits

Project lead, design, engineering
Harald Haraldsson

Timeline
2025 - present