Oliver Dawkins (Data Stories – Maynooth University) and Gareth W. Young (TRANSMIXR – Trinity College Dublin)

Last week, we had the privilege of presenting a series of XR Masterclasses at Dublin’s BETA Festival. The workshops were designed to help participants explore the possibilities for creating and sharing activist narratives and stories using extended reality (XR) technologies like virtual and augmented reality (VR/AR).

The sessions were proposed by BETA to support their presentation of the immersive augmented reality experience Noire. Noire tells the story of Claudette Colvin, a 15-year-old black girl who refused to give up her seat on a segregated bus in Montgomery, Alabama, one day in March 1955. Until writer Tania de Montaigne retold this story, it had largely been forgotten and overshadowed by a similar encounter involving Rosa Parks nine months later, made famous through the support of Martin Luther King. Noire uses Microsoft Hololens 2 headsets and spatialised sound to restage Claudette’s earlier encounter in holographic form for six simultaneous participants who share that space in mixed reality.

Taking a creative lead from Noire, our workshops invited participants to explore the use of similar technologies to create and share their stories about activist causes. In particular, we focused on demonstrating the potential for new forms of previsualisation and immersive storyboarding using VR headsets (Meta Quest 2) with open-source software (Open Brush), which is free to use and enables users to draw scenes and environments from the inside out. Participants draw or paint a scene around them in three dimensions. In this way, they get an immediate sense of what the scene will feel like when they share it as an immersive experience for others. The tool can be used to quickly sketch ideas in 3D but offers excellent scope for painterly expression. Using Open Brush with the Meta Quest headset’s ‘passthrough’ mode also lets designers test what their creations might look like in mixed reality at a fraction of the cost of the more expensive Hololens headsets.

Each session started with an introduction to Noire and a discussion with their team members. In our Tuesday session, we were joined by Emanuela Righi and Louis Moreau, who discussed the production and technology involved in Noire. Tania De Montaigne joined us for our Wednesday session to discuss narrative and storytelling. After a brief break, we moved to a broader discussion of XR technologies and their use in storytelling with the aid of technologies like volumetric video capture and 360° cameras. While volumetric video is fully spatialized, it is costly to produce and generates unwieldy volumes of data that must be processed. 360° video, by distinction, is cheaper to produce but typically limits movement to three degrees of freedom, with the viewer effectively stuck in a bubble. Both have different affordances with different implications for accessibility, interaction, and immersion, impacting the types of experiences that can be created and how they are produced. The unique characteristics of XR require adjustments to traditional storytelling methods.

We also considered the importance of realism and artifice with reference to the documented experiences of users in VR who have reported great feelings of immersion and empathy even toward 3D animated content, suggesting that these are not as dependent on realism as we might suppose. Hence, immersive media show great potential for engaging creators and users in their capacity to affect and be affected by digital media in performative virtual and mixed-reality environments through which users can enact their imagination in ways that can support empathy and identification with a character or cause. At the same time, creators need to be authentic and take responsibility for the stories they tell. They must engage with their subject fully to ensure its validity and veracity. In particular, they must ensure that their production does everything possible to respect the ethics and privacy concerns relevant to their subject material, mainly when representing individuals. These concerns extend to issues of accessibility and inclusivity by ensuring that creators recognize the needs, capacities, and diversity of their intended and potential audiences.

To introduce the practical component of the workshop, Gareth demonstrated the use of Open Brush by streaming the video feed from his VR headset to a shared screen in real-time. Visitors then put on the headsets we provided and worked on their own scenes for about 50 minutes. While some participants took our prompt and worked on storyboarding a scene with activist or empathetic intent, others were satisfied exploring the capabilities of the tools. Both Louis and Tania from the Noire team participated. While Louis was familiar with the technology, Tania had less experience with headsets. Tania was initially skeptical about how her experience would be due to her prior understanding of VR as an isolating technology. However, the activity felt much more connected and collective with the headset’s passthrough feature enabled. Tania enjoyed painting in 3D as much as our other participants, who all became deeply engrossed in the scenes they were creating.

For the workshop’s final part, we asked each participant to talk about what they created and share it with the other participants. On the first day, technical difficulties prevented each person from streaming their video feed, so each took turns trying each other’s headsets after a brief description of what they would see. On the second day, we fixed the issue with streaming the video so each participant could provide a tour of their creation from within the headset.

In each case, the embodied testing of each other’s scenes, rather than merely seeing them on the screen, had the most impact for our participants. What came across in the session was the unique value of being able to both create scenes and share the creations of others in a fully spatialised and embodied way. We also saw the potential for the development of unique personal styles of expression by way of comparison.

We concluded the session by suggesting creative next steps for participants who wish to develop these new workflows further. We thanked both the participants and the team from Noire for their inspiration and kind participation. Moving forward, Gareth and I are excited to explore the potential for immersive storytelling in new research and hope to encourage others to pursue their own journeys in the XR field.