Exploring Nature as a Resource in Virtual Reality

Reposted from the SIGGRAPH blog:

SIGGRAPH sat down with Justin Johnson and Amber Johnson to hear more about their SIGGRAPH 2021 Groovy Graphics Assignment, “Photogrammetry for a Virtual Reality Nature Scene.” As part of their on-demand talk, the two discussed capturing nature using photogrammetry and building a virtual reality nature scene. The assignment is meant to inspire students to consider the natural environment as a resource, while simultaneously challenging them to work with an emerging technique for asset creation. Read on to hear about the inspiration for this project and how it came together.

SIGGRAPH: Tell us about the process of developing “Photogrammetry for a Virtual Reality Nature Scene.” What inspired the project and the ultimate Groovy Graphics talk that you presented?

Justin Johnson (JJ): I have been researching how nature environments provide benefits to mental restoration in a virtual reality (VR) game for a while. Along the way, I started to think about the line between real and virtual and how it can be blurred. Amber had been practicing photogrammetry and developing workflows as a way to make real-time assets. While doing a practice shoot together outside, we thought it would be interesting to integrate nature into the VR experience by adding it into the production process. Since we both teach, it’s natural that we found a way to bring this idea into the classroom.

Amber Johnson (AJ): Creating realistic nature environments can be one of the most time-consuming tasks when it comes to 3D modeling. Usually those types of spaces are taught in “advanced” courses. We did a lot of experimentation, including capturing spaces around our yard. We realized how easy it was to get good assets that would normally take hours to create by hand. We figured that this would be a great project to “hook” students; the process is fun and the results are exciting.

SIGGRAPH: How can the natural environment be used as a resource in creating VR games?

JJ: Our presentation at SIGGRAPH 2021 showed you can use this to find objects to capture with photogrammetry and then process into 3D assets. However, the natural environment can also serve as a strong source of inspiration. We all have a connection to nature that some say is rooted deep in our evolutionary psychology. I believe there is a way to utilize how we interact with and respond to nature in a VR game in order to foster stronger connections and evoke certain emotions and feelings from the experience.

AJ: It is important to work from reference when you’re creating, especially with nature scenes. Being able to look at how a space formed naturally will allow you to create more realistic spaces. How do boulders look after they’ve weathered in a particular environment? What does a tree look like after it has fallen? You have to ask yourself these questions so you can get a decent picture in your head. We want to make sure to get all the subtle details.

SIGGRAPH: What do you love most about creating virtual reality nature scenes?

JJ: I love it all, however if I had to choose, I would say building the scenes in a game engine. I also love processing the captured data — retopologizing, creating materials, and even when the photogrammetry data is messy — resculpting and kitbashing to repair it or make entirely new assets.

AJ: I enjoy the hunt to find a good subject. When you’re trying to find natural objects to capture, you’re at the whim of the environment. It’s possible you might not find anything that would be good to capture. During one of my outdoor shoots, I found an opossum tooth in a stream. You don’t get that experience in front of a computer. We are able to digitize real nature experiences for anyone to experience virtually. Nature is ephemeral and it won’t be long before your subject is changed or gone.

SIGGRAPH: How do you expect the attention to detail of nature/background scenes to evolve in VR?

JJ: As rendering becomes more powerful and software more efficient at making assets, we will be able to add more to the VR environment at higher resolutions. In addition, we expect AI and procedural techniques to play a role to some degree, filling in micro details such as the understory of a forest, bugs, fungi, etc. In the future, we’ll be able to lay out an environment and define areas with visuals and rules while letting AI bring it to life. If sophisticated enough, the chaotic and random aspects of nature might be better represented.

SIGGRAPH: People can still access all on-demand content from SIGGRAPH 2021, including your session. What, in your opinion, is the best part of SIGGRAPH 2021 and why should people register?

JJ: I love how interdisciplinary it is. There really is something for everyone. It’s amazing to see a place where games, film, animation, research, entertainment, and critical thinking all come together. It’s very inspiring to see the work that other artists and designers are doing around the world. It’s also great, as an educator, to see student work and learn from other educators about new ways to help our students foster creativity and develop cutting-edge skills.

AJ: I agree with the interdisciplinary aspect of it. The professional conferences I usually attend are game-focused. Related fields can be inspired by or learned from each other. I have personal interests outside of my field, so it’s fun to see what they’re doing.

SIGGRAPH: Your project was included as part of the Educator’s Forum, how would you recommend educators in computer graphics and interactive techniques apply what you shared in their classrooms?

JJ: Something we like about the project is that it can be scaled and segmented, so we encourage other educators to think about what aspects can be used or altered to fit their needs. For example, if the shooting phase is a challenge, maybe students could focus on VR development and use pre-scanned nature assets like from the Quixel Megascans library. Likewise, students could focus on the shooting/capturing phase. Working at different institutions with students who had vastly different backgrounds forced us to each put more emphasis on different parts of the process. The pandemic also posed a challenge with access to equipment and software. So even though we recommend specific equipment, software, and techniques, our submission really is about the idea of weaving real and virtual nature environments together; all the details along the way can be altered, substituted, and most likely improved.

SIGGRAPH: What advice do you have for someone looking to submit to future SIGGRAPH conferences, specifically Groovy Graphics?

JJ: This is our first SIGGRAPH submission, and although we think it went very well, we don’t think it’s any kind of exemplar — there are some amazing submissions from others, many of which are SIGGRAPH veterans who would be much more equipped to give advice. That said, try to find inspiration from multiple disciplines. Going back and watching previous SIGGRAPH presentations and papers was inspirational in seeing how things typically thought of as dichotomies can come together successfully, things like physical and virtual, 2D and 3D, art and computation, etc. There might be more interesting combinations out there or gaps waiting to be filled!