Emerging Technologies Explored

Advance design studios intentionally push the boundaries of innovation, technology, and research, and Associate Professor of Art + Design Patrick FitzGerald has found that this experiential “think and do” learning process delivers the best results and outcomes for his students. Recently, FitzGerald led ADN 560, Advanced Animation Studio with a focus on experimental multimedia animation, which looks at developing design solutions in concept and execution through the creative process.

As part of the course, FitzGerald added a layer of learning and exploration that required students to utilize Stanford University’s d.School’s Design Thinking process in the development and discovery of each project. The d.School approach to design thinking is a nonlinear design process and methodology for creative problem-solving that is based on a five-stage model: empathize, define (the problem), ideate, prototype, and test. Each stage leads to the next; however, users can go from one stage back to another and jump between stages, utilizing a nonlinear, reiterative process. “We used this process as the theme of our studio where each project had to be built around this,” says FitzGerald. Additionally, the Design Thinking process can be applicable to most all topics or problems.

Art + Design graduate student Andrew Wiedner says that using the d.School’s Design Thinking process “organized and added structure” to his current back-and-forth design process. He says, “the documentation helped me develop steps in my workflow.”

During the studio, FitzGerald assigned three projects that were quite different from each other but presented in a progressive sequence to help students formulate the next project. “Each of the projects offered valuable gains and development that led to better outcomes for the next project,” states Ariana Ehuan, also an Art + Design graduate student.

Making a Difference in STEM

For the initial project, FitzGerald challenged his students to develop prototypes to address the growing trend of low numbers of females entering STEM fields. Included in the scope of the project was the inclusion of Sphero. “Sphero is a small ball-shaped robot that can be programmed to run a circuit or course with blocks or text-based code. Because it does not require you to know code, it provides an easy path for kids to understand coding,” adds Wiedner. Sphero can be controlled by a smartphone or tablet and has many applications in entertainment and education.

The students’ research into the decline of women entering the STEM fields revealed that societal and environmental factors heavily influence girls in their career path choice. “At a young age, girls have not been receiving the same encouragement that boys do,” Ehuan says. Research also showed that by the second grade, girls have started to formulate areas of interests for career paths and have a higher propensity to avoid STEM.

As the team continued to follow the Design Thinking protocol, they discovered, “there is a super high frustration curve to coding. Using the Sphero can reduce this,” says FitzGerald. The students “decided to develop individual exercises that included simple steps that were fun and engaging and offered exposure to STEM,” FitzGerald adds. These exercises were developed with the intent of sharing them with a group of second graders at The Exploris School, a charter school in Raleigh. The school champions project-based learning and encourages parents and family involvement by providing projects that teach students beyond the classroom. FitzGerald has children who attend Exploris and has often led interactive sessions there, such as introducing virtual reality (VR) to kindergarteners.

Having this connection, FitzGerald reached out to Sonja McKay, a teacher at Exploris who has a passion for connecting students to the local and global community as well as utilizing design thinking in the classroom. McKay embraced the opportunity to have FitzGerald’s students share the learning exercises they developed utilizing the Sphero with Exploris students. Exploris purchased a dozen Spheros realizing their potential educational value.

“As part of the collaboration [with Exploris], we requested that their group of students be comprised of an equal number of girls and boys to ensure it was accurately divided,” FitzGerald offers. “Otherwise, it would defeat the goal of the project, which was to see if through exposing girls to coding in a fun and playful way, without directly relating the exercises to computer science, would encourage their interest in STEM.”

Video Above: Students in Art + Design ADN 560 developed a unique curriculum to expose students to STEM basics using the Sphero.

For several weeks, FitzGerald and his students went to Exploris and worked with a group of second graders to introduce engaging exercises using the Sphero. “Each lesson was a building block starting with the basic, where we first introduced the Sphero and how to program and control it,” FitzGerald explains. Initially, the students were excited. “It was a novelty thing,” FitzGerald says, and only a few students were familiar with the Sphero. Part of the tasks involved students directing the Sphero through a maze that required simple directional commands. The design students tried to encourage having fun while learning. “Our learning constructs inspired curiosity to teach kids by playing games,” says Ehuan.

While some of the children grew disinterested, a few were eager to explore on their own. “There were a few kids who really latched onto the concept, and one student actually wanted to learn how to make the Sphero speak—to count down backwards from 10,” Wiedner says. “A few students sat, listened, and observed as I programmed a repetitive loop that required decreasing the number each time until we got to zero.”

The experience for FitzGerald’s students was positive, and the Exploris students have the fundamentals to further explore coding and STEM. “There now is a Sphero robot club at Exploris that students can participate in,” FitzGerald shares. “This project has the opportunity for further benefits. The lessons involved sequential thinking and that is exactly why we chose the Sphero. In order to ensure engagement with the children, the tasks were designed to be fun so that they would come back and do it again. Each one of these lessons incorporates design thinking for problem-solving.” Though the long-term outcome of this Sphero project and others like it won’t be fully realized for some years, FitzGerald and his students speculate that an early introduction to technology in a supportive culture can spark STEM-based career interests in girls.

AR on Campus

Augmented reality (AR) is a direct or indirect view of the real world that is “augmented” or layered by computer-generated perceptual information that involves a user’s physical senses, specifically their sense of touch, hearing, sight, and even smell. “AR is a technology that is coming, and it will be a sort of a layer over the reality of what a user is seeing and can be valuable for training and education,” FitzGerald says. With this in mind, FitzGerald designed the second project for his studio, asking students to explore how AR could be used on campus to address challenges freshman students frequently encounter, while being sensitive to the potential dangers as well as identifying the benefits of the technology.

The design students began by thinking about the implementation and implications of AR on campus and determined that they would learn best by observing other students. The Talley Student Union offered a good environment for this research. “Observation allows you to empathize with your user, which in this case are the students,” Wiedner says. Wiedner was interested in addressing the feelings of isolation and stress that freshmen can experience from being out of their comfort zone. “I was looking at the causes and discovering how AR could encourage communications with face-to-face connections,” Wiedner says.

Apparent from their observations was the lack of connection between students. In many instances, “the students were sitting together; however, they were on their phones or computers, and no actual conversations were taking place,” says Ehuan. These observations led the design students to explore common transitional challenges for new college students including loneliness, depression, and acceptance.

“My interest was specifically depression among new college freshman and how AR could be used to combat this,” says Ehuan. “There is a lot of research on this, so I wanted to combat the factors that cause it.” Ehuan shared that because she went away to college, she could relate to isolation as someone who was in a new environment.

The project developed into a “proof of concept” idea for each of the students. Ehuan’s idea was an AR app that worked with Google Glass; every freshman would be provided the tools, possibly as part of an orientation package. She emphasizes that there would need to be full inclusion “without barriers that would inhibit students from having access or getting involved.”

The app would allow a user to select very general topics of interest, each associated with an icon. If a student were actively using the app and wearing the Google glasses, their selected areas of interests would be visible as icons above their head to other students also wearing the glasses. Seeing shared interests with other students “could trigger a conversation and lead to building new friendships,” says Ehuan.

Ehuan developed a video showing the interaction of two students who had common interests and also shared that the app would have an opt-out setting and that the selected interests could be changed at any time.

A Virtual Tour  

For the final studio project, students were tasked with creating a design solution for the Gregg Museum of Art & Design using conversational interface and VR. This project built on the previous learning outcomes and allowed the students to focus on their individual strengths. “We were more like a four-person design team, rather than a class,” shares Ehuan. “We each had specific roles based on our interests and strengths, and since there is a specific sequence of a project, it worked to our advantage.”

A conversational interface is a computer user interface that mimics human conversation; however, the human has the ability to direct the conversation, rather than imputing commands through a computer prompt, such as clicking on icons or buttons. There are two types of conversational interfaces including voice assistants that the user talks directly to and “chatbots,” which are the most common and require typing such as in an online chat. Voice assistant examples are Siri (Apple) and Echo (Amazon) which can do tasks such as play music and perform a search with only a verbal command by the user.

FitzGerald had previously dabbled in using a conversational interface to explore multiple methods of communication that users could toggle between. Various communications modes provide users with options and allow them to better direct the “conversation.” In the case of a learning application, users can control the learning outcome based on personal interests. Weidner noted the flexibility and customization potential in a conversational interface—“a change in communication methods including video meets you where you [as the participant] are and where you want to go in the realm of a particular topic.”

The Gregg offers a perfect environment for the project. Museum visitors have variable interests, learn differently, and may become interested in seeking unique information associated to an artifact. However, the hours and location of a museum can pose a potential barrier for people who want to learn about a particular artifact or genre.

The problems the design team wanted to solve “addressed the museum, visitors, and potential visitors,” shares Ehuan. “The museum has the issue of space; they can only show a small sampling of their entire collection simultaneously,” says Ehuan. Additionally, visitors can be limited by what is currently on display and the inability to garner additional information about an artifact that interests them. “The solution was to use VR to create a virtual museum, where physical space is not an issue and there are no limitations of showing all the artifacts, or expanding the knowledge base of an artifact,” FitzGerald says.

The student team researched various means to deliver information including VR, video, and audio and more linear aspects using popups and icons that lead to additional learning opportunities. They decided to develop an interactive VR learning app that uses multiple modes of communication to explain museum artifacts and information at varying levels of user interest. Secondary-level information using a hierarchy of content delivers a custom contextual experience for participants. “The options of beginner-, intermediate-, and advanced-level information; the addition of fun facts; and the option to navigate your way through all the levels, skip around, or watch everything available on one object and then move to something else based on your interests, is what makes this so limitless in its application,” adds Ehuan. The app was also created with the intent that it could be expanded upon as new information or artifacts were accumulated.

Associate Professor and Art Historian Russell Flinchum, Ph.D., was asked to participate as the narrator and educator in this project. FitzGerald says they asked Flinchum to participate in the project because of his knowledge base and expertise. “He is an expert communicator that speaks to the audience using verbal and nonverbal cues,” says FitzGerald.

The VR experience begins outside of the Gregg, as a way of introducing the venue. Users can move inside the museum and start to explore and make content choices. Arrows on the floor allow users, through gaze selection, to advance to the next artifact or room. Gaze selection captures eye movement based on direction or duration of time; it will prompt a user to move in a specific direction. Flinchum appears next to an object and offers detailed information about the artifact at the level that the viewer has selected (beginner, intermediate, or advanced). The use of the secondary learning allows the viewer to also customize their experience.

“We have a mostly cohesive tour of the Gregg and are excited about the outcome, and the actualization of the prototype is an excellent proof of concept,” shares Wiedner. “ We did this in a short amount of time and the level of difficulty was actually low,” adds Ehuan. Wiedner and Ehuan both indicate that this project could be easily developed further by adding new artifacts and information. Additionally, it would also be beneficial to apply this to other forms of content topics.

“This can be a growing learning archive that can be posted on the web and all you need is a Google Cardboard to view,” emphasizes FitzGerald. As new content is added, the app just needs to be updated. “There are several factors that make this a powerful mode of communication in a VR setting including immersion, sensory feedback, and interactivity that are all providing this VR experience with sight and sound.”

The team is grateful to be able to use the Gregg in their project. They shared their prototype with Mary Hauser [BID ’01, MID ‘04], associate director and registrar at the Gregg. “The Gregg was so generous in offering the facility and time to us,” FitzGerald says.