In the real world/digital world cross-over of mixed reality, a user’s immersive engagement with the program is called presence. Now, UMass Amherst researchers are the first to identify reaction time as a potential presence measurement tool. Their findings, published in IEEE Transactions on Visualization and Computer Graphics, have implications for calibrating mixed reality to the user in real time.

“In virtual reality, the user is in the virtual world; they have no connection with their physical world around them,” explains Fatima Anwar, assistant professor of electrical and computer engineering, and an author on the paper.

“Mixed reality is a combination of both: You can see your physical world, but then on top of that, you have that spatially related information that is virtual.” She gives attaching a virtual keyboard onto a physical table as an example. This is similar to augmented reality but takes it a step further by making the digital elements more interactive with the user and the environment.

The uses for mixed reality are most obvious within gaming, but Anwar says that it’s rapidly expanding into other fields: academics, industry, construction and health care.

However, mixed reality experiences vary in quality: “Does the user feel that they are present in that environment? How immersive do they feel? And how does that impact their interactions with the environment?” asks Anwar. This is what is defined as “presence.”

Up to now, presence has been measured with subjective questionnaires after a user exits a mixed-reality program. Unfortunately, when presence is measured after the fact, it’s hard to capture a user’s feelings of the entire experience, especially during long exposure scenes. (Also, people are not very articulate in describing their feelings, making them an unreliable data source.) The ultimate goal is to have an instantaneous measure of presence so that the mixed reality simulation can be adjusted in the moment for optimal presence. “Oh, their presence is going down. Let’s do an intervention,” says Anwar.

Yasra Chandio, doctoral candidate in computer engineering and lead study author, gives medical procedures as an example of the importance of this real-time presence calibration: If a surgeon needs millimeter-level precision, they may use mixed reality as a guide to tell them exactly where they need to operate.

“If we just show the organ in front of them, and we don’t adjust for the height of the surgeon, for instance, that could be delaying the surgeon and could have inaccuracies for them,” she says. Low presence can also contribute to cybersickness, a feeling of dizziness or nausea that can occur in the body when a user’s bodily perception does not align with what they’re seeing. However, if the mixed reality system is internally monitoring presence, it can make adjustments in real-time, like moving the virtual organ rendering closer to eye level.

One marker within mixed reality that can be measured continuously and passively is reaction time, or how quickly a user interacts with the virtual elements. Through a series of experiments, the researchers determined that reaction time is associated with presence such that slow reaction time indicates low presence and high reaction time indicates high presence with 80% predictive accuracy even with the small dataset.

To test this, the researchers put participants in modified “Fruit Ninja” mixed reality scenarios (without the scoring), adjusting how authentic the digital elements appeared to manipulate presence.

Presence is a combination of two factors: place illusion and plausibility illusion. “First of all, virtual objects should look real,” says Anwar. That’s place illusion. “The objects should look at how physical things look, and the second thing is: are they behaving in a real way? Do they follow the laws of physics while they’re behaving in the real world?” This is plausibility illusion.

In one experiment, they adjusted place illusion and the fruit appeared either as lifelike fruit or abstract cartoons. In another experiment, they adjusted the plausibility illusion by showing mugs filling up with coffee either in the correct upright position or sideways.

What they found: People were quicker in reacting to the lifelike fruit than they would to the cartoonish-looking food. And the same thing happened in the plausibility and implausible behavior of the coffee mug.

Reaction time is a good measure of presence because it highlights if the virtual elements are a tool or a distraction. “If a person is not feeling present, they would be looking into that environment and figuring out things,” explains Chandio. “Their cognition in perception is focused on something other than the task at hand, because they are figuring out what is going on.”

“Some people are going to argue, “Why would you not create the best scene in the first place?” but that’s because humans are very complex,” Chandio explains. “What works for me may not work for you may not work for Fatima, because we have different bodies, our hands move differently, we think of the world differently. We perceive differently.”

So you program your thermostat to save heating costs, recycle glass and plastic, ride a bicycle to work instead of driving a car, reuse sustainable grocery bags, buy solar panels, and shower with your mate—all to do your part to conserve energy, curb waste and lower your carbon footprint.

A study released last week may just spoil your day.

Researchers at Carnegie Mellon University and Hugging Face, a machine learning community website, report that you might still contribute to climate change if you are one of the 10 million-plus users who tap into machine learning models daily.

In what they term the first systematic comparison of costs associated with machine-learning models, the researchers found that using an AI model to generate an image requires the same amount of energy as charging a smartphone.

“People think that AI doesn’t have any environmental impacts, that it’s this abstract technological entity that lives on a ‘cloud,'” said team leader Alexandra Luccioni. “But every time we query an AI model, it comes with a cost to the planet, and it’s important to calculate that.”

Her team tested 30 datasets using 88 models and found widespread differences in energy usage between varying types of tasks. They measured the amount of carbon dioxide emissions utilized per task.

The greatest amount of energy was expended by Stability AI’s Stable Diffusion XL, an image generator. Nearly 1,600 grams of carbon dioxide is produced during such a session. Luccioni said that is roughly the equivalent of driving four miles in a gas-powered car.

On the lowest end of the scale, basic text generation tasks expended the equivalent of a car driving just 3/500 of a mile.

Other categories of machine learning tasks included classifying images and text, image captioning, summarizations, and answering questions.

The researchers stated that generative tasks that create new content, such as images and summarizations, are more energy- and carbon-intensive than discriminative tasks, such as ranking movies.

They also observed that using multi-purpose models to undertake discriminative tasks is more energy-intensive than using task-specific models for the same tasks. This is important, the researchers said, because of recent trends in model usage.

“We find this last point to be the most compelling takeaway of our study, given the current paradigm shift away from smaller models fine-tuned for a specific task towards models that are meant to carry out a multitude of tasks at once, deployed to respond to a barrage of user queries in real-time,” the report said.

According to Luccioni, “If you’re doing a specific application, like searching through email … do you really need these big models that are capable of anything? I would say no.”

Although the numbers of carbon dioxide usage for such tasks may appear small, when multiplied by millions of users relying on AI-generated programs daily, often with multiple requests, the totals show what could amount to a significant impact on efforts to rein in environmental waste.

“I think that for generative AI overall, we should be conscious of where and how we use it, comparing its cost and its benefits,” Luccioni said.

The findings are published on the arXiv preprint server.

© 2023 Science X Network