Writing

ImmerseGuard: Bridging Immersion and Environmental Awareness

In short: ImmerseGuard is a phone app that listens through your noise-cancelling headphones and tells you when something important is happening around you. It runs on the device, with no internet…

published
read time
2 min
words
380
lang
en

In short: ImmerseGuard is a phone app that listens through your noise-cancelling headphones and tells you when something important is happening around you. It runs on the device, with no internet connection needed.

The problem with good noise cancelling

Noise cancelling has got very good. That is great for music and for immersive content, and it is a problem for everything else. Once the room is gone, so are the sounds you actually needed to hear.

Conceptual framework of the ImmerseGuard system
The system in outline. Users switch on ANC in all sorts of places: walking down the street, doing something at home, or inside a VR headset for full immersion. ImmerseGuard analyses the headphone audio in real time and, when it detects a sound event, alerts the user with a text or voice notification.

What we built

ImmerseGuard puts an on-device sound event detection system alongside active noise cancelling (ANC). It is compressed enough to run on a phone, and it gives you real-time alerts for the sounds that matter without pulling you out of what you are listening to.

ImmerseGuard app screen
ImmerseGuard app screen
ImmerseGuard app screen

How it works

The core is YAMNet, which is known for doing sound event detection efficiently on edge devices. We pair it with a shallower model trained by transfer learning. Together they pick out emergency vehicle sirens, general vehicle noise, room and household sounds, human speech, pets and device alarms.

To make it fit a resource-limited platform we quantized the model. That gets inference times down to something usable in real time and lets the whole thing run locally, with no internet connection.

Benchmark results

What happened when people used it

The user experience studies came back positive. People found it practical for staying aware of their surroundings while immersed in digital content, and they liked that it just sat on their phone. No heavy files or models to download.

What is next

The studies had limits and we know it. The next round is about making ImmerseGuard work across a wider range of devices and real environments. From user feedback, we also want more specific sound classes and notifications people can tune themselves.

Balancing immersive content against staying connected to the room is the whole point. On-device machine learning and model compression are what make it possible.

You can read the full paper on ResearchGate.