Microsoft HoloLens could help you find your keys, and also stalk your every move

You know the trope in Sherlock-esque detective shows where some brilliant sleuth cracks a case by drawing on their nigh-photographic memory? Well, one of the most fascinating and terrifying things about augmented reality glasses is that they turn everybody into that sleuth — and as a new Microsoft patent filing shows, that can be used for things besides crime-solving. The recently published application covers a system that would let its HoloLens glasses track small items like car keys, stopping wearers from misplacing them. More broadly, the patent describes a system that can monitor the status of objects without any instructions from users, keeping tabs on anything that’s important to their lives.

The patent’s basic idea is pretty simple. HoloLens has outward-facing cameras that can make a spatial map of a room, and machine vision technology can identify or track specific objects in an image. So if, for example, you put your keys down on a table, HoloLens could hypothetically spot them through the camera and quietly note their position. When you’re about to leave the house, it could give you the keys’ last known location, even if they’ve since been covered up by a newspaper or slipped under a couch cushion.

This seems like a pretty inefficient way to find your keys today, or possibly ever. For one thing, you’d need to be consistently wearing smart glasses, which is currently staggeringly inconvenient. If you drop your keys somewhere without looking at them, the system might not register that they’ve moved. It’s not clear how much it would help if you slipped something in a coat pocket or bag and then carried it around, unless it’s got some very complex multi-location tracking. It’s not hard to slap a Bluetooth tracker on an item if you’re consistently losing it. And by the time we get to a future of 24/7 augmented reality, we may not have keys at all.

But what’s really interesting isn’t the idea of HoloLens tracking an object. It’s HoloLens learning what items matter to you and choosing what to follow, before you ever worry about losing something. To be clear, you could designate objects: one example has a traveler telling HoloLens to track their passport while abroad. In other cases, though, it could check to see how often you interact with an object, or when you move it around, and start tracking anything that hits a certain threshold.

Any object you pick up and carry to work every morning, for example, could be more important than the glass of orange juice you place on the table beforehand. If you pull an object out of your pocket at a restaurant, HoloLens could automatically warn you to grab it when you leave. This could even extend to things like checking your fridge to see if an item that’s usually present is absent or depleted, and reminding you to buy it when you enter a grocery store hours later.

This is where an augmented reality headset becomes truly transformative: instead of ticking off a checklist, it anticipates needs you didn’t realize you had. That’s also when it starts feeling a little scary. If multiple people are wearing headsets, the patent describes how data could be shared between them, so you could find your keys even if someone else has moved them. This is fairly innocuous, but it could also do something like alert an abusive spouse that their partner was leaving the house. Remember when a targeted advertising flyer reportedly revealed a girl’s pregnancy to her parents? Imagine what a company like Microsoft or Google could do with an all-seeing headset that finds patterns in your every move — or what law enforcement agencies could do with “metadata” that’s essentially an inventory of your life.

HoloLens obviously isn’t the only piece of tech that can track or reveal our secrets, and patents are often about as close to reality as a sci-fi novel. Even if Microsoft had such a system working, HoloLens is so limited and uncomfortable today that nobody’s going to be casually reading the morning paper in it. But it’s a particularly interesting manifestation of panoptic quantified-life tech, and one that it’s not hard to imagine cropping up again in the future — not just in patent filings.