Pin AR Objects With Google’s Instant Motion Tracking

oogle has recently announced the release of the ‘Instant Motion Tracking’ solution in MediaPipe, which is built upon the MediaPipe Box Tracking solution previously released. The recent upgrade to the Instant Motion Tracking, now allows users to place virtual 2D and 3D content on static or moving surfaces, which seamlessly interacts with the real world. 

The feature is essentially a pin functionality for AR content, allowing users to augment the real world around them in real time, rather than having to wait for a pre-recorded video to be analysed and objects pinned after the video has been shot. Now users are able to pin AR objects or GIF animations to both fixed and moving objects, which the user can move around from their chosen anchor point and interact with the AR animation which will respond to their movements accordingly. 

Google stated that, “the Instant Motion Tracking solution provides the capability to seamlessly place virtual content on static or motion surfaces in the real world.” The solution provides six degrees of freedom (6DoF) tracking with relative scale in the form of rotation and translation matrices. This tracking information is then used in the rendering system to overlay virtual content on camera streams to create immersive AR experiences.

The core concept behind Instant Motion Tracking is to decouple the camera’s translation and rotation estimation, treating them instead as independent optimisation problems, enabling AR tracking across devices and platforms without initialisation or calibration, by first finding the 3D camera translation using only the visual signals from the camera. This involves estimating the target region’s apparent 2D translation and relative scale across frames. 

The smartphone’s 3D rotation from its built-in IMU (Inertial Measurement Unit) sensor also plays a pivotal role, by combining this translation and rotation data, the Instant Motion Tracking solution can track a target region with 6DoF at relative scale. This information allows for the placement of virtual content on any system with a camera and IMU functionality and is calibration free.

Google’s Instant Motion Tracking solution will allow developers and users to transform both 3D visuals and GIF animations into AR experiences, creating fun and immersive experiences with animated content blended into the real environment.