Knowing When the Object Is Not There: Calibrated Few-Shot Presence Detection for Teachable Object Recognition on Mobile Devices

Abstract

Teachable object recognition lets blind and low-vision users teach a phone their own objects, such as keys or a cane, from a few short videos and later ask the phone to find them. For this to work, the phone must decide in every frame whether the object is there at all. Standard few-shot recognizers cannot make that decision: they always name one of the user’s objects. The ORBIT benchmark also removes “object not present” frames before scoring, so this failure is never measured, and deployed apps fall back on hand-tuned confidence thresholds. In a pilot with 8 users from the public ORBIT-India dataset, thresholds set from each user’s teaching videos to catch 90% of appearances found only 62% and 55% on average on cluttered video, with EfficientNet-B0 and DINOv2 backbones respectively. A single global threshold instead gave false alarms on about half of the frames without the object. This thesis proposes a conformal calibration method that sets a per-object threshold from the user’s teaching videos alone and corrects for the shift from clean teaching scenes to cluttered use. To our knowledge, it also gives the first ORBIT evaluation that keeps not-present frames and the first recognition results on ORBIT-India, showing whether systems built mostly with users in high-income countries work for users in South Asia. The method will be tested on a mid-range Android phone, so that a find-my-object app can tell its user, at a stated and tested level of reliability, when the object is not there.