My Google Glass Sees Your Passwords!
summary
A computer-vision attack records fingertip motion with Google Glass, webcams or camcorders and maps touch points to a reference keyboard using planar homography, recovering more than 90% of full passcodes at ~3 m and ~97% per-key accuracy; experiments include 12 volunteers, phone/tablet targets, and a 44 m camcorder demo with 100% success. Pipeline uses optical flow, deformable part-based model, k-means clustering and homography to find touch points. Countermeasure is a randomized Privacy Enhancing Keyboard for Android (available on Google Play).
AI-written summary · about
chapters (11)
- 0:00Intro
- 0:43Introduction Blind recognition of touched keys Evaluation Countermeasures Conclusion
- 2:53Use computer vision techniques to recognize possible touched keys and use a language model to correct the prediction. Poor success rate for passwords.
- 4:54Taking Videos Use sneaky cameras including Google Glass, web cameras, smartphone cameras, even smartwatch! - Factors: camera angle, distance, lighting Adjust the camera angle at a distance to record the device and touching fingertip movement.
- 6:40Preprocessing Keep the area of moving hand on screen. - Use Deformable Part-based Model (DPM) - an object tracker - to track the area of interest for a moving target
- 7:58Detecting Touching Frames Derive a pattern of the touching finger movement Finger moves downward, stops and then upward. Track feature points on the hand by optical flow. Al fingers keep the same gesture during touching Use the frame in which velocity of most tracked points changes the direction as touching frame.
- 9:47Deriving Homography Matrix Derive touch screen corners, intersection of the four edges of the touch screen. Canny edge detector to detect edges Hough line transform to get the lines. Use these four pairs of corner points to derive the homography matrix.
- 11:09Locating Touching Fingertip Use the DPM object detector to locate the touching fingertip in touching frames. Derive a large box bounding the touching fingertip.
- 12:44Estimating Touched Area Deriving the fingertip contour Use k means clustering to cluster pixels in a small bounding box to get the fingertip contour. Two groups bright fingertip and dark screen. Deriving the accurate touched area Fit a line over central points of the contour and get
- 14:01Recognizing Touched Keys Which pixels are the touched points in this estimated tiny touched area? If the touched point is found, map the estimated touched point to the reference image of the software keyboard.
- 14:40Step 7. Recognizing Touched Keys (Cont'd) Apply k-means clustering to estimated touched area. -5, because of illumination and shadowing Use a point in the upper part of the darkest cluster as the touched point.
related talks
06mListen to Your Fingers: User Authentication based on Geometry Biometrics of Touch Gesture
19mStay Cool! Understanding Thermal Attacks on Mobile-based User Authentication
32mWhat We Can Learn from Google’s Fight with Phishing
27mPasskeys on Google platforms: updates on latest features and real-world successes
39mHacking Google - Lessons learned running and growing an internal red team
05mAuthentication UX Design Opportunities for Future Smart Glasses
26mPasskeys on Android
15mPasskeys for Google accounts: learnings and our shared path forward