Computer Vision as a technology to test Navigation UI in cars

on YouTube ↗41m

" summary

A practical system built over three years uses OpenCV with Python plus Ruby and Cucumber glue to automate navigation UI checks by combining p hash for quick rejects and template matching plus feature detection (ratio test, cross‑check, cloud‑of‑points radius) for robust hits. Thresholds like 0.8 and 0.75, hamming distances (17 vs 20) and scale up/down attempts handle zoom and style changes. Screenshot copy costs ~60 ms to 1 s, template finds ~3 s, object finds ~4 s; works for ~90% icon/chart cases but breaks on major map restyles.

AI-written summary · about

" chapters (31)

1. 0:00Intro
2. 2:13My article on DOU
3. 2:40How do we test applications today? Maps
4. 5:55Car position. Hamming distance
5. 6:40Zoom out. Hamming distance
6. 7:34Samples. Hamming distance
7. 8:35Template matching. Solution for 90% of cases
8. 9:27Template matching methods
9. 10:13Multiple objects matching
10. 10:49What about the other 10%
11. 12:24Features Detection Description. Puzzle game
12. 13:55Corner detectors
13. 14:54SIFT (Scale-Invariant Feature Transform)
14. 15:27Descriptors
15. 16:10Feature matching
16. 16:28Cross check & Ratio test
17. 19:55Brute Force matching
18. 20:30Core scripts. Find templates. Sparsing points
19. 22:09Core scripts. Find templates. Zoom search
20. 23:46Core scripts. Find templates on image. Example
21. 24:23Core scripts. Find objects on image. Example
22. 26:05Core scripts. Find objects. Negative
23. 26:50CV based test framework
24. 27:51Google Map tests
25. 29:04Google Fit tests. Steps
26. 30:06Selectors
27. 33:10Map tests Examples, query images, templates
28. 33:50Examples of tests. Farry
29. 34:46Examples of tests. HOV lines
30. 35:30Examples of tests. Maneuvers
31. 37:14Some numbers

" related talks

• 57mComputer Vision in UI testingDmytro Dundych · Selenium Camp 2020
• 54mVisual ValidationAngie Jones · Connect.Tech 2021
• 23mReimagining Mobile App Testing with Vision AI-Powered AgentsYash Varyani · droidcon Berlin 2025
• 27mVisual UI Testing – Testing the Last MileCraig Cavalier · React Boston 2019
• 30mHow Visual AI Makes Testing a BreezeAndrew Knight · Conf42 Machine Learning 2022
• 12mAutomation of GUI testing using visual regression testingAudun Wigum Arbo · NDC Oslo 2023
• 37mSpot the Difference; Automating Visual Regression TestingViv Richards · TestCon Vilnius 2018
• 21mContinuous Monitoring + Continuous Testing in DevOps delivery modelsMohamed, Bhargav · Agile India 2022