A system for understanding the content of street signs using finger-tracking

Papoulakis, G

Abstract

The work describes the development of a computer vision system which detects and
interprets the text on signs in a scene. A key component in this system is the use of
finger detection to select a particular sign in the scene, or text on a sign. The system is
comprised of a Head-Mounted Display equipped with a light-weight camera which are
connected to a wearable computer or a laptop. A set of Image Processing, Pattern
Recognition and Optical Character Recognition methods are aggregated to process the
sign's writing in real time. The system employs symbol recognition to detect graphics
on a sign and OCR to process text. These merge to annotate the sign on the HMD for
the user to read. Implementation details of the system are presented along with results
gathered through experimental usability tests with prototype applications. A method to
detect a finger-pointing hand gesture is used which together with a finger-tracking
method serve as means of making a selection of the content on a sign. The gesture is
used in two ways, one being to simply point at a sign's content with the finger, while
the other method involves flexing the finger to perform an action resembling a mouse
click. The two selection methods are compared through a series of indoor usability
experiments where small teams of 5-10 users try both methods with single and multiple
attempts on mock information signs. Statistical comparisons of subjective and objective
data indicate the conditions that affect the time performance, comfort and preference of
each method when selecting the entire writing of a sign or part of it. The method of
pointing to select the entire writing on a sign showed constant preference and achieved
higher performance. This was tested outdoors with real information signs on the campus
of the University of Salford to confirm the comparison's results.

Citation

Papoulakis, G. A system for understanding the content of street signs using finger-tracking. (Thesis). Salford : University of Salford

• Thesis Type: Thesis
• Deposit Date: Oct 3, 2012
• Award Date: Jan 1, 2011