Skelesense : Arabic sign language recognizer, communicator and tutor for the deaf and mute

Khadragi, AI

Abstract

The deaf and mute society is a closed society that faces many obstacles in communication
with the outside world. Through the use of the sign language the deaf and mute people express
themselves to each other and to those who know their signs. There are many sign languages
worldwide. British Sign Language (BSL) and American Sign Language (ASL) are amongst the
most highly used and attention receiving sign languages. One of the less evolving sign languages
is the Arabic Sign Language (ArSL) in general and its Egyptian Sign Language (ESL) dialect in
specific. There are very few resources and fewer researches that were conducted to promote the
ArSL or the ESL. So, the aim of this work is to develop and evaluate a computer-based system,
SkeleSense, to teach the deaf and mute their sign language and enhance their communication
skills with the surrounding world.
This thesis describes the design and implementation of this system which is used for the
recognition, teaching and communication of ESL. The approach taken is to develop a low-cost
sensing glove which provides movement signals which are input to a computer system and used
to control a 3D module of the user's hand. A graphical interface provides a user friendly
environment allowing the user to choose various modes of operation for the system, including
training of a user's hand-movements and comparison with expert's pre-stored sign movements
for different words. SkeleSense allows the deaf to communicate with others through the
production of sign to speech translator recognizing input signs and converting them into sound.
Four local ESL experts helped to produce a 65 evaluation sign pool. Ten deaf users tested
the system by comparing their signs against the experts' stored references. Each user was
allowed 5 trials for each of the 65 signs. SkeleSense users achieved an overall average
correctness of 4.69 out of 5 (93.8%). The highest average correctness was 4.92 out of 5 (98.4%)
and the lowest was 4.09 (82%). Based on users' scores, 18.5% of the signs were difficult to
perform because of certain joints bending angles. Those signs specifically showed enhanced
correctness measures with the use of SkeleSense. Questionnaires were conducted showing that
users including the one who achieved the least correctness recommended the system's use and
performance over the traditional human based sign language systems for tutoring ESL. This
indicates the significant role of SkeleSense in sign language tutoring.

Citation

Khadragi, A. Skelesense : Arabic sign language recognizer, communicator and tutor for the deaf and mute. (Thesis). Salford : University of Salford

• Thesis Type: Thesis
• Deposit Date: Oct 3, 2012
• Award Date: Jan 1, 2011