Sign Language Video Game in Augmented Reality
Educational augmented reality game teaching sign language through interactive challenges.
Project
Educational augmented reality game teaching sign language through interactive challenges.
01
Vision
A short augmented-reality visual novel designed to make introductory sign-language practice more engaging. Players communicate with characters and make narrative choices by performing signs in front of the mirror.
02
Characters and tutorials
The interactive avatars build on the MediaPipe and Kalidokit pipeline. Recorded poses and signs are used both for character animation and for demonstrations shown to the player.
03
Recognition
A bidirectional LSTM classifies temporal landmark sequences. When the probability of a detected sign exceeds a threshold, the game engine validates the action and advances the scene.
04
Game engine
P5VN was substantially adapted to run asynchronously inside GOSAI and receive recognition events through Redis. This connects the narrative system, computer vision and mirror interface.
05
Learning experience
The project tests gamification as a way to sustain motivation while keeping the interaction equipment-light: signs are estimated by computer vision rather than gloves or dedicated controllers.