top of page

HandiCaptain: A Collaborative Gamified Solution to Hand Rehabilitation

In my M2.1 project, I loaned a pair of Manus Meta-gloves. The gloves can track the finger movements instantly and restore them in a virtual scene. With such devices, the virtual avatars of hands can mimic the finger/hand movements in reality.


According to the Self-determination Theory (SDT), the three key factors that affect motivation are competence, autonomy, and relatedness. The form of the Meta-gloves can be categorized as Realistic Tangible Natural Mapping, which increases the level of competence and autonomy in the context of gameplay. The other factor, relatedness, could be achieved by collaborative gameplay.


So far, the majority of the studies around gamification for hand rehabilitation have been done without Hand Rehabilitation Devices (HRDs). Therefore, this project focuses on employing collaborative gameplay to increase patients’ motivation for hand rehabilitation using hand rehabilitation devices. Specifically, the target group is people after stroke.

Design Concept

The gloves come in pairs. Since most patients suffering from stroke lose the motor function of one side of their bodies, it is only required to rehabilitate one hand for each patient. In this case, both patients could use their affected hands to cooperate, which also suits the accessible hardware setting (one pair of gloves). The pair of patients are then integrated into the game, acting differently as solitary parts. An asymmetric mechanism was also incorporated to enhance players’ perceptions of connectedness, social engagement, immersion, and comfort with a game’s controls.

First Iteration

Mechanics: The two players control the movement of the robot together. They apply different hand/finger movements to achieve different movements of the robot, to eliminate the enemies in the scene one by one.

Dynamics: The actions are linked to the different parameters: extending fingers separately to move in certain directions; swiping fingers to move (jump) forward. The two players need to negotiate when to take certain actions to achieve an expected outcome, and instruct each other on the behavior (e.g. how much to swirl the fingers).


Aesthetics: Resorting to the taxonomy of Aesthetics which came up with the MDA framework, the current game involves Challenge (moving the robot to a precise location), Fellowship (active asymmetric collaboration with the other player), and Discovery (exploring different places and possible obstacles in the scene).

Evaluation: Interview about the general clinical solutions in practice, and experts; opinions on the current prototypes.


Results Expert Interview


The collaborative mechanism is fine for the training of rehabilitation.


However, the interaction semantics is still a bit abstract. It is hard to match the actions with the changes on the robot. It is better for the game to bring reflections on real-life actions, making patients feel that what is done in the game can also be done in their daily life.


In the rehabilitation groups, patients usually share similar symptoms. Therefore, the game should consider providing options for both players using left/right hands.


Even in the same group, the competence levels could be slightly different. The game should consider balancing this, for example, making the task for the stronger side a bit tougher.

Second Iteration

After employing the feedback from the expert interview, a new game was developed. The goal of the game is to reach the final destination. However, instead of creating competition between the two players/hands, the new game still employs the collaborative mechanism, by asking the two players to work together to remove the obstacles on the way. The necessary tools are ready for hand interaction with special prerequisites. The aforementioned new training modes of the three parameters are employed meticulously.

1. A pillar is moving horizontally in the way. Player L needs to grasp the moving pillar at the left point and hold its position. Player R then grabs the static brick on the side and puts it beside the pillar to block the pillar from moving. Then both players release their hands, and the character keeps moving forward. (Training modes: L-Speed of Movement; R-Range of Motion)

2. A huge net with a base is in the way. Player R needs to grasp the moving ball on the right side, put it in the middle and release it. Then player L grabs the hammer on the left side, moves it to the middle, and uses it to punch the ball. The ball receives the force, moves forward, and bumps into the net to remove it. (Training modes: L-Range of Motion; R-Speed of Movement)

3. A door is in the way. Two rings appear in front of each hand. Both players need to move their hands forward, put their corresponding fingers through the rings, and drag them down (for player L, the index and the middle; for player R, the thumb and the pinky). When all the rings are dragged down, the door falls and the character keeps moving (Training modes: L&R: Fractionation)

4. Two stacks of bricks appear in the way. Each player needs to remove the corresponding stack brick by brick, by grabbing them, moving them aside, and dropping them. When all the bricks are removed, the character is free to move forward. This obstacle raises the intensity of training by employing repetitive movements (Training modes: L&R: Range of Motion)


Results User Test & Expert Interview


The player experience was measured with the Play Experience Inventory (PXI). For all the items in the construct Goals & Rules, no negative feedback was received. However, there are several negative comments regarding the construct Progress Feedback: some players found they were not informed sufficiently about the game’s progress. Moreover, for the construct Autonomy, neutral and negative scores exceeded half. Many players could not

The overall enjoyment measurement adopted the category Interest/Enjoyment from the Intrinsic Motivation Inventory (IMI). these items got very few negative scores, except the last one, which got 20% of negative scores.

To justify the relatedness of the design concept from a clinical perspective, another expert interview was conducted. The experts expressed the following opinions.

1. This collaborative framework would work for the current rehabilitation system. The game is providing patients with abilities that resemble those in real life. The game should be motivating and engaging for patients’ training.

2. The game would suit the current structure of rehabilitation groups at the center. What people are currently doing in groups is interesting but not reflective activities like playing football with fingers, and this game would bring something different to them. It would be nice if, in further work, there could be an extension for more people involved in one game.

3. The game would be nice for active exercise. In terms of passive exercise, there are external forces that assist or urge people’s actions. The collaborative setting is still suitable, but the parameter of strength can be definitely involved in the game as well.


Competence Development


In this project, I have acquired the capabilities of setting up Meta-gloves using the external service application, implementing them as the interaction technique in Unity, and extracting gestural data instantly for performance analysis. I could handle the hands’ models as common objects in the virtual scene, and utilize my skills in Unity to make them interact with the environment.

Besides, I have summarized the advantages of them over Leap Motion, which achieves hand tracking in a different way. First, Leap Motion is camera-based, which means that in its view, the different points must not overlap. On the contrary, Meta-gloves enable more gestures to be recognized regardless of the complexity and orientation. Second, Leap Motion could only track a limited space above its camera, while with the base station of Steam VR, the Meta-gloves could move around in a large area, even at locations that are not visible. These comparisons help me distinguish the features of different interaction techniques, and reflect on the contexts that each of them would suit.

Regarding U&S, I have employed various ways of studying the users, from literature research, benchmarking (in the retake), to expert interviews and player experience evaluation. I have got more familiar with the relevant tools, and in conjunction with T&R, I have learned more about how a certain type of technology could be used to suit a certain context (in this case, using the precise tracking of the gloves to achieve training of different parameters and recognition of basic hand gestures).

Professional Identity & Vision

As “an interaction designer with a technological perspective”, I am glad to see that I have discovered a number of intrinsic features and advantages of the Meta-gloves, and used them properly to create the prototype and make the FMP proposal. In this project, there is a combination of the technological features with the users’ requirements, which has broadened my view on the application criteria of emerging technologies.


When doing that project, I found that as expected, I enjoyed working with the Meta-gloves and exploring the features of this awesome technology. However, I also found that working on hand rehabilitation was not appealing to me. After considerate contemplation, I think that compared to those domains that have rigid and well-established protocols, I prefer to use emerging technologies in exploring new ways of interaction in places where the technique is completely unused. Therefore, in my FMP, I have chosen to work on rhythmic music games with Meta-gloves, as they have not been applied to that domain and the advantages are evidently able to achieve that.


I'm always looking for new and exciting opportunities. Let's connect.


bottom of page