top of page

DeepTouch: Displays and Input Techniques for Simulated Reality

This is my M1.2 Research Project. Working with Dimenco, I had access to hardware like stereoscopic display and Leap Motion Controller. This comparative study explored how the different types of displays and how the input techniques influence users’ performance and experience in non-gamified and gamified contexts.

The demonstration video is at the end of the page.


Simulated Reality (SR) brings the concept of immersive and interactive experience without the need for unnatural peripherals such as headsets and controllers. To achieve this, SR uses specially manufactured stereoscopic screens that present unique and parallel images for each eye, precisely the same way humans’ eyes perceive the world. For some applications, SR also uses Leap Motion Controller to achieve natural and intuitive interaction with the virtual environment.

Research Question

As a newly-created technological concept, research and development around SR are fundamental and relevant. Most previous works focused more on the application of SR, e.g. 3D printing and digital twins. However, little research has been done on the essential characteristics of the technologies.

Based on such, this study aims to address the research problem: How do the different types of displays (2D and 3D displays) and input techniques (mouse control and hand tracking) influence users’ performance and experience?

Research Method

Target acquisition is one of the most fundamental tasks in Human-computer Interaction. While setting it as the main task of the study, I applied Fitts' Law and relevant parameters for measuring user performance, and used questionnaires to measure user experience in a quantitative way.

To further segment the task, I also induced the context of the tasks: non-gamified and gamified, which were studied separately.

Non-gamified Prototype

Four conditions were implemented by two types of displays and two types of input techniques.

A standard Fitts' Task was implemented in a 3D environment. Users use mouse cursor or virtual fingers to click the targets. After each round of nine clicks, the size and distance of the targets are randomized. There are a total of six rounds in each session.

Gamified Prototype

Similarly, four conditions were implemented for this context. A narrative was construct for gamification. Users use mouse cursor or virtual finger to press the targets (bubbles). After each successful click, a gadget is added to the scene. After one round of nine clicks, one part of the scene is decorated nicely with trees, flowers, floors, etc., and the camera is switched to a new place. Samely, a total of six rounds is implemented in each session.

Performance Evaluation

Experience Evaluation

Data to be collected:

Throughput (movement time)

Reaction time

Mean velocity

Peak velocity

Methods of evaluation:

Mean value

Standard deviation

Positive and Negative Affect Schedule (PANAS):

Consists of ten positive emotions and ten negative emotions to be rated from 1 to 5

Test Sessions

A total of ten participants were recruited. Each of them was asked to try out four conditions. By arranging the allocation, everyone could experience both types of displays, both types of input techniques, and both contexts. After accomplishing each condition, they were asked to fill in the questionnaire once. After completing the entire process, open conversations were conducted to gather their perspectives on the prototypes and the technologies.


Performance Evaluation

  • Mouse input brings better performance (movement time)

  • When using hand tracking, using 3D display generally brings better performance than 2D display

  • Using hand tracking has a shorter reaction time

  • In both contexts, using mouse input has shown a higher stability

  • When using hand tracking, 3D display brings higher stability

Experience Evaluation

Generally, using hand tracking for input brings both higher positive affect and higher negative affect. In the gamified context, it is particularly important to combine 3D display and hand tracking (3D input) 


Lack of Proficiency in Hand Tracking: Most participants had never used Leap Motion Controller before. This probably counts for the lower performance and stability of using hand tracking for input, as well as the corresponding higher level of negative affect.

Threshold of Manipulation: The mouse has its weight and friction with the plane below, therefore, there is a time gap between the reaction of the hand and the movement of the mouse. However, the hands were in the air with no resistance and the motion could be captured almost without delay. This probably explains why such a distinct discrepancy would exist in terms of reaction time.

Occasional Malfunction: It was also observed that occasionally the participants placed their hands so high that they covered the vision of eye-tracking cameras, which caused the stereoscopy to be switched to 2D display, and then switched back. These could have resulted in the lower performance of using hand tracking, as well as the higher negative affect.when designing interaction for SR, it is important to keep the hand position lower than a specific level relative to the camera or the screen.


Competence Development: This project is my first master’s project in an XR-related field. As I had set Technology and Realization as my first expertise area to develop, I specifically focused on the novel features of Simulate Reality (SR) during the whole process. Even before determining the research question, I dived into the relevant literature, including demonstrations of the technologies and the application of them, to get a clear view on the mechanism of SR, e.g., how does stereoscopic display create depth perception and how it is technically achieved. Simultaneously, I made practice with relevant resources, to explore how the technologies functioned and what was possible through them. After such, during the implementation phase, the technological things went smoothly since I knew clearly how to use virtual hands to touch things, and how to create a stereoscopic view and adjust it to a proper proportion. In terms of development with Unity 3D and C#, which were already familiar to me, there is still much I had learned: the application of geometry in Unity, e.g., playing with absolute position and relative position, and using C# I/O system to instantly record performance data.

Professional Identity & Vision: This project is a steady step towards my professional identity of “designer with a technological perspective”. I like to explore various types of technologies, study their features and seek for space for fantastic creations. Having much experience in VR, the study around SR provides me with deeper insights into stereoscopy and in what ways the flat screen could differ from the headsets. After using Leap Motion Controller for touchless interaction, it also makes me think about the development trend of interaction technique for immersive experience: whether it is going towards a touchless and burdenless way, or it will keep the physical controllers and adopt the techniques of wearable devices for intuitive input and rich (haptic) feedback? This might be an interesting topic that I will dive into in further study.

Demonstration Video

bottom of page