University thesis:
Dissertation, Universität Bremen, 2021
Footnote:
Description:
Computer systems have been continuously evolving since the first computer invented in the 1950’s up to the revolution of smart devices, which we are witnessing nowadays. Consequently, the number of computer users has been exponentially increasing, from only few experts to hundreds of millions users: novices, intermediates, and experts. Human Computer Interaction (HCI) has been widely discussed for improving the interaction between a computer system and its users. Interdisciplinary research evolved for integrating psychological studies with HCI studies to model cognitive skills during HCI sessions, and thus, to design an effective User Interface (UI). Due to the high dynamic nature of HCI sessions, a dynamic UI adaptation is required when user performance is impaired. User performance impairment in an HCI session should be detected "online", i.e. during the system use, for an appropriate UI adaptation; It is valuable to detect the reason which caused the performance impairment during an HCI session, so-called HCI obstacle, because different UI adaptations are required to compensate for different HCI obstacles. Different HCI obstacles impair several human processes, namely perception and cognition processes. Consequently, several human processes can be impaired during an HCI session. Human processes can be tracked by recording appropriate multimodal data during an HCI session: brain activity data to track the cognition process and behavioral data, depicted by encoded user actions, to track the user behaviour. Modeling of multimodal HCI obstacles is very important because there is no "silver bullet" UI adaptation which could be activated by default. In other words, while different HCI obstacles, e.g. memory-based and visual obstacles, impair the user HCI performance, different UI adaptation mechanisms will suit each individual HCI obstacle, because an UI adaptation should appropriately compensate for the impaired human process. Moreover, a good UI adaptation mechanism for a specific HCI obstacle can be detrimental if applied for other HCI obstacles. In this thesis, a novel user modeling based cognitive adaptive system is proposed. The adaptive system dynamically models memory-based and visual HCI obstacles during system use, and accordingly applies the suitable UI adaptation mechanism for each detected HCI obstacle. Appropriate machine learning models are used for multimodal HCI obstacles detection. The multimodal obstacle detectors outputs are passed to an overarching probabilistic model to decide for the most suitable UI adaptation mechanism. The proposed approach is dynamic in consecutive HCI sessions, i.e. it treats not only persistent HCI obstacles which remain impairing the user performance in consecutive HCI sessions, but also volatile HCI obstacles which suddenly appear or disappear in the HCI sessions. Moreover, the model is dynamic in case of wrongly decided UI adaptation for an HCI session, where it recovers in the subsequent sessions. The approach is systemically evaluated through data collected from many user studies, and the experimental results show that our approach: 1) models the HCI obstacles well, where it can simulate the user behaviour under different conditions, 2) detects multimodal HCI obstacles in consecutive sessions, and 3) dynamically learns from consecutive HCI sessions to accurately adapt the UI.