Understanding personalised auditory-visual associations in multi-modal interactions

Thumbnail Image
3462244.3481277.pdf(702.48 KB)
Published Version
O'Toole, Patrick
Journal Title
Journal ISSN
Volume Title
Association for Computing Machinery (ACM)
Published Version
Research Projects
Organizational Units
Journal Issue
Can we sharpen our auditory and visual senses and better understand the relationship between these modalities to benefit our interactions in human-computer interfaces? This research paper proposes a framework to understand auditory-visual associations and explore the impact of emotion, personality, age and gender in understanding information from both modalities. Studies into the areas of emotion and personality as well as their association with the auditory and visual senses have increased within the fields of psychology, neuroscience, affective computing and human-computer interaction (HCI). From a HCI perspective, advances in technologies and machine learning techniques provide a new way to understand people and to develop systems where computers work along side people to help develop efficient interactions and clearer perceptions of our environment. The proposed framework will be developed along side a personalised auditory-visual interface that can be used to provide intelligent interactions with users that can help them learn from efficient associations between their senses. This research can be used to create personalised auditory-visual-emotion-personality profiles that can be use in adaptive musical teaching platforms, as well as mental health and wellness applications for more personalised care programs.
Auditory-visual associations , Multi-modal interactions , Music , Synaesthesia , Machine learning
O'Toole, P. (2021) 'Understanding personalised auditory-visual associations in multi-modal interactions', ICMI '21: Proceedings of the 2021 International Conference on Multimodal Interaction, Montréal QC, Canada, 18-22 October, pp. 812-816. doi: 10.1145/3462244.3481277
Link to publisher’s version