Restriction lift date: 2030-12-31
Mitigating unconscious bias in information systems design: a design science research approach
Loading...
Files
Date
2025
Authors
O'Reilly, Katie
Journal Title
Journal ISSN
Volume Title
Publisher
University College Cork
Published Version
Abstract
Information Systems (IS) continue to transform modern organisations as well as our daily lives. In an organisational context, the development of conversational agents, for example, has generated forms of new business value for companies across sectors such as marketing and customer support (Seeger et al., 2021). In an individual context, social media platforms like Instagram have enhanced global connectivity and brought new opportunities for content sharing across networks (McCarthy et al., 2023). How-ever, despite the significant improvements IS has brought, unintended consequences can arise that lead to adverse outcomes. Recent issues such as high rates of problematic smartphone use and technostress, the spread of disinformation, and a widening digital divide highlight the adverse outcomes that can arise for specific user groups (Kroeze, 2024; McCarthy and Busch, 2024).
In light of these issues, there is an urgent need for IS research to reflect on the ‘dark side’ of IS design and the sources of potential unintended consequences for different users. One significant source of adverse outcomes is the embedding of unconscious biases in IS design (Barocas and Selbst, 2016; Akter et al., 2021). Unconscious biases are defined as the deeply ingrained within us which often operate without our aware-ness (Brownstein, 2019). While many of our daily actions and decisions are influenced by our biases that we are unaware of, they can nevertheless, make us act in a discrimi-natory way towards others around us (Pritlove et al., 2019). When developers allow unconscious biases to seep into the design process of IS, this can negatively impact users as unmonitored biases seep into a design, leading to user groups being discrimi-nated against (Ferrer et al., 2021)Unconscious bias in IS design is a significant chal-lenge, and as technology continues to expand and adapt, it is important that we work to prevent users from being adversely affected.
This thesis, therefore, aims to investigate unconscious bias in IS design and how we can work to mitigate its effects. The research questions assess the types of bias present in IS design, how we can mitigate these biases and how the elements of a Design Sci-ence Research (DSR) artefact can be combined in a conceptual model. Empirical re-search is conducted using qualitative and DSR methods to address our research ques-tions. The research was broken into two stages. The first stage of research, Stage One, consisted of qualitative interviews and a focus group to explore participants' under-standing of unconscious bias and experiences in IS design. Stage Two then consisted of design science research workshops, which aimed to formulate a novel solution to the real-world problem of unconscious biases (Hevner et al., 2004; Peffers et al., 2007). Specifically, the workshops facilitated the iterative design of an artefact to pro-mote an awareness of unconscious biases and challenge participants’ underlying as-sumptions.
This thesis makes several theoretical and practical contributions to the fields of IS de-sign and responsible IS research. Stage One of the research revealed prominent forms of bias that appeared in conversations around IS design, such as assumption bias, au-thority bias, bias blind spot, and stereotyping. Key aspects of ethical IS design were also uncovered, such as transparency, ensuring accountability, technology communica-tion, and diversity. Design principles were developed based on Stage One of the re-search to guide the development of a solution that would help mitigate the negative im-pacts of unconscious biases for different user groups. Stage Two of the research pre-sents further significant contributions from this research in the form of an artefact, which was developed as a novel solution to address the issue of unconscious bias in IS design. ‘The Unconscious Bias Self-Assessment Rubric’ helps guide developers through the design process of IS, ensuring that they consider the impact of their deci-sions throughout key stages of design. The rubric’s formation was informed by the design principles from Stage One in addition to insights from Cognitive Dissonance Theory (Festinger, 1957) and The Theory of Communicative Action (Habermas, 1981). Together, these theories can help explain how the rubric helps uncover and challenge our biases (cognitive dissonance) and how we engage in dialogue to over-come these biases (communicative action). The importance of empathy is also high-lighted in this stage of the research, and how IS developers feel empathy towards users is essential when addressing biases. Questions and topics that triggered deep conversa-tions around biases are also highlighted as well as ethical considerations practitioners emphasised during discussions. The practical contributions of this thesis lie in providing actionable insights into issues around unconscious bias in IS, specifically offering the UB-SAR and design principles that can be directly applied to industry practices, highlighting the importance of addressing unconscious bias in IS design.
Description
Keywords
Unconscious bias , Information systems design , Design science research , Ethics
Citation
O'Reilly, K. 2025. Mitigating unconscious bias in information systems design: a design science research approach. PhD Thesis, University College Cork.