Weak programmers need not apply, LLMs welcome! Survey screening in the AI era

Simple item page
dc.contributor.authorRyan, Ita
dc.contributor.authorRoedig, Utz
dc.contributor.authorStol, Klaas-Jan
dc.contributor.funderTaighde Éireann - Research Ireland
dc.date.accessioned2026-01-27T00:31:39Z
dc.date.available2026-01-27T00:31:39Z
dc.date.issued2026-04-15
dc.description.abstractQuantitative research on software development practice often depends on the results of anonymous online surveys. However, it is difficult to verify that the respondents to such surveys are genuine developers, especially if survey responses are paid for, incentivising respondents to game the system and attracting the attention of bots and professional survey farms. One solution is to include screening questions to assess respondents' coding skills, but this introduces the risk of unfairly excluding considered responses from developers who spent time and energy on the survey. We reviewed the responses of 86 developers who were excluded via screening from analysis of a large unpaid developer survey (n=1,048). We estimated that up to 86% of the developers who were excluded were genuine developers. The advent of large language models (LLMs) casts further doubt on the use of screening questions. Investigating LLM ability, we found that five sample LLMs could answer most widely-used screening questions. Powerful LLM-based survey-taking tools now exist. We researched current survey screening techniques and found that they are susceptible to fraud via LLM use, bots and survey farms. Recruitment strategy may be a better screening technique. We recommend that if survey respondents are compensated only known developers should be invited. Surveys that are widely distributed, e.g. on social media, should not compensate respondents. Instead, researchers should focus on good survey design and sustained, imaginative recruitment strategies.en
dc.description.sponsorshipResearch Ireland|23/US/3939
dc.description.sponsorshipResearch Ireland|13/RC/2094_P2
dc.description.sponsorshipResearch Ireland|13/RC/2077_P2
dc.description.versionAccepted Version
dc.format.extent13
dc.format.mimetypeapplication/pdfen
dc.identifier.authororcidRyan, Ita§0000-0001-6844-6365
dc.identifier.authororcidRoedig, Utz§0000-0002-4020-0889
dc.identifier.authororcidStol, Klaas-Jan§0000-0002-1038-5050
dc.identifier.citationRyan, I, Roedig, U & Stol, K-J 2026, Weak programmers need not apply, LLMs welcome! Survey screening in the AI era. in Proceedings of the 48th IEEE/ACM International Conference on Software Engineering. Proceedings - 2026 IEEE/ACM 48th International Conference on Software Engineering, ICSE 2026, pp. 2622-2634, 48th IEEE/ACM International Conference on Software Engineering , Rio de Janeiro, Brazil, 12/04/26. https://doi.org/10.1145/3744916.3787789
dc.identifier.doi10.1145/3744916.3787789
dc.identifier.endpage2634
dc.identifier.isbn9798400720253
dc.identifier.otherORCID: /0000-0001-6844-6365/work/203446680
dc.identifier.otherORCID: /0000-0002-1038-5050/work/203446726
dc.identifier.otherORCID: /0000-0002-4020-0889/work/204225925
dc.identifier.startpage2622
dc.identifier.urihttps://hdl.handle.net/10468/18472
dc.language.isoen
dc.relation.ispartofseriesProceedings - 2026 IEEE/ACM 48th International Conference on Software Engineering, ICSE 2026
dc.relation.urihttps://www.scopus.com/pages/publications/105051870429
dc.rights© 2026, Copyright held by the owner/author(s).
dc.rights.accessrightsopen access
dc.rights.licensenameAttribution 4.0 International
dc.rights.urihttps://creativecommons.org/licenses/by/4.0/
dc.statusPeer reviewed
dc.subjectSoftware developers
dc.subjectSurvey research
dc.subjectSurvey screening
dc.subject[ComputerScience]
dc.titleWeak programmers need not apply, LLMs welcome! Survey screening in the AI eraen
dc.typeConference item
Files
Original bundle
Now showing 1 - 1 of 1
Thumbnail Image
Name:
3744916.3787789.pdf
Size:
866.59 KB
Format:
Adobe Portable Document Format
Description:
Published Version
Download
Collections
School of Computer Science and Information Technology