Obsurvant ads

AI survey fraud in online research is hardly news to anyone working in surveys. But hearing it discussed on BBC Radio 4's More or Less last month really does show the extent of the issue. If you’ve not listened to it before, More or Less is a show that explains, questions or debunks numbers and statistics across the news and everyday life. “Long story short, in the age of AI, if you see a survey response that you can’t triangulate with other data and trends, then you need to be very wary” Annette Jäckle, a Professor of Survey Methodology at the University of Essex highlighted that AI has become so good at impersonating real respondents that telling them apart is nearly impossible. Nearly…. One of the many layers we use to ensure reliability within Obsurvant's data is through our partnership with Roundtable. We've written about how AI agents behave in surveys, why they're getting harder to detect, and what genuine behavioural detection looks like in practice. Link to article: https://lnkd.in/embanXK9 🤝 Matt Hardy Mayank Agrawal

My final post on the topic of polling and the Assisted Dying debate…but based on my recent opinion piece in Research Live this article in the Mail Online felt too timely not to comment on. A prime example of hefty claims pulled from a questionable research methodology with a clear objective behind the line of questioning. Before we run a poll the survey has to pass through a stringent QA process. We ran the questions that these headlines were pulled from through the Obsurvant methodology and our specially trained compliance agent. Both raised concerns around the overall survey and the question framing. For example: Q2. Do you agree or disagree with the following? > “I would not want my MP to support a law pushed through Parliament without full scrutiny and approval by both the House of Lords and House of Commons, and which wasn’t in the Government’s election manifesto.” Our compliance checks flagged that: 'It contains multiple embedded assumptions presented as a single statement: that the law was 'pushed through' (loaded pejorative), that it lacked full scrutiny, and that manifesto absence makes it illegitimate. Respondents who agree with any one element but not others cannot express a nuanced view. It is double-barreled, loaded, and predetermines a negative headline about the legislative process.' This article is backed by questions that are incredibly leading, have consistent oppositional framing statements that prime respondents, and provide a lack of context. As a result, the answers veer away from the wider data and extensive research on assisted dying. But they do generate eye-catching statistics… Even some in the comment section agreed, which is relatively rare to see on the topic from the Mail Online, with a number of amusing responses to the article. My two favourite being: “Strange or if you like a very ODD poll, I do not know anyone who is against assisted dying so the poll must have been carried out in a nunnery” Or “No doubt the question was loaded to give the answer wanted. Whoever manipulated this outcome….we see through you” I’m not sure how the respondents were sourced for this larger sample size, however what is clear is that credibility of polling data goes far beyond standard ‘data quality checks’. Headlines can backfire. It doesn’t look like this one will, however for data to be truly reliable each stage of a project should be scrutinised, and the first step is how the questions are asked. (Link to the Research Live article in the comments below)

What seemed like an outrageous conspiracy 10 years ago, suddenly seems a lot less far-fetched. Supporters of the Dead Internet Theory have been claiming since 2016 that the internet consists largely of bots and automated content. This video is an extreme case, but illustrates how rife social media is with fake accounts trying to manipulate and influence social interaction online. Throw the ability to earn money anonymously in there and then imagine the world of online surveys. My question to PR professionals and opinion pollsters is: If you can’t be sure that the person you are having a political argument online with is real, then how certain are you that the survey data set you derived headlines from consists of genuine human answers? The barrier to entry for survey fraud has dropped, anyone can build an AI agent which can take surveys. On top of that, residential proxies, link manipulation, VPNs (to name a few options), let fraudsters pretend to be someone they're not. The data set looks great but the people behind it aren’t real. Last year we saw people charged in the USA for a $10 million survey fraud. We have the recent YouGov case inflating the rise of Christianity in the UK. All fraud. The speed of change is terrifying, the quality checks we had four months ago would not be enough now. We have to constantly review and evolve our methodology, and we’re good at it. If you're running PR research and putting stats out to the press, it's vital to ask the right questions. Headlines need humans.

“A researcher might design a clever new trap that catches today’s models, but model development moves so fast that the fix is likely obsolete within months.” It’s encouraging to see the attention from the national press on the threats of AI-driven fraud in polling and it shouldn’t surprise anyone who’s been close to this space. Another interesting article in The Guardian by Sinéad Campbell over the weekend stating: ‘Experts say paid participants are using automated tools to generate unreliable survey responses at scale’ The barrier to entry has dropped, AI literacy has increased and the tactics to commit survey fraud are getting more sophisticated and they’re moving fast. This isn’t the end of opinion polling but it definitely does expose who’s doing it properly. At Obsurvant it has never been about a single solution. We’ve put in place cascading layers of protection across multiple behavioural checks, intricate security signals, all working together. Most importantly we’re constantly reviewing and evolving them. The quality checks we had four months ago would not be enough now. If you’re running research online you should be asking your suppliers how they’re dealing with this. How are they identifying fraudulent AI-enabled respondents? 🦠 What is their current approach for residential proxies? 🏴☠️ How often are they reviewing and updating their methodology? ♻️ What happens when something gets through? 🌊 If they can’t answer those clearly, that’s a red flag and they possibly don’t fully understand the problem; let alone how to fix it. This is only going to get harder but ultimately, no matter how advanced this all gets: Headlines need humans. If you’re running PR research and care about the validity of your data, please get in touch. There is an effective way to approach data quality, and we’ve been focused on it since day one of Obsurvant. (Referenced Guardian article in the comments)

Before we send any PR data to clients, we pressure test it as a team. For example, if we suddenly see a disproportionate number of 18-24 year olds going on cruises, something doesn’t add up. These skews in the data are the kinds of signals we interrogate and question. We only deliver data with confidence. But even with robust security and quality controls, advanced fraud and problematic respondents are an ongoing battle across the industry. Despite researching younger audiences daily, in particular across nightlife and culture, I know I’m definitely out of touch, but I don't think I’m that off… That’s why reports referencing youth survey data, like the recent Bible Society's findings, was of particular interest. Many others, including David Voas highlighted by the BBC, were quite rightly questioning the results. Some of the figures, in particular around younger audiences, didn’t align with what we’re consistently seeing in our own research and wider trend data in the UK. A few examples from the report were: 40% of 18–24s say they pray at least monthly 21% of men aged 18–24 are regular churchgoers 19% read the Bible weekly outside of church These are striking numbers, and as a result they gained traction. But they just seemed off. This case highlights a broader point that methodology matters as much as messaging. How respondents are sourced, how they are validated, and how questions are framed all have a material impact on outcomes. Fair play to YouGov for engaging openly on this and investigating it further. As an industry, we need to be more transparent about how data is collected, especially when findings cut against wider trends. Opinion polling has never been more influential. Survey fraud completely undermines trust. Ultimately, headlines need humans. Get in touch if you’d like to chat about data quality in PR research.

Looking for a new research partner? ✅ The right audience, the right answers, fast. ✅ Answer three quick questions and we’ll send a tailored quote within the hour.