It’s not a new problem.
Fantastic question. It’s one we should all be asking ourselves: how far can you trust AI for the quality of its responses, or it’s “intent”?
Here’s the thing: it applies to more than AI. How do you know whether to trust typical search results? Or what you find on social media? Or what you find on random websites online?
The short answer is: you can’t! At least not without further effort on your part.
The truth versus AI (and others)
You can’t always tell if an AI answer is right, but that’s true for search results and websites too. Ask a different AI, compare with a search engine, or look at plain links. Use what you already know, and when something doesn’t smell right, dig deeper.
Getting answers from AI
Recent years have seen a revolution in how we find things online. Rather than searching for answers with a traditional search engine, more and more people are turning to AI chatbots like Claude and ChatGPT to get answers. Search engines like Google are encouraging this by providing AI-based summaries above traditional search results. Those summaries are positioned as The Answer, implying that there’s no further need to click through to other websites.
One characteristic of AI-generated responses is their confidence. They make it easy to trust by providing information that feels authoritative because of the way it’s written. In some ways, they’re an excellent example of the power of the written word.
The problem? Confident-sounding or not, the answers are sometimes wrong.
And if you’re not already somewhat familiar with the subject matter yourself, it’s almost impossible to tell.
Help keep it going by becoming a Patron.
Checking AI’s work
I’ve written about my concerns before. It’s not that people are using AI to get answers; it’s that they’re not the least bit skeptical about whether those answers are correct.
But if you are skeptical, what then?
Here are some ideas for checking AI work.
AI vs. AI
Give the same question to a different AI and compare the answers. This is interesting to me because it exposes the different approaches used by the various tools. That they will differ to some degree isn’t good or bad; it’s just worth knowing about. And when the answers conflict, you know your skepticism was completely warranted. All the major AI chatbots — ChatGPT, Claude, Copilot, Gemini — have free versions that are perfect for this.
In the example above, I asked four different AIs, “How do I get Windows 11 26H2?” Each gave a similar answer. Had the answers differed significantly, perhaps none of them would be trustworthy, and a different approach to getting your answer might be more appropriate.
AI vs. search summaries
Give a question to the AI you use. Then ask the same question to one of the popular search engines that include summaries or answers as part of its results. In theory, this might be similar to the previous technique, but search summaries are often designed to serve a slightly different purpose by being more concise and to the point.
A traditional Google search returns results beginning with an “AI Overview”. Compare AI and Google responses as well as the additional results shown below the summary.
AI vs Plain Search Results
Give that same question to a search that includes no AI or summarized component at all. Use something like udm14.com, which still uses Google but returns responses in the old “nothing but links” style.
This technique is likely the most reliable and the most work. Because it presents only links to sources of information, the onus is on you to ferret out the information you’re looking for (from presumably authoritative sites).
YouTube is the second most popular search engine on the internet. If video content appeals to you, it’s another source of possible confirmation.
Here’s something important to realize with all of the approaches above: they’re not unique to AI.
Related
How Much Can I Trust Information on the Internet?
There's so much information at your fingertips online, but how much of it can you believe? I'll show you how to spot trustworthy sources, why seeing something everywhere doesn't make it true, and four steps to help you sort fact from fiction.#5639
It has always been thus
The problem we’re addressing here is nothing new. We’ve always been at the mercy of the results we find online, and it’s always been on us to be skeptical of what we’re told. AI serves as a starker example of what that means, especially as early versions of AI were particularly and confidently error-prone.
The good news is that AI provides an additional channel to help research and confirm or deny the information we’re attempting to vet. If your go-to research tool is a traditional (or specialized) search engine, you can now also use AI as a high-level path to gather more information on the subject. In particular, AIs that cite their sources of information seem to be good at finding sources we might have overlooked in the past.
But here’s the thing: even if you’re not using AI at all, how do you know what to trust? Google might serve up links to sources of information related to your query, but that doesn’t mean those sources are accurate. They could be just as confidently wrong as AI.
Doing the work to compare multiple independent sources of information has always been important.
What I do: the sniff test
Your question started with “If I do not understand it…”, meaning you don’t have enough knowledge to know whether an answer is accurate or not. Indeed, that’s a problem, and everything above applies.
But if you have even a little bit of knowledge, that can go a long way to calibrating your comfort level with the answers you’re given.
I use AI a fair amount, and when it comes to tech, it’s pretty easy for me to see whether something makes sense. I may not know the answer going in, but typically answers are built on existing knowledge I have. If an answer fails to make sense — if it fails the “sniff test” — I know to dig deeper or look elsewhere.
By and large, I think we often have some knowledge or understanding that can be helpful in the same way, regardless of what we’re looking for.
If you don’t have the knowledge for even a gut feel, then it’s even more important to do the work and dig deeper. Put simply: don’t trust any answer you don’t take the time to confirm in multiple places. Don’t use the sniff test if you don’t know the smell to start with.
Do this
Be skeptical about any information you find online. Rather than seeing AI as an easy source of information, look at it as another tool you can use to confirm whether something is legit, regardless of its source.
Subscribe to Confident Computing! Less frustration and more confidence, solutions, answers, and tips in your inbox every week.
I’ve asked AI questions before, usually to test if my thinking is correct. I know the answer that I think iz correct. I’m using AI to check my answer. But sometimes, I don’t know the answer. I will review the links it claims it got its answers from. But if I can’t figure out how it got it’s answer from those sources, I will challenge it. I’ve also challenged it to provide me with an exact reference to the income tax act or a Canada Revenue Agency (IRS for you Americans) publication, for example. Often when I have to challenge AI, it comes back and tells me that I caught them making assumptions based on X, Y, and Z.
Most recently it was a provision in the traffic act regarding emergency vehicles. When the provision cited had nothing to do with emergency vehicles, I challenged it. It admitted that it didn’t bother to check if the reference was still current. It admitted it was an old reference and the updated reference was something else.
I’m learning to challenge AI’s answers to make sure that it is doing proper research and not just being lazy.