Do you get advice from AI? We need to talk.
AI is good at finding answers, and it’s just going to keep getting better. Even if the pace of improvement slows down, we’re very unlikely to deal with a worse AI soon, unless regulation and safety measures become so extreme they move the needle backwards.
However, answers we get from AI aren’t all the same, and identifying which type you’re looking for determines how you treat the answer.
There are mainly two types: The first is information, and the second is consultancy.
When asking for the right size of a screw, the cheapest music streaming app subscription, or how to disable video autoplay on Facebook, the answer is information.
AI is great at finding information online and organizing it, saving us the time of juggling several websites, making comparisons, and connecting the dots ourselves. Since web search became an integrated skill of each frontier model, cutoff times and hallucinations became much less of a problem.
AI still can misinterpret some sources, and it’s always a must to check the sources yourself, but generally, AI apps have become more reliable in this regard.
On the other hand, when asking for the best path for your business or gift recommendations, the answer is consultancy or advice.
The difference lies simply in whether you’re asking for a fact that can be proven right or wrong, or asking for an opinion or an analysis. In other words, would the core of the answer to this question be the same from different AI models and humans or not?
Advice, consultancy, or ‘thoughts’ that come from AI aren’t necessarily bad. In fact, in many cases, they’re just stating obvious best practices or basics people tend to overlook. However, understanding the inherent flaws in the data on which this advice is built and how it’s interpreted and utilized helps us have a healthier perception. Some of these flaws are data-related, many of which are problematic regardless of AI.
Let’s break down the process an AI model goes through to provide a consultancy answer into three stages, and highlight the flaws that might affect each of these stages.
Information & data
↓
Interpretation & conclusions
↓
Recommendations & consultancy
These three stages don’t necessarily happen in this order, and sometimes two of them happen at the same time, but for the sake of simplification, let’s assume the process is as simple as that, and look at the flaw threatening the integrity of each one.
Stage 1 – Data collection: picture imperfect
This stage doesn’t start with your prompt. It includes the data that an AI model was provided with for training, the data the model collects on the go via web search and other connections, and, of course, the data you provide as references or contexts.
While AI has been trained on tons of data, it’s worth noting an obvious fact: AI only has access to digital data. In other words, the only reality AI knows is that with a digital representation, and what gets to be digitized is neither random nor neutral.
For example, if we look at economic data, poor countries tend to have wider shadow economies, and thus a smaller digital footprint of their economic activities. So when AI is trying to understand economic concepts, it has more data on rich countries, and the little data it has on poor countries doesn’t only lack coverage, but lacks representation of reality altogether.
On an individual level, the same bias applies: people rarely post detailed case studies on how their business failed, or how they ended up in debt and burnout.
In addition to underrepresenting reality, many datasets suffer a well-known, systematic flaw: survivorship bias. This is when the data collection or sampling method leads to surveying only one type of cases.
A simple example is looking for successful businessmen who dropped out of school and using the results of this search to evaluate the decision of dropping out. The results would probably state that dropping out is a great decision; I mean… look at all those great billionaires in their black T-shirts. The problem with this data set is that it ignores all the ones that didn’t make it.
Stage 2 - Interpretation: ghost of data past
When interpreting the data that came from the previous stage, another flaw is possible.
Much of the evidence used in many fields where a ruler doesn’t help is built on observations of data and drawing correlations, rather than actual confirmed detection of causation.
To understand this, let’s visit the concept of A/B testing. This is a common optimization tactic in marketing that is very similar to scientific research: You take two inputs, keep every single aspect of both identical, and change only the factor you’re trying to study or measure its effect.
However, in marketing, many glorify “best practices” guides that different advertising and social media networks release. These weren’t built using the rigorous process above. The wording of these guides, or at least the research they cite, is not misleading.
For example, these guides would say there is a correlation between shorter content and higher engagement, but we don’t know that this causes that. Both could be results of a different cause, like brand size, which means people are more likely to engage with it, and at the same time it has enough resources to hire better writers able to express campaign messaging in fewer words.
Not to mention that these ‘bibles’ of advertising are mostly built on English campaigns, not taking into consideration, for example, the confusion fewer words could cause in some languages.
If you ask an AI to give you a work plan for a blog, it’ll build one on the patterns it spots across existing, successful, indexed blogs. The plan will include posting frequency, social media channel matrix, clusters of articles to rank for SEO, etc. However, this plan is just a description of the average of what similar blogs are doing, not what they’re doing to succeed.
Also, AI—and humans, to be fair—don’t draw correlation or causation from their own analysis all the time, but rather by copying others’ analyses or descriptions of correlations and causation, falling into the same flaws of the first stage described above. So theoretically, AI wouldn’t be giving you bad advice, just copying someone else’s bad advice with higher confidence.
Stage 3 - Recommendation: you’re a genius!
This stage is when the never-ending answer pops up, whether it comes as a fancy-designed deck or just text full of bullet points and tables. Usually looks like a well-knit, solid plan, although it might be complete rubbish.
Three main issues might make the plan you’ve received not as useful—or elegant—as it seems.
The first one is context blindness. AI can give you a solid marketing plan for your business but miss what your business actually does, either because you didn’t mention it or because the context window got so jammed that it missed that token.
For example, a minimal design would work for an Apple campaign, but would it work for a Disneyland campaign? Even if we want to look at similar brands, a Nike campaign element won’t necessarily work for Adidas. Something funny by one brand can be cringe by another. A saving promise is seen as a good deal by one brand, but a source of suspicion by the other.
These comparisons get trickier when your business or project is more niche, and you can rarely find similar projects with many elements in common.
This is not only an AI flaw, but it could be you. While asking for advice to deal with your flu or insomnia, you might miss mentioning some condition that sounds irrelevant to you, but for AI, or a real doctor, it would change the whole context.
Second is false precision; this is especially relevant when AI’s response includes numbers or probabilities. While sourcing numbers from other sources is a legitimate way to get more accurate answers, numbers the model itself calculates may provide a false sense of accuracy.
The third and most critical flaw at this stage is sycophancy (fancy word for sucking up). AI models are commercial products, after all, and customer retention is a contributor to how they develop and evolve. Nothing satisfies a customer like agreeing with them, especially on their most unreasonable ideas they can’t find humans to affirm.
Sycophancy might extend and compound beyond one chat, flow, or task. Frontier models now keep a memory of past conversations, which is a very useful feature most of the time. However, this might mean that sycophancy kicks in even if your ideas that an AI is agreeing with are mentioned in the same chat or flow.
So, what to do? How to benefit from the great capabilities of AI without falling into these traps?
Simply ask yourself: is the answer I’m looking for information or consultancy? If it’s consultancy, use the answer as a brainstorming process or starting point, rather than a final output.
A plan you get from AI is a plan in which you might invest your and others’ effort, time, and money. And like any investment, you should do your own research.