AI Does Not Replace Knowledge: The Dangerous Side of the AI Revolution

Artificial intelligence is changing business at extraordinary speed.

ChatGPT, Claude, Gemini and the growing number of AI-powered and white-labelled platforms built on large language models can research, write, analyse, brainstorm, code, calculate, summarise and recommend in seconds.

I use AI. I see enormous value in it. And I believe businesses that learn how to use it properly will have an advantage.

But there is something increasingly concerning about the way we are talking about AI.

AI does not replace knowledge.

It does not replace experience.

It does not replace professional judgement.

And it certainly does not remove the need to understand what you are doing.

AI is a tool.

A remarkably powerful tool, but a tool nonetheless.

And just like any other tool, putting it into the hands of someone who doesn't understand what they're doing can produce a result that looks impressive while being fundamentally wrong.

The problem isn't always the AI. It's our trust in it.

One of the most dangerous things about generative AI is how confidently it can communicate.

Ask a question and you can receive a beautifully structured answer in seconds.

There are headings. There are explanations. There may be statistics, recommendations, calculations, references and even citations.

It looks authoritative.

That doesn't mean it is correct.

Large language models can generate incorrect information and, in some circumstances, completely fabricate information. This phenomenon is commonly referred to as an AI "hallucination".

OpenAI acknowledges → OpenAI – Does ChatGPT tell the truth?

OpenAI – Why Language Models Hallucinate

Anthropic similarly warns → Anthropic – Claude is providing incorrect or misleading responses

Google recommends users double-check → Google – Understanding Gemini responses

But there is another problem that receives less attention.

AI can give you a perfectly plausible recommendation that is simply the wrong recommendation for your circumstances.

That's where knowledge becomes critical.

If you don't understand the fundamentals of the subject you're asking AI about, how do you know when it is wrong?

How do you challenge the recommendation?

How do you recognise what information is missing?

And perhaps most importantly:

How do you proof something when you don't know what you're proofing?

I've seen what happens when AI recommendations are followed without enough scrutiny

This isn't theoretical.

We've recently seen it in digital marketing.

A website strategy for Sports Tape Wholesalers Australia involved changes to website structure and URLs. Following those changes, organic search performance deteriorated significantly, including a 42% fall in search impressions.

A change in advertising campaign strategy was followed by an even more dramatic decline, with traffic falling by approximately 95%.

The lesson isn't that AI should never be used for SEO or advertising.

Quite the opposite.

AI can be extremely useful for analysing websites, researching keywords, developing content structures, interrogating data and identifying potential opportunities.

But an AI recommendation to restructure a website needs to be considered alongside SEO fundamentals.

What URLs already rank?

Which pages have authority?

Where are backlinks pointing?

What is currently indexed?

What redirects will be required?

How will internal linking change?

What is the existing traffic contribution of the pages being changed?

What happens to Google's understanding of the website when its architecture changes?

An experienced SEO professional should be asking those questions before implementing the recommendation.

The same principle applies to Google Ads.

AI can help analyse campaigns and recommend structures, keywords, bidding strategies and targeting. But blindly implementing recommendations without understanding the existing data, conversion tracking, search intent and campaign history can be extremely expensive.

AI can make a recommendation. Knowledge tells you whether you should follow it.

When AI invents the law

The consequences become considerably more serious when we move beyond marketing.

In 2025, an Australian lawyer was referred to the NSW Legal Services Commissioner after using ChatGPT to assist with court documents. The Guardian – Australian lawyer caught using ChatGPT in court documents

The problem?

The documents included references to cases that didn't exist.

The lawyer admitted using ChatGPT and failing to properly verify the material before it was submitted.

The court then had to spend time attempting to locate authorities that had effectively been invented.

This wasn't an isolated warning about technology.

It was a warning about outsourcing knowledge and verification to technology.

A lawyer understands—or should understand—how legal authorities are researched and verified.

AI may accelerate that process.

It cannot remove the professional's responsibility to establish that the information is real.

In healthcare, the consequences can be far greater

The stakes become higher again when AI and automated decision-support systems are involved in healthcare.

Research from Macquarie University's Australian Institute of Health Innovation examined 266 safety events involving approved machine-learning-enabled medical devices reported to the US Food and Drug Administration. Macquarie University – AI-Enabled Medical Devices Raise Safety Concerns

The reported incidents included problems involving radiotherapy planning, insulin dosing, diagnostic ultrasound, mammography and consumer ECG devices.

Examples included patients receiving radiation overdoses or radiation being delivered to an incorrect location following data-input errors, and a patient experiencing hypoglycaemia in an incident involving insulin dosing software.

Importantly, this research doesn't simply tell us that "AI is dangerous".

It tells us something far more useful.

The relationship between technology and the human using it matters.

AI systems operate within a broader system of data, software, processes, assumptions and human decisions.

If the inputs are wrong, the output may be wrong.

If the technology is used outside its intended context, the result may be wrong.

And if the person using the technology doesn't understand its limitations, they may not recognise when intervention is required.

Even ordering a taco isn't immune

The consequences aren't always life-changing. Sometimes they're just ridiculous.

Fast-food companies have experimented extensively with AI-powered voice ordering at drive-throughs. The Verge – Taco Bell rethinks AI drive-through use

The promise is obvious: faster service, lower labour requirements, greater consistency and the ability to process huge numbers of orders.

But real customers aren't controlled test environments.

They have accents.

They change their minds.

They interrupt.

There's background noise.

They make unusual requests.

And they occasionally order things an algorithm isn't expecting.

Some AI drive-through trials have consequently produced bizarre incorrect orders that became viral social-media content.

It might be funny when AI gets someone's fast-food order spectacularly wrong.

It's considerably less funny when the same unquestioning trust in technology is applied to your website, advertising budget, financial decisions, legal work or healthcare.

AI democratises access to expertise. It doesn't automatically create expertise.

This is perhaps the biggest misconception of the AI revolution.

AI gives us extraordinary access to knowledge.

That is not the same thing as possessing knowledge.

I can ask an AI system about structural engineering.

That doesn't make me a structural engineer.

I can ask it to interpret legislation.

That doesn't make me a lawyer.

I can ask it about a medical condition.

That doesn't make me a doctor.

And someone can ask ChatGPT to develop an SEO strategy.

That doesn't suddenly give them years of experience understanding search behaviour, website architecture, technical SEO, content, analytics and what can go wrong when a strategy is implemented.

This distinction is going to become increasingly important.

The beginner may actually be at greater risk

There is an interesting paradox emerging with AI.

The less you know about a subject, the more impressive AI can appear.

An expert sees an AI response and thinks:

"That's useful, but this part isn't quite right."

"It hasn't considered this variable."

"That recommendation would create another problem."

"I need to verify that statistic."

"That citation doesn't look right."

A beginner may simply think:

"Wow. That sounds amazing."

And hit Publish, Submit, Send, Apply or Implement.

That's the danger.

You don't need to be the world's foremost expert to use AI.

But you need enough knowledge to question it.

AI should make experts better—not make us believe expertise is unnecessary

Some of the greatest benefits from AI will come when it is placed in the hands of people who already understand their field.

A marketer can use AI to analyse more information.

A lawyer can accelerate research.

A doctor can use AI-supported technology to identify patterns.

A software engineer can code faster.

An accountant can interrogate financial data.

A business owner can explore ideas that previously required hours of research.

But the human still needs to understand the objective, evaluate the information, identify errors and make the final decision.

The opportunity isn't:

Human OR AI.

It's:

Human knowledge + AI capability.

That's a very different proposition.

So how should businesses use AI?

Use it enthusiastically—but don't use it blindly.

Ask AI for ideas.

Ask it to challenge your thinking.

Ask it to analyse information.

Ask it for alternatives.

Ask it what you may have overlooked.

Then question the answer.

Check important facts against primary sources.

Verify citations.

Check calculations.

Interrogate recommendations.

Consider the consequences if the recommendation is wrong.

And for high-risk decisions involving legal, financial, medical, technical or major business consequences, involve someone who actually understands the field.

Most importantly, learn the basics yourself.

Because the ability to ask AI a question is rapidly becoming commonplace.

The ability to recognise a bad answer isn't.

The skill of the AI era may be knowing when AI is wrong

We're entering an extraordinary period.

AI will undoubtedly eliminate some tasks. It will change jobs. It will increase productivity. It will allow smaller businesses to access capabilities that once belonged almost exclusively to large organisations.

I'm excited about that.

But I don't believe the future belongs to people who simply know how to use ChatGPT.

Soon, almost everyone will know how to do that.

The real advantage will belong to people who combine AI with knowledge, experience, critical thinking and judgement.

Because AI can produce an answer in seconds.

Knowing whether it's the right answer is still up to us.