top of page
Search

7 Stats, Zero Trustworthy Sources: My ChatGPT Experiment, Gone Awry

What happens when you ask ChatGPT for statistics? Can you trust the data at face value? According to my experiment ...

Recently, to strengthen a blog post I was writing, I searched for stats related to case studies.


While I normally would turn to Google, I decided to try out ChatGPT.

In response to my prompt ("Share stats and cite sources about the power of case studies"), I received seven stats and sources. A few of them appeared to be VERY strong, and from reputable sources. I got excited!

And then ... I clicked on the URLs to each citation.


The results were rather disturbing — and they serve as a cautionary tale about using Artificial Intelligence for research and information gathering.

My ChatGPT experiment results: the TL;DR version

The longer version of my ChatGPT experiment is below this table. As for the short story?

  • ChatGPT provided seven stats + seven links

  • Not one of the links went to the recommended stat

  • I couldn't find any of the stats anywhere on the internet

ChatGPT "source"

Where did the link go?

Is the ChatGPT stat available elsewhere on the internet?

Harvard Business Review

Error page

No

Demand Gen Report (twice)

Wrong page, both times

No

Content Marketing Institute

Error page

No

Search Engine Land

Error page

No

HubSpot

Error page

No

Journal of Education for Business

Error page

No

Stat #1: Harvard Business Review
The stat, verbatim from ChatGPT

"According to a study by the Harvard Business Review, 79% of B2B buyers said that they have consumed at least one case study in the past 12 months. (Source: Harvard Business Review)"


The link provided by ChatGPT

https://hbr.org/2018/03/the-rise-of-the-case-study


The result

404 Error


What I did next

I copied and pasted the entire stat, verbatim, into Google. No results. I then plugged in a variety of search terms related to 79%, B2B, case study / case studies, 12 months / one year, and Harvard Business Review.


What I discovered

I could not find the stat ANYWHERE. I gave up after 10 minutes.


Can I use the ChatGPT stat?

Absolutely not.


Stat #2: Demand Gen Report
The stat, verbatim from ChatGPT

"Case studies are the second most popular type of content among B2B marketers, after blog posts. (Source: Demand Gen Report)"


The link provided by ChatGPT

https://www.demandgenreport.com/resources/research/2018-content-preferences-survey-b2b-buyers-value-content-that-offers-data-and-analysis/


The result

The URL took me to this page, which has nothing to do with the stat:


What I did next

As I did with Harvard Business Review, I copied and pasted the entire stat, verbatim, into Google. The top result took me to this page, which cited the stat. However, that stat was from 2013, not 2018 as the name of ChatGPT's original URL suggested.


I then added "2018" to my Google search, and this page came up as the first result. To check if the ChatGPT case study stat was in THAT report, I downloaded it.


What I discovered

Case studies are certainly mentioned in the DGR's Content Preferences report. However, the wording of the ChatGPT stat is somewhat misleading. This report doesn't ever explicitly say that case studies are the second-most popular type of content for B2B. And even when you compare some of the case study stats in the report to blog stats, there is at least one instance where case studies score HIGHER than blogs.


Can I use the ChatGPT stat?

I don't think so. The Content Preferences 2018 report does have some interesting nuggets in it about B2B content marketing preferences. But there's nothing in there to confidently support ChatGPT's original stat.


Not to mention, through a quick Google search, I found a 2022 Content Preferences report, which includes more up-to-date stats.

Stat #3: Content Marketing Institute

The stat, verbatim from ChatGPT

"A survey of 63 marketing and sales professionals found that 72% believe that case studies are effective in generating leads and sales. (Source: Content Marketing Institute)"


The link provided by ChatGPT

https://contentmarketinginstitute.com/2018/01/b2b-case-study-research/


The result

Error page:


What I did next

I first copied and pasted the ChatGPT stat, verbatim, into Google. The top result took me to this CMI report.


What I discovered

Hmmm ... there are a few mentions of case studies in the report. But there is literally no reference to 73 percent, as per ChatGPT's recommended stat. Rather, 61% of B2B marketers stated they had used case studies in the previous 12 months (but there's no mention of this "belief" in case studies generating leads and sales).


Back to Google! I searched around a bit more for the source of the stat, using different search terms. Finally, I found this page, which quoted a 73% stat from the Content Marketing Institute. But it was related to applying content marketing to an organization's overall marketing strategy. No mention of case studies.


Can I use the ChatGPT stat?

Honestly, I don't think so. The CMI report has some interesting stats about case studies, but none that align exactly (or even somewhat closely) with what ChatGPT provided.

Stat #4: Search Engine Land

The stat, verbatim from ChatGPT

"Case studies can increase website traffic by up to 300%. (Source: Search Engine Land)"


The link provided by ChatGPT

https://searchengineland.com/the-importance-of-case-studies-in-a-digital-world-165614


The result

This error page:

What I did next

I looked everywhere on Google! I used the exact phrase. I used the title provided in the URL. Keywords. And many other search strategies. I also went to the Search Engine Land website to see if I could find it.


What I discovered

Nothing. I found absolutely nothing. No articles online referencing this 300% increase in web traffic due to case studies.


Can I use the ChatGPT stat?

Certainly not. Next!


Stat #5: Demand Gen Report (again)

The stat, verbatim from ChatGPT

"In a study of 900 B2B buyers, 82% said that they found case studies to be influential in their decision-making process. (Source: Demand Gen Report)"


The link provided by ChatGPT

https://www.demandgenreport.com/resources/research/2018-content-preferences-survey-b2b-buyers-value-content-that-offers-data-and-analysis/


The result

Just like Stat #2 above, ChatGPT sent me to the wrong page.


What I did next

Another spin on Google! Using the stat verbatim, Google took me back to the 2022 Content Preferences Report. Although I looked for the stat within the entirety of the report, there was nothing about case studies pertaining to 82%.


What I discovered

A pattern, lol! Once again, no amount of digging via Google could produce that stat for me.


Can I use the ChatGPT stat?

Hell no.


Stat #6: HubSpot

The stat, verbatim from ChatGPT

"Case studies have been shown to improve retention of information by up to 70% compared to other forms of content. (Source: HubSpot)"


The link provided by ChatGPT

https://blog.hubspot.com/marketing/case-studies-better-than-other-content-formats


The result

Surprise (but not really)! Another error page. Albeit a cute one.

What I did next

Yet another Google search. Copying and pasting the stat. Plugging in keywords. The usual.

What I discovered

Despite all the other misses above, I still held out hope I'd find this stat somewhere across the internet. But sadly, no luck.


Can I use the ChatGPT stat?

What do you think?


I began wondering if I'd ever be able to trust ANY ChatGPT stats.


Stat #7: Journal of Education for Business
The stat, verbatim from ChatGPT

"A meta-analysis of 40 studies found that case-based instruction is more effective than traditional lecture-based instruction in promoting critical thinking skills. (Source: Journal of Education for Business)"


The link provided by ChatGPT

https://www.tandfonline.com/doi/abs/10.1080/08832320701583238


The result

Error page


What I did next

Nothing. Did you notice the stat isn't even about case studies? Thanks, ChatGPT!


What I discovered

ChatGPT takes liberties with language when interpreting human prompts.


Is ChatGPT full of BS when it comes to "facts"?

Maybe I'm naive, but I sincerely believed that the AI tool would provide some accurate data! Instead, ChatGPT provided not only seven incorrect URLs, but seven seemingly fabricated "facts."


I went down a research rabbit hole to see if the chatbot might have misinterpreted some of the numbers it provided. After all, stats can often be difficult to qualify, even for humans. But these weren't misinterpretations. We're talking about made-up numbers.


As a freelance content writer and editor, I have to do a lot of research for my B2B, government, and non-profit clients. But what if I had used ChatGPT for a client project and taken the AI's responses