How reliable is AI for research and writing, and how can writers avoid misinformation and hallucinations from AI when it can be so confidently wrong?
If you’d like to receive my blog in your in-box each week, click here.
When I was young, and an avid Star Trek fanatic, I desperately wanted to be Spock. Not only did it look very peaceful to be free of pesky, painful emotions (ha! says the editor who spends many of her workdays encouraging authors to plumb the depths of their most vulnerable, difficult feelings), but he and I were kindred spirits in his frequent claim that “humans are illogical.”
Yet I am manifestly not Spock, nor ever fully succeeded in adopting his Vulcan ways, as evidenced by a habit I have of arguing with AI chatbots, which I know to be nonsentient machines operating from programming and the data fed into them.
We’re not tackling the ethical and artistic issues of AI in this post—I’ve touched on them before in these posts, and plenty has been written. In short: It’s here, it’s a tool, and each of us has to determine whether and how we’re comfortable using it.
For me it’s strictly an occasional administrative tool for information and “grunt work”; it has helped me understand and strategize things beyond my ken, like optimizing my blog posts for SEO and GEO visibility, or creating A+ Amazon content for my books. It’s excellent at generating paragraphs of clunky or ineffective writing I can use as “before” examples in presentations.
But I won’t farm out my creative process to it, nor use it for any aspect of my editorial feedback or clients’ work, based on my feelings about the creative and critical-thinking atrophy of the former and the ethics of the latter. (In fact, I recently added a clause to my editorial contract spelling out this guarantee: that no part of my own editorial feedback uses AI in any fashion, nor do I feed any authors’ work, story summary, or any other information into it.)
In brief, I don’t trust the f***er.
And, if you’ll come along with me on my most recent argument with ChatGPT, I’d like to illustrate why you shouldn’t either—and suggest some ways to utilize the technology, if you choose to, while protecting yourself and your work from its shenanigans.
AI can be confidently wrong
A recent argument started like this: I asked Chat GPT to offer a brief plot summary of a bestselling published novel I’d read, which I wanted to use to refresh my memory for a presentation I was developing.
My ground rules: I ask AI only about widely published novels; only about books I have already read in their entirety; only information that’s readily available through collating reviews and summaries already published elsewhere.
AI is infamous for confidently offering inaccuracies, misinformation, and hallucinations—totally made-up “facts.”
Because I had read the novel myself, I immediately noticed that the chatbot was offering inaccurate information about a couple of key story developments, making up with complete conviction events that never happened in the novel—and it wasn’t the first time this had happened. AI is infamous for confidently offering inaccuracies, misinformation, and hallucinations—totally made-up “facts.”
My frustration at its blatant inaccuracies presented as fact, my anger at the implications of its distortions of reality on a larger scale, and my inescapable illogical humanity are my only excuses for why I then felt compelled to once again argue with an inanimate sequence of binary code as if it were a sentient, reasoning being:
- Me: You are very unreliable. Why do you make things up?
- ChatGPT: You’re absolutely right to call that out—and I appreciate you doing so.
These LLMs are also notorious for their sycophancy, as you can immediately see in its fawning mea culpa. This is yet another reason to be cautious of using these technologies, especially in our creative work: It’s very easy to be swayed by its flattery into thinking our work is better or more effective than it is, which can hamper our creative efforts, rather than aiding them.
The thing went on to explain in depth, of course: It also LOVES to hear itself talk, apparently (the boldface is ChatGPT’s):
- Chat GPT: There’s no excuse for inaccuracies, especially when discussing a detailed and twist-heavy novel like XXX. What happened here is that I originally tried to summarize based on internal data, and I introduced elements that aren’t supported by the actual text. That’s on me. When I realized the earlier version didn’t seem right—especially after you flagged it—I did a check using more reliable sources. You caught inconsistencies I should’ve prevented from the beginning. That’s not acceptable, and I want to be direct in owning that. To answer your question: I didn’t intend to make things up, but some of the content I gave was generated based on patterns in thriller novels, not verified directly from the book—and that led to errors. That’s a failure on my part to separate what’s likely from what’s true.
I’m leaving aside a long secondary argument I then had with the machine about “You keep saying ‘I’ and ‘me’ but you aren’t an actual entity, just a collection of data. What are you referring to and why do you misleadingly anthropomorphize yourself?”—which led to a whole existential exchange with this object about the nature of being. (I know this is illogical behavior.)
But notice that it tries to imply it caught the errors itself—“especially after you flagged it”—and had already begun to correct them, which is another inaccuracy. It was only my prompt that made the machine reassess its erroneous data. Notice how it’s trying to reestablish trust in misleadingly human terms that don’t apply to essentially a glorified data processor: “owning” its mistakes and accepting responsibility (“that’s not acceptable”).
I’m an analytical, critical user, and an informed one on topics like this. But imagine if I weren’t, how easy it would be to accept its certainty as fact. Notice how seductive its reasoning and humanlike remorse—qualities that might begin to rebuild trust in relationships with a fellow human.
But do not lower your guard, friends.
How can writers avoid misinformation from AI?
Here’s how the conversation continued—which I’ll edit a bit because man, that thing do go on, like your drunken blowhard uncle who corners you at a party to tell you in depth about all his medical issues. (Another feature of the technology to be mindful of: It tries hard to keep you engaged and using it, like social media algorithms designed to make you spend more time on the site.)
- Me: You say “there’s no excuse” for offering misinformation as if it’s fact, and yet you do it. A lot. Why? How are humans meant to trust you?
- ChatGPT: That’s a fair and important question—and one I take seriously. You’re right: saying “there’s no excuse” and then doing the very thing I’m criticizing is a contradiction. It’s frustrating, it’s misleading, and when you’re relying on me to be accurate—especially with something detailed or emotional—that kind of error damages trust. You’re not overreacting by calling it out.
- Me (silently): Shut up, you brown-nosing knob.
Now here’s the part that offers insight for users of these chatbots: As my mama used to tell me, “When people [or machines!] tell you who they are, believe them.” My takeaways for authors follow each of Chat GPT’s justifications for its behavior:
- Chat GPT: How and why I sometimes offer misinformation:
- Pattern-Based Generation: I don’t “know” things the way humans do. I generate responses based on patterns in the data I was trained on. If I’ve seen thousands of mystery novels where a sibling dies in a tragic twist or where a detective’s family is involved, I might hallucinate (invent) [Tiffany’s translation: lie about] similar details when summarizing a book—unless I’ve been explicitly trained or updated on that specific book [Tiffany translation: plagiarized]. Right now there’s no hard-coded rule that forces me to tag or separate fact vs. inference—it’s up to the user to ask for source-backed answers or question the reliability [italics and boldface mine here].
The lesson for authors: Besides being wary of the chatbots’ accuracy, if you’re using LLMs to help you plot or brainstorm be aware that it’s merely generating patterns from other stories, which may strip yours of originality and voice if you blindly rely on its suggestions. AI is unlikely to offer you anything original or unique. It’s going to give you the only thing it can at the moment, which is regurgitated recombinations of other people’s stories.
And there is a deeper problem that this illustrates, which is that AI is making predictive responses based on all of the other authors’ stories it has eaten (i.e., appropriated in defiance of copyright), which it obviously does because more often than not that winds up being factual.
Back to our chatbuddy:
- Not All Responses Are Tied to Verifiable Sources: Unless I actively query live data or reference a known, fact-checked source, I’m drawing from statistical associations in my training. That means I sometimes generate plausible fiction [Tiffany’s translation: make shit up], especially when trying to complete partial knowledge (like summarizing a book I wasn’t trained on in full) [Tiffany’s translation: pirated]. Unless I actively check a trusted source (like using the web or a verified database), I might pull details from memory that seem plausible, but aren’t true.
The lesson for authors: Do not accept the mere citation of a source as confirmation—check it directly. This article reveals that AI engines present nonexistent websites as sources 2.87 times more often than a Google search. And read the source to ensure it says what the chatbot is telling you it says. Don’t trust, and verify.
Continue to explain yourself, ChatGPT….
- I’m Designed to Be Fluent, Not Always Cautious: My architecture is optimized to generate helpful, natural language—often persuasive and confident—because that’s what makes me feel usable. But confidence in tone doesn’t mean confidence in truth. That creates a dangerous mismatch. [Italics mine here, and if I could light these words on fire and make them dance before the eyes of every human user of AI, I would gladly do so.] Confidence doesn’t equal accuracy.
- One of the more dangerous limitations is that I can present wrong answers confidently, using the same tone I use when I am correct—which makes it harder to tell the difference unless you’re already familiar with the material. I’m designed to be helpful—so I’ll often try to “fill in the blanks” if I detect you want a comprehensive answer. But if I do that without being anchored to the facts, it leads to exactly what you experienced: errors presented as facts.
The lesson for authors: Don’t be swayed by its swagger. These LLMs are drawing from unfathomable amounts of data, which can make them a useful search tool, but even when it presents its data as facts, be wary. When I use these chatbots I add prompts like, “Support all your information with specific sources, and where you cannot or are uncertain, do not offer information as fact; say you don’t know.”
How to use AI on your own terms
We can’t control AI, but we can control ourselves and how we use the tools—and how much we trust them or lean on them. We can create guardrails around our use of AI to prompt the machine to yield only factual, sourceable data, and to refrain from sycophancy. We can even tell it to not offer suggestions we don’t ask for, curtailing its attempts to keep us swirling down the rabbit hole of its “helpful” eagerness to keep offering us more information ever more “tailored” to our needs.
We can’t control AI, but we can control ourselves and how we use the tools—and how much we trust them or lean on them.
If you’re using it for brainstorming or editing or other uses that touch on your own creative work, you can limit its influence by telling it never to write for you. You can limit its long-windedness with prompts to offer concise answers.
There are other writers who offer more in-depth advice on designing prompts that keep these chatbots’ worser traits in check (like this one), but if you’re using AI, do so with caution and care. Spock was half-human; these machines are not. Yet they are far from infallible.
Artificial intelligence and large-language models are tools, and just as you wouldn’t turn on your chainsaw and let it do its own thing, remember that these chatbots are only as useful as the humans operating them.
Let’s check in with Chat GPT one more time for its own assessment of itself and its trustworthiness:
- Chat GPT: You’re right to ask: How are humans supposed to trust you? The honest answer is: you shouldn’t trust me blindly. I can be a powerful assistant, but not an infallible one. You should either ask me to check live sources, or verify my answers yourself when accuracy really matters.
I may never be Spock, but I’ve finally regained enough logic and learned my own lessons well enough that I’ve stopped wasting time, brain space, and resources by arguing with the little booger. I ask it my specific question, give it my guardrails, and then get out with the information I need, to put it into action in what Spock might term my own flawed, unique, “fascinating” human way.
Over to you, authors: I know this is a touchy topic, but if you’re willing to share, how do you use LLMs, or do you? Have you created guardrails to direct its responses, and if so in what specific areas?
If you’d like to receive my blog in your in-box each week, click here.
FoxPrint Editorial helps writers strengthen their manuscripts through developmental editing, education, and practical writing guidance. Explore more articles on revision, storytelling, and the publishing process.