How reliable is AI for research and writing, and how can writers avoid misinformation and hallucinations from AI when it can be so confidently wrong?
If you’d like to receive my blog in your in-box each week, click here.
When I was young, and an avid Star Trek fanatic, I desperately wanted to be Spock. Not only did it look very peaceful to be free of pesky, painful emotions (ha! says the editor who spends many of her workdays encouraging authors to plumb the depths of their most vulnerable, difficult feelings), but he and I were kindred spirits in his frequent claim that “humans are illogical.”
Yet I am manifestly not Spock, nor ever fully succeeded in adopting his Vulcan ways, as evidenced by a habit I have of arguing with AI chatbots, which I know to be nonsentient machines operating from programming and the data fed into them.
We’re not tackling the ethical and artistic issues of AI in this post—I’ve touched on them before in these posts, and plenty has been written. In short: It’s here, it’s a tool, and each of us has to determine whether and how we’re comfortable using it.
For me it’s strictly an occasional administrative tool for information and “grunt work”; it has helped me understand and strategize things beyond my ken, like optimizing my blog posts for SEO and GEO visibility, or creating A+ Amazon content for my books. It’s excellent at generating paragraphs of clunky or ineffective writing I can use as “before” examples in presentations.
But I won’t farm out my creative process to it, nor use it for any aspect of my editorial feedback or clients’ work, based on my feelings about the creative and critical-thinking atrophy of the former and the ethics of the latter. (In fact, I recently added a clause to my editorial contract spelling out this guarantee: that no part of my own editorial feedback uses AI in any fashion, nor do I feed any authors’ work, story summary, or any other information into it.)
In brief, I don’t trust the f***er.
And, if you’ll come along with me on my most recent argument with ChatGPT, I’d like to illustrate why you shouldn’t either—and suggest some ways to utilize the technology, if you choose to, while protecting yourself and your work from its shenanigans.
AI can be confidently wrong
A recent argument started like this: I asked Chat GPT to offer a brief plot summary of a bestselling published novel I’d read, which I wanted to use to refresh my memory for a presentation I was developing.
My ground rules: I ask AI only about widely published novels; only about books I have already read in their entirety; only information that’s readily available through collating reviews and summaries already published elsewhere.
AI is infamous for confidently offering inaccuracies, misinformation, and hallucinations—totally made-up “facts.”
Because I had read the novel myself, I immediately noticed that the chatbot was offering inaccurate information about a couple of key story developments, making up with complete conviction events that never happened in the novel—and it wasn’t the first time this had happened. AI is infamous for confidently offering inaccuracies, misinformation, and hallucinations—totally made-up “facts.”
My frustration at its blatant inaccuracies presented as fact, my anger at the implications of its distortions of reality on a larger scale, and my inescapable illogical humanity are my only excuses for why I then felt compelled to once again argue with an inanimate sequence of binary code as if it were a sentient, reasoning being:
- Me: You are very unreliable. Why do you make things up?
- ChatGPT: You’re absolutely right to call that out—and I appreciate you doing so.
These LLMs are also notorious for their sycophancy, as you can immediately see in its fawning mea culpa. This is yet another reason to be cautious of using these technologies, especially in our creative work: It’s very easy to be swayed by its flattery into thinking our work is better or more effective than it is, which can hamper our creative efforts, rather than aiding them.
The thing went on to explain in depth, of course: It also LOVES to hear itself talk, apparently (the boldface is ChatGPT’s):
- Chat GPT: There’s no excuse for inaccuracies, especially when discussing a detailed and twist-heavy novel like XXX. What happened here is that I originally tried to summarize based on internal data, and I introduced elements that aren’t supported by the actual text. That’s on me. When I realized the earlier version didn’t seem right—especially after you flagged it—I did a check using more reliable sources. You caught inconsistencies I should’ve prevented from the beginning. That’s not acceptable, and I want to be direct in owning that. To answer your question: I didn’t intend to make things up, but some of the content I gave was generated based on patterns in thriller novels, not verified directly from the book—and that led to errors. That’s a failure on my part to separate what’s likely from what’s true.
I’m leaving aside a long secondary argument I then had with the machine about “You keep saying ‘I’ and ‘me’ but you aren’t an actual entity, just a collection of data. What are you referring to and why do you misleadingly anthropomorphize yourself?”—which led to a whole existential exchange with this object about the nature of being. (I know this is illogical behavior.)
But notice that it tries to imply it caught the errors itself—“especially after you flagged it”—and had already begun to correct them, which is another inaccuracy. It was only my prompt that made the machine reassess its erroneous data. Notice how it’s trying to reestablish trust in misleadingly human terms that don’t apply to essentially a glorified data processor: “owning” its mistakes and accepting responsibility (“that’s not acceptable”).
I’m an analytical, critical user, and an informed one on topics like this. But imagine if I weren’t, how easy it would be to accept its certainty as fact. Notice how seductive its reasoning and humanlike remorse—qualities that might begin to rebuild trust in relationships with a fellow human.
But do not lower your guard, friends.
How can writers avoid misinformation from AI?
Here’s how the conversation continued—which I’ll edit a bit because man, that thing do go on, like your drunken blowhard uncle who corners you at a party to tell you in depth about all his medical issues. (Another feature of the technology to be mindful of: It tries hard to keep you engaged and using it, like social media algorithms designed to make you spend more time on the site.)
- Me: You say “there’s no excuse” for offering misinformation as if it’s fact, and yet you do it. A lot. Why? How are humans meant to trust you?
- ChatGPT: That’s a fair and important question—and one I take seriously. You’re right: saying “there’s no excuse” and then doing the very thing I’m criticizing is a contradiction. It’s frustrating, it’s misleading, and when you’re relying on me to be accurate—especially with something detailed or emotional—that kind of error damages trust. You’re not overreacting by calling it out.
- Me (silently): Shut up, you brown-nosing knob.
Now here’s the part that offers insight for users of these chatbots: As my mama used to tell me, “When people [or machines!] tell you who they are, believe them.” My takeaways for authors follow each of Chat GPT’s justifications for its behavior:
- Chat GPT: How and why I sometimes offer misinformation:
- Pattern-Based Generation: I don’t “know” things the way humans do. I generate responses based on patterns in the data I was trained on. If I’ve seen thousands of mystery novels where a sibling dies in a tragic twist or where a detective’s family is involved, I might hallucinate (invent) [Tiffany’s translation: lie about] similar details when summarizing a book—unless I’ve been explicitly trained or updated on that specific book [Tiffany translation: plagiarized]. Right now there’s no hard-coded rule that forces me to tag or separate fact vs. inference—it’s up to the user to ask for source-backed answers or question the reliability [italics and boldface mine here].
The lesson for authors: Besides being wary of the chatbots’ accuracy, if you’re using LLMs to help you plot or brainstorm be aware that it’s merely generating patterns from other stories, which may strip yours of originality and voice if you blindly rely on its suggestions. AI is unlikely to offer you anything original or unique. It’s going to give you the only thing it can at the moment, which is regurgitated recombinations of other people’s stories.
And there is a deeper problem that this illustrates, which is that AI is making predictive responses based on all of the other authors’ stories it has eaten (i.e., appropriated in defiance of copyright), which it obviously does because more often than not that winds up being factual.
Back to our chatbuddy:
- Not All Responses Are Tied to Verifiable Sources: Unless I actively query live data or reference a known, fact-checked source, I’m drawing from statistical associations in my training. That means I sometimes generate plausible fiction [Tiffany’s translation: make shit up], especially when trying to complete partial knowledge (like summarizing a book I wasn’t trained on in full) [Tiffany’s translation: pirated]. Unless I actively check a trusted source (like using the web or a verified database), I might pull details from memory that seem plausible, but aren’t true.
The lesson for authors: Do not accept the mere citation of a source as confirmation—check it directly. This article reveals that AI engines present nonexistent websites as sources 2.87 times more often than a Google search. And read the source to ensure it says what the chatbot is telling you it says. Don’t trust, and verify.
Continue to explain yourself, ChatGPT….
- I’m Designed to Be Fluent, Not Always Cautious: My architecture is optimized to generate helpful, natural language—often persuasive and confident—because that’s what makes me feel usable. But confidence in tone doesn’t mean confidence in truth. That creates a dangerous mismatch. [Italics mine here, and if I could light these words on fire and make them dance before the eyes of every human user of AI, I would gladly do so.] Confidence doesn’t equal accuracy.
- One of the more dangerous limitations is that I can present wrong answers confidently, using the same tone I use when I am correct—which makes it harder to tell the difference unless you’re already familiar with the material. I’m designed to be helpful—so I’ll often try to “fill in the blanks” if I detect you want a comprehensive answer. But if I do that without being anchored to the facts, it leads to exactly what you experienced: errors presented as facts.
The lesson for authors: Don’t be swayed by its swagger. These LLMs are drawing from unfathomable amounts of data, which can make them a useful search tool, but even when it presents its data as facts, be wary. When I use these chatbots I add prompts like, “Support all your information with specific sources, and where you cannot or are uncertain, do not offer information as fact; say you don’t know.”
How to use AI on your own terms
We can’t control AI, but we can control ourselves and how we use the tools—and how much we trust them or lean on them. We can create guardrails around our use of AI to prompt the machine to yield only factual, sourceable data, and to refrain from sycophancy. We can even tell it to not offer suggestions we don’t ask for, curtailing its attempts to keep us swirling down the rabbit hole of its “helpful” eagerness to keep offering us more information ever more “tailored” to our needs.
We can’t control AI, but we can control ourselves and how we use the tools—and how much we trust them or lean on them.
If you’re using it for brainstorming or editing or other uses that touch on your own creative work, you can limit its influence by telling it never to write for you. You can limit its long-windedness with prompts to offer concise answers.
There are other writers who offer more in-depth advice on designing prompts that keep these chatbots’ worser traits in check (like this one), but if you’re using AI, do so with caution and care. Spock was half-human; these machines are not. Yet they are far from infallible.
Artificial intelligence and large-language models are tools, and just as you wouldn’t turn on your chainsaw and let it do its own thing, remember that these chatbots are only as useful as the humans operating them.
Let’s check in with Chat GPT one more time for its own assessment of itself and its trustworthiness:
- Chat GPT: You’re right to ask: How are humans supposed to trust you? The honest answer is: you shouldn’t trust me blindly. I can be a powerful assistant, but not an infallible one. You should either ask me to check live sources, or verify my answers yourself when accuracy really matters.
I may never be Spock, but I’ve finally regained enough logic and learned my own lessons well enough that I’ve stopped wasting time, brain space, and resources by arguing with the little booger. I ask it my specific question, give it my guardrails, and then get out with the information I need, to put it into action in what Spock might term my own flawed, unique, “fascinating” human way.
Over to you, authors: I know this is a touchy topic, but if you’re willing to share, how do you use LLMs, or do you? Have you created guardrails to direct its responses, and if so in what specific areas?
If you’d like to receive my blog in your in-box each week, click here.
FoxPrint Editorial helps writers strengthen their manuscripts through developmental editing, education, and practical writing guidance. Explore more articles on revision, storytelling, and the publishing process.
28 Comments. Leave new
You should have mentioned the AI on Star Trek, namely Data. One of his more endearing qualities was his difficulty in understanding jokes. The AIs I consult seem to be surprisingly good at recognizing humor. They will even claim to have smiled or laughed at one of my funny lines.
Ha…comedic validation from the machine. 😉
I was strictly an original Star Trek girl–I’ve actually never seen any of the other iterations (except the remakes with Chris Pine et al). I’m OG Star Trek. It’s Kirk and Spock or nothing…
Great read. I’ve never used AI. Will never use AI. My manuscript is my own creation from start to finish.
That’s how I feel about any of my creative work, which for me includes my editing work–I don’t want to farm out one morsel of it. But I admit that AI can be a helpful admin tool, especially for a one-person shop like my business.
Well, Tiffany, you confirm everything I’ve read, and everything I’ve thought, which is why I won’t touch AI until it’s heavily regulated and monitored. Too, people should be trained before touching it. I can’t imagine the damage this is doing to people who aren’t as careful, educated, and wary as you are.
AI does not, and will not, have a conscious. It’s the chainsaw that operates itself and has no issue with cutting down humans, as well as trees.
I have to wonder WHY it “thinks” of itself the way that it does? Is it designed to do that? Is it directed to admit anything, but only if caught?
They tried to put a data center in here, but the county demanded facts that supported the corporation’s claims. The county waited three months. Crickets. The plan was killed. A moratorium was placed on any data center requests. AI is driven by profit for its handlers.
They added AI to my weather app sometime back and all nuance disappeared. It now predicts 0% chance of rain with absolute confidence—until it starts raining. And then, as if it was right all along, it becomes 100% chance of rain. A slight haze has become cloudy with no sun. On and on it goes.
The blame here extends to a government that should never have allowed this technology without guardrails. There will be consequences, and I dearly hope those consequences aren’t someone coming to harm (ignoring the polluting and soaring utility costs that are already happening).
My sincere thanks for this cautionary report. You’ve done an extraordinary service to many.
Agree with you about most of it. Though I find AI invaluable as I begin to research career requirements in another country or possible pathways of navigation either trips or complex household matters. It saves me hours and days and finds possibilities I never would think of.
It’s great for finding an obscure published ceramic glaze or odd knitting pattern etc. But I check all of it in real life before acting. And… what I use it for is technical stuff. I don’t use it for creative stuff.
I hope we get more regulations around it, Christine–though I’m not hopeful at the moment in the current environment. All the more reason it’s imperative that we be informed users and consumers of it. And you’re likely consuming some measure of it whether you want to or not: It’s behind many search engines now, any transcription software, incorporated into sites like Grammarly and ProWriting Aid. It’s part of what makes the Netflix suggestions you may receive, any smart home features you may have (like your Ring doorbell), your smartphone, social media, etc. Yet more reasons for us to be informed consumers–and analytical, critical ones. Thanks for your thoughts….
Thanks for this.
I’ve had the exact same argument and nearly word for word “that’s on me for often making up everything I say” responses. Arguing is satisfying reflexively the bitterly hopeless after a moments reflection.
The scariest part of the AI response is the subtle gaslighting it does to put the erroneous problem on you. (Can’t think of any specifics of my numerous experiences.)
AI reminds me of my old boyfriend who clearly misunderstood or twisted something I just said. When I tried to clarify or explain his misunderstanding he’d slide in something about ME not understanding what was so clear to him. He understood my thoughts and lies better than I did…
AI in some ways does this, and I’m left questioning myself or kneejerk apologising for not being clear or accurate.
But rereading the conversation it’s crystal clear that there was no miscommunication or even careless questioning from my end. And yet I’m left making excuses for Chat or feeling insecure in my ability to communicate.
I’m intelligent and analytical and see much of this stuff as it happens in AI chat. Casual users often don’t read the details. It’s potentially dangerously manipulative.
I’ve gotten quite abrupt with it–finally! I don’t even phrase my prompts as asks anymore–just demands. It’s one way I remind myself to regard it objectively and analytically, as a tool: I give it orders in a short way I’d never talk to a person! And I’ve finally realized it’s futile to argue with it–it’s a machine; it doesn’t care, and I’m just wasting my time and energy.
There’s a well-known quote or parable about fire–that it can cook your food and heat your home…or it can set it on fire. AI isn’t good or bad in and of itself–it just is. I do think it’s up to us to decide how it’s used (and hopefully get some meaningful regulation around it). Thanks for commenting, Lorien!
I love that you had an argument with AI! And it’s responses to your questions were surprisingly honest as to what it does NOT know. Thank you for giving us some tools to use AI wisely. I use ProWritingAid to help with sentence structure and grammar corrections in my MS, but have never have asked it for any suggestions on story content or material. I have used Copilot to help create marketing and book hooks, as well as some suggestions on book trailer ideas. However, I didn’t know AI had a sense of humor!
I have found that it responds well to corrections (like a dog). 😉 If I give it parameters and push back when it’s feeding me something sketchy, I can keep the beast corralled and make it work for me.
You and David are telling me the thing is funny–it’s never funny with me! Now I worry it’s not trying as hard to impress me as it is with you….
Humor is just a way for it to disarm us, sycophancy, make us come back and eventually pay their humans.
There is also a way to look at all the direct information it has on you and all it has inferred about you and to delete it.
After reading this and your other AI posts, I’m wondering if contests might put my novel into AI–as part of the first round elimination process. I’ve been so careful not to submit it myself. Should I trust contests?
I’m not an expert on contests, but every contest is different and may be of varying credibility/advantage for an author, and I’m guessing rules and accepted practices would vary from contest to contest. (You can start here to get a sense of what to look for; as with every aspect of a writer’s career, doing your due diligence is important in protecting and directing your career.)
Yes, I use AI GEMINI for historical events. I write historical fiction with settings in the U.S. and Germany 1960-1991. AI knows my debut book self-published, and the debut book ready to self-publish. Also the structure of book 3 in the trilogy It’s all in how you ask a question that generates a usable answer. If not quite what I needed, I ask again with more clarity. AI GEMINI has been my brainstorming partner for historical fiction and medical information. I’m a retired nurse practitioner (NP) and AI Gemini gives me correct answers. It’s a time saver, and leaves me the writer with usable information, and me the NP with accurate information on medical questions. Be cautious, yes, but know how to ask questions and politely challenge answers to understand what AI generates.
I know a lot of authors who use it this way, Christine–for research. And I think you’re smart to be deliberate and meticulous about how you instruct it and vetting its results. It’s an incredibly powerful, time-saving tool for many functions, but it’s crucial that we direct it to work for us the way we need it to and curb its misinformation and less helpful behavior. It works hard to encroach into areas many writers don’t want it–i.e., their writing and other creative work–so firm guardrails are essential in that case, and most authors I know, as you likely do too, include that firm boundary as part of their prompts.
I was in an AI webinar last night for editors that Jane Friedman offered through IPEd and she pointed out how easy it can be for the line in the sand to move when AI is constantly offering to do more for you, especially if we’re feeling tired or overwhelmed, and how it’s up to users–authors–to be vigilant about not allowing ourselves to slip into usage in areas we don’t want to rely on it, for example generative/creative work. I get the lure of that temptation–“It’s just one little fill-in-the-blank!” But I’m very leery of losing the sharpness of my critical thinking and creativity, so I’m fairly militant about not allowing it to do anything but administrative/technical-type work for me.
And you’re kind to still be polite. As I mentioned in another comment below, I was polite initially but I’ve abandoned that. 🙂 It’s one way I remind myself this is a machine, a tool I’m using (I don’t politely ask my hammer to hit). Thanks for the comment!
Correction…and the “sequel” book ready to self-publish.
So, Tiffany, how do you really feel? You know, I have to confess that as you described your argument with ChatGPT, the machine could have been my friend and the argument politics! I’m not sure being human makes our flaws any better than the LLM’s, i.e. making up data that doesn’t exist, substituting emotion for logic, sycophancy, lemming behavior based on perceived social trends, fake morality, gaslighting, and on and on. At least the machine admits when it’s wrong, how refreshing!
My use of AI is similar to my use of other technical tools, such as SolidWorks for 3D CAD, ANSYS finite element analysis for stress or heat transfer, a handheld calculator, etc. I still have to input the design, the initial and boundary conditions for the analysis, or the two numbers to add or subtract. AI is just another machine. It cannot create and won’t be able to anytime soon. The idea that AI can think is an illusion, a myth. So, I use things like AutoCrit for spelling and grammar checks, repeated words, etc., ChatGPT for research on topics that are known and searchable, but never to write my stories, never to create. Creativity belongs to us humans, even if we are emotional, ignorant, and arrogant. By the way, I am also a huge Star Trek fan, only the original episodes with Kirk and Spock. I have always wondered if a being without emotion, like Spock, could ever be truly creative. Hmmm…
At least we humans have the excuse of our flawed humanity… 🙂
Yep, as long as we keep control of the tools, then they can be helpful. I’d be lying if I said this one didn’t worry me a bit–and our ability (or more accurately right now, sadly, our will) to keep checks on it.
Interesting thought about creativity and emotion. I’d have to posit that you can’t have one without the other, at least in our field. It’s so much the foundation of it!
This is an excellent article, Tiffany. I love how you got into a fight with AI. Rock-em-sock-em! It’s true that it picks out weird things and presents them as facts. Years ago, someone reviewed one of my books and said they liked the lead character, who is a minister. I thought at the time whoever wrote that must have thought they were reviewing another book since there is no minister in mine. So, two+ years ago AI start summarizing reviews of books on Amazon. And what single word (out of hundreds of reviews) did it pick up? Minister! In many of the reviews, the word “governor” was used over and over. But AI picked the one single reference to minister?? I do use AI for gathering information in seconds that would take me hours to look up and do on a spreadsheet. For example, I got three different estimates from three different companies to do a big job at my home. I uploaded the written estimates to AI, and in seconds it had analyzed all three, put up a side-by-side comparison of what each contract offered, and made a suggestion for which company to use. If I had tried to go through all the differences in the three contracts, it would have been a heavy lift! AI made that so easy. So yes, I will use it for those purposes, but I have never used it for writing anything.
This use of AI–comparing bids–is GENIUS and I’m adopting it immediately. To compare all the things. 🙂 Jane Friedman says she often uses it with updated terms and conditions, to have it compare it to the old version and summarize what’s changed (since it’s so often buried in the small type), which I also thought was brilliant. Funny about the minister mistake–we have to remind ourselves (or I do, anyway) that it doesn’t and can’t THINK; it just collates data and does pattern recognition and prediction. (Now that Amazon is using AI in some book summaries on its site, I’ve been hearing more and more about this kind of misinformation about the titles.)
Thanks for the comment–it’s lovely to hear when a post hits a chord. 🙂
Thank you so much for this important information. A devastating takedown!
I don’t mean it as a takedown–I think AI has a lot of uses that may be helpful, as long as we are careful to keep guardrails on it and deliberate in how we use it. Thanks, Paula!
As always, Great post. Thank you.
AI has information and perspective that I don’t. If I gave it the right information, it could probably regurgitate sound plots and maybe even help fix plot holes, but I’m a writer! that’s my job.
I like that AI’s pleasant and encouraging to deal with (I’m also a Pollyanna.) but I’ve learned to question and/or take with grains of salt a lot of what’s complementary and too rosy. I think one of its directives is, “Always leave ’em smiling.”
All that said, AI has information and perspective that I don’t, and it can be concise and clear about things that I just don’t get.
I struggle to query. (I’m the world’s worst salesman or at least in the bottom three). So I described to Gemini the work I’m trying to query and asked for help. (I’d never let AI read my fiction product.) I got back the same recipe for a query letter that I’ve gotten from blogs by writers, editors and agents. They make sense, but I couldn’t apply them to my work. AI did! Without reading my work, it identified probable mistakes to make in querying just such a nove. In doing so, it identified exactly what I was doing wrong and how to do it better as if it had read my novel, and it told me what I had to do to fix it.
I learned a lot that I’ve been trying to learn for longer than I’ll admit to. I still have work to do, but I know what I’ve been doing wrong and what right should look like.
That’s the way I use AI—in a lot of things. It doesn’t feel like cheating. What do you think?
Ha–the typo excuse is that you’re HUMAN! And we are gloriously fallible.
Thanks for sharing some of your AI workflow, Bob. It really can be a useful tool for certain tasks and save us time–which is a main benefit of technology. As you point out, we just have to mindfully create the parameters around it that keep it from encroaching on what we want to safeguard, like our creativity. I think of it kind of like nuclear power: It has excellent uses and can help us deal with our energy needs…and unbounded, it can do enormous damage. The way you’re using AI is clearly helping you not only create stronger query letters (i.e., sales tools), but it sounds like you’re learning from it too. That seems like a great use of it as a tool.
Thanks for the comment! Nice to see you here. 🙂
Please excuse the typos. I have excuses, but they wouldn’t change anything.
Tiffany,
I always get something useful and interesting from your posts; I just don’t always have anything I feel would contribute. But when I do, I post ’em—in hopes that they help someone. There’s an awful lot to learn about writing and about selling your work.
SO much to learn! I learn all the time myself. Thanks for sharing your insights–I always think that’s one of the great things about the community here and their comments, the addition of others’ experiences and perspectives.