There is a tendency on this subreddit and other places similar to it online to post breathless, gushing commentary on the capabilities of the large language model, ChatGPT. I see people asking the chatbot questions and treating the results as a revelation. We see venture capitalists preaching its revolutionary potential to juice stock prices or get other investors to chip in too. Or even highly impressionable lonely men projecting the illusion of intimacy onto ChatGPT.

It needs to stop. You need to stop. Just stop.

ChatGPT is impressive in its ability to mimic human writing. But that's all its doing -- mimicry. When a human uses language, there is an intentionality at play, an idea that is being communicated: some thought behind the words being chosen deployed and transmitted to the reader, who goes through their own interpretative process and places that information within the context of their own understanding of the world and the issue being discussed.

ChatGPT cannot do the first part. It does not have intentionality. It is not capable of original research. It is not a knowledge creation tool. It does not meaningfully curate the source material when it produces its summaries or facsimiles.

If I asked ChatGPT to write a review of Star Wars Episode IV, A New Hope, it will not critically assess the qualities of that film. It will not understand the wizardry of its practical effects in context of the 1970s film landscape. It will not appreciate how the script, while being a trope-filled pastiche of 1930s pulp cinema serials, is so finely tuned to deliver its story with so few extraneous asides, and how it is able to evoke a sense of a wider lived-in universe through a combination of set and prop design plus the naturalistic performances of its characters.

Instead it will gather up the thousands of reviews that actually did mention all those things and mush them together, outputting a reasonable approximation of a film review.

Crucially, if all of the source material is bunk, the output will be bunk. Consider the "I asked ChatGPT what future AI might be capable of" post I linked: If the preponderance of the source material ChatGPT is considering is written by wide-eyed enthusiasts with little grasp of the technical process or current state of AI research but an invertebrate fondness for Isaac Asimov stories, then the result will reflect that.

What I think is happening, here, when people treat ChatGPT like a knowledge creation tool, is that people are projecting their own hopes, dreams, and enthusiasms onto the results of their query. Much like the owner of the parrot, we are amused at the result, imparting meaning onto it that wasn't part of the creation of the result. The lonely deluded rationalist didn't fall in love with an AI; he projected his own yearning for companionship onto a series of text in the same way an anime fan might project their yearning for companionship onto a dating sim or cartoon character.

It's the interpretation process of language run amok, given nothing solid to grasp onto, that treats mimicry as something more than it is.

EDIT:

Seeing as this post has blown up a bit (thanks for all the ornamental doodads!) I thought I'd address some common themes in the replies:

1: Ah yes but have you considered that humans are just robots themselves? Checkmate, atheists!

A: Very clever, well done, but I reject the premise. There are certainly deterministic systems at work in human physiology and psychology, but there is not at present sufficient evidence to prove the hard determinism hypothesis - and until that time, I will continue to hold that consciousness is an emergent quality from complexity, and not at all one that ChatGPT or its rivals show any sign of displaying.

I'd also proffer the opinion that the belief that humans are but meat machines is very convenient for a certain type of would-be Silicon Valley ubermensch and i ask you to interrogate why you hold that belief.

1.2: But ChatGPT is capable of building its own interior understanding of the world!

Memory is not interiority. That it can remember past inputs/outputs is a technical accomplishment, but not synonymous with "knowledge." It lacks a wider context and understanding of those past inputs/outputs.

2: You don't understand the tech!

I understand it well enough for the purposes of the discussion over whether or not the machine is a knowledge producing mechanism.

Again. What it can do is impressive. But what it can do is more limited than its most fervent evangelists say it can do.

3: Its not about what it can do, its about what it will be able to do in the future!

I am not so proud that when the facts change, I won't change my opinions. Until then, I will remain on guard against hyperbole and grift.

4: Fuck you, I'm going to report you to Reddit Cares as a suicide risk! Trolololol!

Thanks for keeping it classy, Reddit, I hope your mother is proud of you.

(As an aside, has Reddit Cares ever actually helped anyone? I've only seen it used as a way of suggesting someone you disagree with - on the internet no less - should Roblox themselves, which can't be at all the intended use case)