Back again with a winding conversation with two of my favorite dudes and long-time friends of the blog, Jeremy Kaish and Grayson Murphy, to chat about ChatGPT, AI text generation, why AI fatalism is overblown, orcas, and how much cars suck (ancillary, but true!).
Jeremy is a software engineer and avowed tech skeptic. Grayson is a digital preservation librarian and info org wiz. This conversation was recorded the day after we experienced “Taylor Swift: The Eras Tour” at MetLife Stadium, so words were slow. This has been edited for clarity and length.
Shira Buchsbaum (SB): Who’s ChatGPT chatting with? That’s what I initially titled this.
Jeremy Kaish (JK): All of us, in a sense.
SB: In a way.
JK: In a way. Or really, the ghosts of all of us.
SB: What was it I said the other night about…I should have recorded whatever we said Friday night.
JK: When we were talking to your Dad about it.
SB: Yeah. I said something like, “ChatGPT cannot have a thought that’s never been thought before,” which, fundamentally, is a labor issue from every angle because the people whose existing work is mined for those thoughts (i.e. data) are not compensated, and then the people who tag and categorize that data to make meaning in ChatGPT are not properly compensated either.
JK: It’s really three ways, because then people are trying to leverage the results in ways to displace actual workers.
SB: Right, it’s this circular effect. You’re taking writers’ work, not compensating them for it; you’re hiring people to tag it, not compensating them; and then you’re using the results to displace the writers who you already were not compensating for having stolen their work to train your algorithm. Or like, that’s what the business bros on Twitter are crowing will be the cost-efficient future of writing.
JK: If it were to become prolific enough, future language models would be trained on stuff written by this one, so you’d have like this—
SB: —this profound dilution of—
JK: —this downward spiral of garbage. [Editor’s note: this is already happening.]
SB: Garbage all the way down. Though ClickHole convincingly argues that no one can rehash IP as well as Disney. Anyways, I had this long riff on Friday about signposting and meaning-making that I’m trying to conjure again: ChatGPT cannot make anything meaningful. It is a pattern-matching program that is taught signs of meaning. Signal words to indicate understanding. But it cannot process or create new meaning. It’s just recycling what it’s been told. Which is why, at first glance, when you plug something in, it seems logical, but when you dig deeper, it ends up being nonsense.
Grayson Murphy (GM): I think that’s the bigger problem of calling it artificial intelligence. Because the layperson assumes that means that AI is itself intelligent.
SB: Independently intelligent.
GM: And not like the computer science way of thinking about intelligence.
JK: Even the way we talk about it—we anthropomorphize it, as we do many things. We refer to ChatGPT as “it”—it said, it thought, it did. But it’s not actually doing anything. It’s not autonomous. It has no agency.
What’s also concerning to me about the current ~discourse~ around ChatGPT is how eager some are to use it to replace human writers, and, specifically, how inevitable that transition is being portrayed by many in the media. I have to reiterate: AI bots cannot think. Their outputs are the result of a gigantic pile of words and some very finely tuned probabilities. All a bot can do is output an arrangement of words that probably makes sense.
SB: I haven’t read that piece from The New Yorker that poses “There is no A.I.,” but that was the gist of it, right Grayson?
GM: Yep.
JK: Every few weeks, there’s a guy who’s like, “It’s sentient!” But he told it, “Pretend to be sentient.”
GM: It’s been trained on decades of humans being scared of computers taking over, so it knows how to mirror that in language.
SB: Grayson, a while back you wrote, “An important question I have is how will we vet information in the age of AI search engines? My initial worry is AI search could potentially take away a user’s agency in terms of information literacy. What resources will the discerning user have to check sources?” Did you want to elaborate on that?
GM: I feel like we three each have our biggest concerns with AI, and my biggest concern is information literacy, which is such a staple of librarianship, and I’m wondering why everyone is worried, asking me, “Oh, as a librarian, are you worried that your job will become obsolete?” [Editor’s note: No.] But, the whole purpose of librarianship is helping people to navigate information literacy and establish skills for finding information that is useful and accurate. So yeah, the proliferation of AI in information search is the biggest thing concerning me. Everyone is doing their own thing and it’s going to be even harder to vet information than it was before. Engineers and developers are integrating AI into Snapchat, I think. Your average fourteen year old is not going to be vetting the information he gets on SnapChat.
JK: There’s also an interesting phenomenon in that the model takes a while to train. It doesn’t know current events because it can only know what is in the training data, which was harvested umpteenth units of time in the past. So if you were to ask ChatGPT about the war in Ukraine, it wouldn't know about it, because it was trained on data from before that happened.
SB: So not only is there an issue of information dilution, but considerable information lag, especially if you are looking at AI search engines. Let’s set aside ChatGPT and the factually inaccurate interactive chatbot. I’m less interested in that. I’m more interested in the potential compositional-analytic elements of AI search; unlike Google Search where you plug in your key words and the existing websites with the top traffic become your top search results (barring ads, hello capitalism) but an interface that is becoming increasingly popular on Google. Now, if you ask Google a question, it will provide you an initial answer. It won’t send you immediately to Wikipedia for “Oppenheimer'' or the “Pythagorean Theorem.”
JK: Usually those are excerpts from Wikipedia, though.
SB: Yes, and sometimes Google will extract a section and highlight part of a sentence, but not the whole sentence, and then the extract emphasizes the wrong meaning. So I see AI search consolidating that information in those little boxes the way that Google has started to. If AI is trained on information that is diluted and delayed, that means those answers will become increasingly incorrect. This concerns me for the same reasons you were saying, Grayson, regarding an inability on the user’s part to discern what the sources of that information were. We already see that ChatGPT can’t cite its sources. Its answer is cobbled together from keywords that are plugged into the dataset. You can start going down that rabbit hole of verification and you find that this is signposting, again: ChatGPT can mimic verified sources, but that does not inherently verify them.
JK: ChatGPT’s function is to provide the highest probability answer to your query according to previous points of the conversation, but factual accuracy is not built into its model.
GM: Granted, I think it has its advantages in the banal of life. Writing an email that doesn’t need a personal touch. Or when I use it to write simple Python scripts. I’ve done a lot of relocating file hierarchies in digital preservation and thought to myself, “This could take me half an hour, or I could ask ChatGPT to write a Python script to do it.” It spits it out, tells me every element. I test it on some test data. It works. I have a workable formula in five minutes. But when I ask it for information—
SB: —qualitative information—
GM: —That’s where I don’t trust it.
JK: I will say, I haven’t really seen OpenAI, the people who made ChatGPT, trying to push the use case of “Let’s replace all writers.” That’s coming from opportunistic MBAs jumping on it.
SB: You’re totally right. These text-based AI engines are really good for formulas and pattern recognition.
JK: Because they are giant pattern matching machines. And there are cases where I have to do a rote task and I’m like, “I could write a script for this!” And it takes me two hours to do it. *laughs* So ChatGPT’s got me there.
SB: Having a little engine to create those formulas for you is great. Frankly, writer/podcaster/online funny person Kelsey McKinney put it better: the people who are cheering about ChatGPT as the future of creativity are not creative. They see a tool that will allow them to substitute their own lack of creativity for profit.
JK: She also said that they are resentful towards creative people.
SB: They are! For having a skill and aptitude for something they cannot do. And, yeah, I’m going to paint in broad strokes here, but the worst of terminally online AI acolytes want to profit off of that skill without dealing with complications of properly compensating another human being for their creative labor. Because they don’t see creativity as labor. Or they do, but only when they control the prompts. Or they fundamentally misunderstand what makes self expression interesting. Or what they see as a profound revelation ripe for capital opportunities is actually the phenomenon of engaging in minor creative thought for the first time ever. It’s not that deep! That’s what we see playing out in real time with the writer’s strike and the resistance on the writer’s part to use AI or be replaced with AI, from one-offs to wholesale.
JK: I’ve seen a number of interviews where the interviewer will ask the writer, “Wouldn’t you–”
SB: Oh, like the Ari Shapiro and David Simon interview on NPR. *Proceeds to read from the interview, with respective voices for Ari and David*
SHAPIRO: OK, so you've spent your career creating television without AI, and I could imagine today you thinking, boy, I wish I had had that tool to solve those thorny problems, or saying...
SIMON: You mentioned that.
SHAPIRO: ...Boy, if that had existed, it would have screwed me over.
SIMON: I don't think AI can remotely challenge what writers do at a fundamentally creative level.
SHAPIRO: But if you're trying to transition from Scene 5 to Scene 6 and you're stuck with that transition, you could imagine plugging that portion of the script into an AI and say, give me 10 ideas for how to transition.
SIMON: I'd rather put a gun in my mouth.
SHAPIRO: You would rather put a gun in your mouth.
SIMON: I mean, what you're saying to me effectively is there's no original way to do anything, and...
SHAPIRO: No.
SIMON: Yes, you are.
SHAPIRO: That seems like a kind of absolutist take.
SIMON: Not only, I think, is it a fundamental violation of the integrity of writers and also of copyright - you know, when I sold all the scripts I sold - you know, 150 to HBO and, you know, maybe another 50 to NBC - I didn't sell them so that they could be thrown into a computer with other people's and be used again by a corporation.
SB: …And then they continue talking about the contract implications of using AI as a “writer.”
JK: It’s mildly off-topic, but it’s interesting to see that fundamental disconnect of people being like, “Aren’t you excited about this? It’ll help you do your job more easily!” And the people who actually do that work are like, “No.”
SB: I think that’s true whenever there are new tools.
GM: We haven’t really addressed the alarmist side, which I think is unnecessarily inflammatory: “It’s gonna take over everything!” I don’t even know if that’s worth addressing.
JK: I was talking to a colleague of mine in marketing about this, and I asked if he was worried about this, because he writes a lot of copy for the company, and he was like, “Mm. No. Because a lot of copy is more or less the same between different use cases. It would take me longer to come up with a prompt, refine that, edit its answer, because you can’t take anything ChatGPT says at face value, than it would if I just wrote it myself.” No such thing as unskilled labor!
SB: ChatGPT is not independently intelligent, as we’ve reiterated. It is trained on existing data. So to try to grant it agency represents to me a fundamental misunderstanding of how the tool is built. And, it demonstrates a disregard for all the kinds of intelligences that do exist in the world. Why are we scared of ChatGPT when the orcas are orcanizing? You can unplug a robot. You can’t unplug an anti-yacht orca army.
GM: A big existential-ish question in academia is how to address AI text-generation and plagiarism and amending plagiarism rules. A big conversation is whether AI text generators are just another thing to attribute in your work, like, “Portions of this paper were generated by ChatGPT,” or if it should be off the table. I don’t think it’s possible to keep everyone from using it.
Then there’s the matter of functional relevance: we held a panel about AI in academia, and this one faculty member came who teaches rudimentary math and noted that students should know how to do certain formulas without asking a computer how to do it. And my cynical ass was like, “Well we used to know how to do long division, and now we just use the computer to do it.” We have people who are concerned about whether what they bring to the table will continue to be valued. But the reality is that computers are going to take over certain levels of stuff we’ve done manually, continually, forever.
JK: That returns to what you were saying earlier, Grayson, about writing boilerplate professional emails or cover letters.
SB: This brings up for me the utility of formulaic versus creative writing. One of the basic criticisms to emerge from this coinciding firestorm of AI text generation and the writers’ strike is, “Why are we using AI to replace screenwriters? Why can’t we use AI to clean up the oceans?” Well, there are a lot of steps between ChatGPT and the great ocean cleanup project.
GM: It’s not a one-size fits all to every world ill.
SB: I think people see AI as a wholesale interruption in how life works, and they don't see it as a tool constituent to many parts.
JK: I can't tell you how many early stage startups emailed me when I was looking for a job that said, “We are using AI to do [insert literally anything here].” And every time I thought to myself, “This is horse shit.”
Grayson, you were saying that your university is convening a panel to figure out what to do about this. That reminds me of what your dad was saying the other night, Shira, about how we are still in the figuring it out stage, which is true. But we are figuring it out and doing it at the same time. I love that companies are going into Congress being like, “You guys need to do something about this!” There’s a sense of an AI arms race. We build it all up, but to what end? What are we going to do with it?
GM: This goes back to my point about how ChatGPT is to AI text generation as Tesla is to self-driving. Tesla made it flashy, so everyone scrambled to catch up, and people are out on the road using this under-tested product, and people are dying.
JK: The only practical use case of self driving is long-haul trucking. That’s it. Trying to put autonomous vehicles in cities or populated areas is a waste of time.
SB: That’s really good to know. We’ll put that in.
JK: That’s my opinion.
SB: I’m sure it’s backed by sound research. I saw Hyundai is creating cars with rotating wheels so you can drive sideways into tough parking spots and I thought “What a waste of time.” Create a whole axis of rotation for a vehicle to accomplish one task. This is not how we should be spending our energy.
GM: Well, they have to justify producing the same car year after year. They have to add bells and whistles.
JK: I read that the car makers are finally getting rid of those demonstrably awful giant touchpad screens in cars that are mega dangerous.
SB: Oh thank goodness. This is an anti-car podcast.
GM: We haven’t even gotten into the LED headlights.
*collective groans*
GM: They advocated for higher lumens because it increases the safety rating.
SB: No it doesn’t!
GM: For the people measuring safety rating, it increases it. Not literally.
SB: And you’re blinding people on the road and putting other drivers at risk.
JK: Especially when you’re driving a sedan, a sensible car, and someone behind you is in their suped up F150 and their headlights beam directly into your rearview mirror?
*collective groans*
GM: Okay to bring it back. My boss is always complaining about bad cover letters and the altogether lack of composition skills in today’s applicant pools.
SB: I remember in my first year seminar in undergrad, on our first paper, I was the only person who got an A. My professor was this very brusque older gentleman and he made us read aloud my paper in class, paragraph by paragraph, to demonstrate why.
GM: Did he tell the class it was yours?
SB: No, thank goodness. It was still mortifying. Afterwards, he went to the board and told us he was going to teach us how to write a paper. He began, “It begins with the word.” Everyone groaned. But he kept plugging along: “Every word is deliberate. And then the sentence. Every sentence is deliberate.” Then the paragraphs, then the sections. He went through the whole paper like that, meticulously plotting out our choices as writers. I took that to heart. It was the best writing advice I’ve ever received! But man, there is so much bad writing out there.
JK: And all of it is ending up in ChatGPT.
SB: It is! Let’s bring this home, gentlemen. We’ve identified the main issues: there’s no way to cite sources; there’s currently no way to track where these data pools are coming from; the people whose writing comprise these data sets are not compensated; that work is being manipulated in an exercise of analytical grift; the people who are tasked with processing this data are not sufficiently compensated; the results are aspirationally being ill-leveraged to replace creative people by people who have no composition skills; and it’s being overblown as the end-all be-all replacement to human creativity and thought. But truly, it is a diluted pile of bunk.
JK: Yep.
SB: It begins with the word. And ChatGPT is not it!
housekeeping and birdseeking
house
What Shira read recently, among other selections: The Absolute Book by Elizabeth Knox (tightly wound thriller inside a sprawling fantasy epic); Goldenrod by Maddie Smith (poems on parenting, love, and the body); Made for Love by Alissa Nutting (individual resistance against megalomaniac tech billionaires trying to infiltrate our minds…relevant, no?).
What Shira is currently reading: The Light Pirate by Lily Brooks-Dalton. 10/10 recommend.
What Jeremy read recently: All of the Hunger Games books because the films were on Netflix a few months ago
What Jeremy is currently
readingplaying: The Legend of Zelda: Tears of the Kingdom (no time to read when you’re sinking 100 hours into a game in one month)What Grayson read recently: A lot of digital preservation documentation. 0/10 don’t recommend.
What Grayson is currently
readingplaying: The Legend of Zelda: Tears of the Kingdom. (Recently while spending too much time on NJ transit, Jeremy and I taught Shira why the Nintendo Switch is called a Switch. Groundbreaking!).
bird
More later.