the appearance of understanding

we live in a weird time. ai slop gets millions of likes, and at the same time work done by hand by artists gets called ai slop. in my place of work, we are still trying to work out where the boundaries lie in terms of what is acceptable for communication between ourselves. when/where does ai-generated messages provide value to our team? does it ever?

i don’t intend for this post to insinuate that i hold the answer to this question, but instead hopefully prompt some of you to be asking these questions yourself with a bit broader perspective. this is actually going to be a series of posts, helping me to work through my own thoughts, questions, and concerns introduced by this new and modern age.

the question

stated more formally, the first question i want to ponder is:

what changes when generative ai can produce work that signals understanding without requiring the person presenting it to possess comparable understanding?

the filter

i think generally if i’m talking with someone, i’m subconsciously determining at every moment how much i trust their presentation of knowledge on a topic. this judgment is very imperfect, and there are plenty of cases where i or others just take what’s said at face value depending on the confidence exuded by the presenter.

before generative ai, the effort required to produce a detailed work acted as an imperfect filter. i think previously, i had the ability to rely on a produced piece of work and the dialogue generated from it as an indicator of their comprehension of the problem space. this does not imply correctness, accuracy, or any such claim about the work. effort has never guaranteed truth.

of course, ghostwriters, templates, plagiarism, canned reports, all of these predate generative ai. these still took time, money, and/or effort to produce in such a way that they were relevant and fit into the discussions at hand. that’s why the filter is imperfect. but it’s a filter nonetheless.

now, with generative ai, both the work and the dialogue can be imitated with minimal effort, if any. ai manufactures a bespoke argument which implies the possession of knowledge in a way that was previously unattainable. it can manufacture a confident, coherent, sophisticated, and seemingly well-considered argument of which the presenter only has a half-baked understandingor none!. this is not copying and pasting. it’s customised, able to respond to specific critiques, adopt the terminology of the field, anticipate objections, and present a coherent chain of reasoning tailored to the specific context in which the human behind the screenif there is one! finds themselves.

with ai, i can produce a coherent argument, or an incoherent argument. i can maybe explain it, depending on how much labor i’ve expended to understand it myself, although i may also be able to get ai to explain it depending on circumstances.

without knowing whether they used ai to generate it, and the level of effort they expended to determine its validity (and even the background they have to inform that verification), i cannot easily say whether they understand any of it. i suppose ai makes it more challenging for us to rely on those subconscious indicators, even though it was possible to game them before.

learning through creating

you can read about how to write code all day long, but until you go through the motions and actively exercise that recall, you will struggle to have it retained in your memory generally. when you do something once, it’s easier to do it again, even in a different context. the more you do it, the bigger the class of problems you can tackle. that’s how you grow.

limited as our minds are in our capacity for understanding any given topic, the process of interacting substantially with that topic often requires misunderstanding parts of it. coders leave comments explaining why they implemented a bug fix a certain way, to clear up the misunderstandings they encountered when initially implementing the functionality,or even to make jokes. writers revise their articles many times before publishing, only to make corrections or follow-up articles to address the misconceptions strewn throughout their work, or to share new things they’ve learned that have updated their perspective.

the process of creation exposes confusion, or at least it should. taking the time to put your jumble of thoughts onto paper helps you become aware of the shortcomings. maybe not always or not to the same extent, depending on your ability to exhibit socratic thinking. i think that grappling with the questions presented or answered in the production is what helps to deepen the creator’s understanding.

many years ago, much of how i learned to code was through helping others learn to code in a discord server. much as i was advised, i advised others to avoid simply copy+pasting code from stack overflow without putting in the work to thoroughly understand what that code accomplished. failing to heed that advice led to many messages from beginners who couldn’t comprehend why their code wouldn’t compile, or why it didn’t behave the way they expected. when you take code that you don’t understand and utilise it in a separate context, you find that you have no idea what assumptions it made, what tradeoffs it balanced, or why it failed. big surprise

ai forces upon us a more sophisticated version of the same problem. instead of an obviously mismatched snippet, it generates a plausible, bespoke solution to the described code problem. the code might even work. yet the lack of comprehension remains.

sure, you can work with the ai to truly form an understanding of it, but it’s too easy in the human condition to rely on a tool once it’s established. if you know that the ai can help you answer your questions about it, you might be tempted to do so. in such a case, not as much mental effort is being expended, and the lessons are less likely to stick in your brain. in the age of ai, a competent-enough text can be generated without expending such intellectual labor. it allows us to ‘complete the task’ without any improvement of our perspective.

going through intellectual effort through the act of learning and creating provides you a foundation on which you can be trusted to perform similar works/analysis in the future. if we are outsourcing all thought to an ai, how do we progress our collective knowledge further? what happens when the ai is unavailable? what happens when knowledge grows beyond our capacity for comprehension?

in plato’s phaedrus, socrates recounts a tale of the invention of writing. writing is posited as a tool to improve wisdom and memory, yet socrates fears that it may instead allow people to acquire the appearance of wisdom without its reality. he too saw the clear distinction between the possession of knowledge, and the parroting of that which implies knowledge.

At the Egyptian city of Naucratis, there was a famous old god, whose name was Theuth; the bird which is called the Ibis is sacred to him, and he was the inventor of many arts, such as arithmetic and calculation and geometry and astronomy and draughts and dice, but his great discovery was the use of letters. Now in those days the god Thamus was the king of the whole country of Egypt; and he dwelt in that great city of Upper Egypt which the Hellenes call Egyptian Thebes, and the god himself is called by them Ammon. To him came Theuth and showed his inventions, desiring that the other Egyptians might be allowed to have the benefit of them; he enumerated them, and Thamus enquired about their several uses, and praised some of them and censured others, as he approved or disapproved of them. It would take a long time to repeat all that Thamus said to Theuth in praise or blame of the various arts. But when they came to letters, This, said Theuth, will make the Egyptians wiser and give them better memories; it is a specific both for the memory and for the wit. Thamus replied: O most ingenious Theuth, the parent or inventor of an art is not always the best judge of the utility or inutility of his own inventions to the users of them. And in this instance, you who are the father of letters, from a paternal love of your own children have been led to attribute to them a quality which they cannot have; for this discovery of yours will create forgetfulness in the learners' souls, because they will not use their memories; they will trust to the external written characters and not remember of themselves. The specific which you have discovered is an aid not to memory, but to reminiscence, and you give your disciples not truth, but only the semblance of truth; they will be hearers of many things and will have learned nothing; they will appear to be omniscient and will generally know nothing; they will be tiresome company, having the show of wisdom without the reality.

this worry that a new technology will inhibit the memory or learning of people is not a new fear. writing ended up revolutionizing humanity, so ai perhaps can do the same for us now. it could usher into another age where memory becomes less important but our capacity for development and creation grows.

but that’s a big bet to make, and just because the technology seems revolutionary doesn’t imply it will make memory less important. i can’t really predict the future.

so what does this mean for us?

intellectual responsibility is a curious thing. we’ve long had humans put into situations where they’re responsible for presenting something they don’t fully comprehend. product managers rely on engineers to understand the inner machinations of a feature while the product manager knows the user story. CEOs rely on their CTO to ensure the application is on stable footing and doing what it needs to accomplish. sometimes those engineers (or CTOs) don’t have the right answers either. it’d be a beautiful world one without bugs.

the challenge is, we’ve long used someone’s output, their work, to assess their competence. this is something that becomes more difficult in the day of ai. suddenly, the filter once relied upon has had holes poked throughout it. someone who presents what appears to be a well-thought-out explanation of a difficult topic can no longer be inherently trusted to have wrestled with the questions that underpin such an explanation. someone who delivers an excellent and performant technical solution to a problem cannot inherently be trusted to produce similar solutions in the future.

consider the code example from earlier. you have to understand the code and its implications in the context of the broader system. whereas previously, you might have consulted another engineer with similar understanding of the broader system and how the work fits into it, now potentially the presenter of the pull request doesn’t fully understand these implications themselves and is not able to help onboard you to that understanding.

how can you verify it without understanding, unless you are offloading the verification to someone with such understanding? reviewers have always had to verify contributions, but when the presenter cannot explain the work, more of the intellectual effort required to understand it may fall on the reviewer.

the cost to generate code goes down, but the cost to review may even go up. on the other hand, ai tools perhaps enable the reviewer to understand this context quicker on their own, so maybe it balances out? even if reviewing individual contributions becomes easier, what happens to the total burden on reviewers when substantially more pull requests can be produced?

this can be painful. your coworkers may be smart and intelligent people. you might have trusted the work they did, giving a second look in the weaker parts of what they present. you assumed they knew the topic well enough to answer the questions you have. you can no longer always trust that to be the same. this feels hurtful. like a breach of trust. it also makes it harder for people with a deep understanding of a topic to demonstrate and educate this, and can breed misunderstanding.

i wish i did have the magic book with all of the answers here. i don’t. i just have thoughts that swirl around and occasionally make it into my head. maybe some of you do. get in touch with me if so. i’d love to tell you why they’re wrong.