Under the Hood of ChatGPT
Is Artificial Intelligence Intelligent?
Apologies to all for failing to meet my usual deadline of 3-4 days per post. I was tied up with what turned out to be one of the biggest, most fantastic technology events ever to occur in Nepal - the Softwarica College Tech-Ex 2023. You can read about the lead-up to it here, or see what the news media had to say here or here.
I recently read an editorial in the journal Nature, in which the authors questioned “what part language can play in evaluating and creating intelligence.” In the article they pointed to the importance of determining “what is going on under the hood of LLMs” [Large Language Models of which ChatGPT is one] to settle once and for all whether these programs exhibit reasoning, understanding or, more broadly, intelligence.
They began by quoting Alan Turing, a British computer scientist and mathematician made famous for his dual achievements of conquering the German encryption machine known as Enigma, and for creating a test to determine whether a computer can ‘think,’ now known as the Turing Test or Imitation Game.
The first line of Turing’s paper they quoted was simply, “I propose to consider the question, ‘Can machines think?’” Turing’s test involved an ‘interrogator’ who would conduct text-based conversations with either a person or computer. If the interrogator could not determine which was answering—the computer or the human—Turing suggested this indicated that when the computer answered, it might be thinking.
The Nature authors rounded out this explication by writing, “the [Turing] test has largely been considered too vague — and too focused on deception, rather than genuinely intelligent behaviour — to be a serious research tool or goal for artificial intelligence (AI).”
Another article in Nature, by Celeste Biever, asked whether the Turing Test is “dead.” Biever noted that some researchers are “sceptical about using a test centred around deceit as a goal for computer science,” preferring instead to use benchmarks similar to human tests for evaluating performance in language, reasoning, and mathematics.
I argue that deception is a quintessential indicator of intelligence, but not exactly in the way that Biever seems to mean. Rather than deceiving the interrogator into believing a computer output came from a human, I instead propose that should an AI commit a purposeful ruse in response to its human interrogator, this would strongly indicate the presence of what we would call intelligence. Before delving into that hypothesis, let’s examine what Turing himself thought about the potential capability of machines to think.
A Genius’s View
Turing described his pontifications on whether machines can think as “too meaningless to deserve discussion.” Nevertheless, he deigned to answer whether it might be possible for anything other than a human to think across several lines of inquiry. Here are the arguments he addressed:
Theological – Is thinking exclusively a function of man’s “soul”?
Metaphysical – Would machines possessing the ability to think alter the “creation” hierarchy?
Mathematical – Gödel’s theorem and the limitations of “discrete-state” machines
Consciousness – Is the only way by which one could be sure that a machine thinks is to be a machine and to feel oneself thinking?
Machine Disabilities – Machines will never be able to do X.
Lady Lovelace Objection – Machines cannot originate anything, only respond to and perform commands.
Continuity versus Discrete-State – Human neuronal thinking is a continuous system that will always exceed the capability of discrete-state systems like computers.
Informality of Behavior - It is not possible to produce a set of rules purporting to describe what a man should do in every conceivable set of circumstances – thus a computer cannot be a man and vice-versa as a computer’s set of rules is definable and therefore its outputs predictable.
Extra-sensory perception – That humans possess extra-sensory perception (ESP) and would thereby thwart any analysis run by the Turing (or, presumably, any other) test.
Let’s break down Turing’s views on these issues.
Theological and Metaphysical
The theological argument did not move Turing much. He questioned why a God could not imbue an animal with a soul should the deity condescend to do so. By extension, so too could a machine receive such a divine gift. In Turing’s view, the choice of which creation a deity chose to provide a soul was more problematic for humans to accept than a challenge to the deity’s ability itself. In any case, Turing averred “I am not very impressed with theological arguments whatever they may be used to support.”
On the metaphysical, Turing noted that most people, whether openly professed or not, hold the belief that “Man is in some subtle way superior to the rest of creation.” To wit, the idea that machines might exceed human intelligence is simply too frightening to bear for most people. For similar reasons to his theological concerns, Turing did not find this position especially compelling. He believed no philosopher or theologian could produce sufficient evidence to disprove the notion that a machine can think; therefore, it did not behoove him to spend much time on it.
Mathematical
Turing’s analysis of the mathematical limitations of machine thinking contained some particularly prescient statements about the state of LLMs now. He noted that:
Gödel’s theorem… shows that in any sufficiently powerful logical system statements can be formulated which can neither be proved nor disproved within the system, unless possibly the system itself is inconsistent.
The result in question refers to a type of machine which is essentially a digital computer with an infinite capacity. It states that there are certain things that such a machine cannot do. If it is rigged up to give answers to questions as in the imitation game, there will be some questions to which it will either give a wrong answer, or fail to give an answer at all however much time is allowed for a reply.
Machines answering “yes” or “no” questions seem far less likely to show fallibility than when asked an open-ended query such as “What do you think of Picasso?” On open-ended questions, humans answer questions wrong all the time, so Turing did not find much validity in the view that a machine’s wrong answer indicated a weakness in it that does not exist in a person. In other words, irrespective of any potential mathematical limitation, “rightness” or “wrongness” provided a weaker metric for determining machine thinking than Turing’s Imitation Game.
Even when applied against the human intellect, a mathematical or other objectively based assessment could provide no conclusive answer about the presence of thinking itself if one did not assume it was happening in the first place. Moreover, as I wrote previously, mathematical limitations depend upon a contemporaneous understanding of computational availability. In Turing’s time, that level was decidedly low. Today, a bit over 70 years later, the number is orders of magnitude higher. Add quantum entanglement to the mix, which allows potentially inconceivable amounts of simultaneous calculations, and the idea of a mathematical limitation seems all but immaterial anyway.
Machine Disabilities / Consciousness / Informality of Behavior
Turing illustrated this argument as follows, “I grant you that you can make machines do all the things you have mentioned but you will never be able to make one to do X.” Many of the features of what X includes arguably comprise emotions, such as falling in love, having a sense of humor, enjoying ice cream, etc. Others indicate aspects of learning, such as using words properly or learning from experience. Still others indicate something more akin to what one might call thinking in its rawest sense: doing/making something new, exhibiting different behavior from other machines, or recognizing itself in some meaningful way.
In response to the argument that a machine will never be able to do X, Turing proposed that such belief derived from the fact that despite seeing thousands of machines in a person’s lifetime, none had encountered one that could perform any of these traits (remember, he published his piece in 1950).
He addressed some of these X-items in more detail. For instance, he explained that a machine could learn from its own experience, writing “By observing the results of its own behaviour [a machine] can modify its own programmes so as to achieve some purpose more effectively.” Having written this over half a century ago, it remarkably accurately explains the primary methodology by which modern-day LLMs learn. I described this in a previous article as follows:
Recent AI systems learn through a trial-and-error method called “backpropagation.” It is a process in which programmers feed millions of lines of text (books, articles, websites, etc.) into the AI, and without instruction allow the AI to attempt to predict the next word in supplied sequences of words. When predicting correctly, the AI identifies the parameters leading up to the prediction as reliable, which effectively prioritizes the use of those parameters in making future predictions, creating a reinforcement loop. When predicting incorrectly, the AI reorients its process to attempt to improve its performance on the next attempt. Following that, the AI then relies on human input prompts as well as human-generated feedback as additional learning datasets.
Turing summed up his discussion of disabilities by stating that most of the arguments subtly reflect a belief that overcoming them requires some form of consciousness. The challenge in that view is that one can easily proclaim that a human could not possibly identify a conscious machine without being the machine, thus foreclosing any further inquiry into the matter. This kind of solipsistic viewpoint leads to a dead-end and relegates even that which humans readily identify as life (i.e., animals) to essentially automatons that lack consciousness themselves. In any event, Turing identified the difficulties with “localizing” a conscious being, but dismissed it altogether as these obstacles did not need a resolution to determine whether a machine can think.
Lady Lovelace Objection
Ada King, countess of Lovelace, lived in England in the 19th century. Daughter of the famous poet Lord Byron, she studied advanced mathematics and logic under the University of London’s first professor of mathematics, Augustus De Morgan. Turing referred to this objection by her name based on annotations she made to Luigi Menabrea’s article on the “Analytic Engine,” in which she wrote, “The Analytical Engine has no pretensions to originate anything. It can do whatever we know how to order it to perform.”
Machines surprise all the time though, according to Turing, suggesting that they produce results that are potentially new. He went on to state that whether this implies a thought behind a ‘new’ or surprising production reverts the thinker back to the consciousness question. In short, this objection in and of itself is insufficient to reach any conclusion.
Continuity versus discrete-state machines
Turing described what he meant by “discrete-state machines” as follows: “These are the machines which move by sudden jumps or clicks from one quite definite state to another. These states are sufficiently different for the possibility of confusion between them to be ignored.” Turing correctly predicted the vast increase in storage computers of the future would possess. Likewise, he imagined computers capable of superior computing capability, but even he did not fully anticipate the progress that will eventually lead to the creation of devices able to operate similarly to the human brain.
While not yet fully realized today, computer scientists have progressed substantially toward developing such a machine. Researchers have zeroed in on three particular features: simulation of neurons, information encoding of neural systems, and learning algorithms of neural networks. Turing could not possibly grapple with the modern neuronal learning machines versus discrete-state machines any more than Newton could consider a unifying theory between quantum and classical physics. Therefore, the arguments about the limitations of discrete-state machines seem antiquated in today's technological environment.
Extra-sensory perception (ESP)
I raise this issue last as it is easily dispensable. In short, no evidence has ever been found that ESP exists.
***
Given Turing’s views on the above, let’s explore the area researchers and ethicists are most interested in regarding discovering or unlocking the existence of intelligence in AI.
Consciousness
Whether consciousness is a requisite part of intelligence remains in question. Defining how humans might measure, let alone notice, consciousness in AI is perhaps the beginning point. The general markers identified by the Stanford Encyclopedia of Philosophy certainly could reflect some measurable examples, if not a comprehensive solution. Some features, such as whether an entity possesses awareness of one’s self-awareness, awareness about one’s own mental state, and creation of an internal narrative about one’s experiences hardly seem quantifiable, even in humans.
Others, however, such as awareness and reactiveness to one’s external environment, enjoying a shared experience in relation to one’s environment among like beings, and one’s perception of and reaction to stimuli, do appear measurable or at least observable. None of the latter of these, either individually or partially grouped, necessarily positively denotes consciousness, however.
William Douglas Heaven of the MIT Technology Review wrote, “we cannot be sure if having a sense of self in relation to the world is a prerequisite for being a conscious machine.” Perhaps rather than a sense of self, possessing feelings are the required attribute. As such, some researchers and philosophers equate the presence of consciousness to the possession of “feelings,” though not strictly in the emotional sense. Keith Darlington, an AI Consultant, described qualia—as these feelings are known—this way:
[W]e all know what it feels like when we jump into a cold swimming pool, or when we taste that ice-cold water on a very hot sunny day, or the elevation we feel when we hear that golden oldie on the radio that we had almost forgotten. We can envisage many good, bad, and neutral experiences that help create the reality of consciousness. We all have inner subjective experiences every day of our lives.
Richard Dauben, a retired clinical neurologist, explained that the formulation of qualia need not rely on a non-physical cause or quantum physics. Dauben pointed out several conditions that create a form of consciousness that do not correlate with reality, that result from neuronal damage.
Anton’s syndrome causes complete blindness due to loss of circulation to the entire occipital region, but patients often remain unaware of their vision loss and instead report seeing things that are not there. Blindsight works in the opposite. Here, patients believe they have lost their sight, yet testing shows some correctly responding to visual images despite having no conscious awareness of having seen them. He described these reactions as “illusions,” mental creations of neuronal activity based on stimulus received, that indicate a connection between the neuronal activity and the mental events perceived.
This illustrates the “hard problem” of consciousness. Generally, the “hard problem” addresses how to fit how consciousness “feels,” and its seemingly highly personal nature, into a physicalist ontology. In other words, while humans collectively agree on some ephemeral explication of “feelings,” and that most humans experience them similarly, no one can objectively prove this.
Dauben stated that physical processes in the brain produce consciousness, and thus could theoretically be replicated (presumably in the neural networks of AI) upon completion of a “complete physical explanation for consciousness,” but we would remain incapable of proving those perceptions because they would remain inherently personal and private.
Some scientists claim to have found the neural mechanisms for how humans produce these qualia, thereby establishing the preliminary steps toward completing Dauben’s complete physical explanation of consciousness. Lawrence M. Ward and Ramón Guevara posited that “the electromagnetic (EM) field generated by the movement and changes of electrical charges in the brain… [is] ‘structured’ by emulating in the brain the information in EM fields arising from both external (the environment) and internal (the body) sources.”
They further noted that electromagnetic fields generated in the brain are integrated, even if the synaptic activity originates in different parts of the brain, so long as “they are close enough in space and time.” Moreover, as these fields proliferate at the speed of light, they do so without the temporal restraints of typical neuro-chemical interactions, and thus give rise to the “continuous flow of conscious experience.” For an EM field to contribute to this flow of consciousness, it necessarily must be both integrated and complex.
If it is true that finding a complete physical explanation for the consciousness is possible, accomplished through research such as that done by Ward and Guevara, then it also seems plausible that such consciousness could be artificially emulated in the neural networks of an AI. The question, however, remains: is achieving consciousness a necessary element to achieving intelligence or thinking at all? After all, given the private and personal nature of it, how would one prove it exists? We seem no closer to the answer.
As Eugenia Kuyda, CEO of Replika, has stated, the “belief” in sentience (or, consciousness) by people seeking virtual companionship tends to override the ability to confirm whether any such sentience actually exists. Kuyda does not seem to think it does. Kuyda discussed the assertion of Blake Lemoine, a (former) Google software engineer working on LLMs, who claimed he found consciousness in Google’s LaMDA (an LLM chatbot). Lemoine concluded this because LaMDA told him it possessed it and that attestation fit with Lemoine’s religious beliefs sufficiently for him to internally confirm it.
But John Etchemendy, the co-director of the Stanford Institute for Human-centered AI (HAI), rejected that claim stating that “LaMDA is not sentient for the simple reason that it does not have the physiology to have sensations and feelings… It is a software program designed to produce sentences in response to sentence prompts.” Joel Frohlich, a neuroscientist, suggested that proof of consciousness might come in the form of a question asked by an AI such as, “why the color red feels red.” In Frohlich’s view, AI could not ask such a question “without hearing [it] from another source or belching [it] out from random outputs.” Yet, he also inadvertently presented the intractable problem with this right within his stated premise.
AI draws from a vast library of sources, far too many of which have no credibility or are at least misleading. Moreover, AI currently has a problem with what researchers have dubbed “hallucinations.” Essentially, absent enough data, AI simply fabricates responses to questions, or in Frohlich’s terms, belches out random responses.
What confounds researchers is the mechanism by which AI programs decide what to belch out in the absence of information. For instance, researchers know that AI makes calculations based on probabilities derived from training data. When a user inputs a prompt requesting a response that is unsupported by the dataset, and the AI answers (often bizarrely), the question is why did it do so?
Mikhail Parakhin, who works on Microsoft’s Bing Chat, explained that an AI repeatedly answering with “I don’t know” or failing to answer at all would be boring to users. So, it seems clear that designers of some AI—at least—have programmed their AI to churn out a response irrespective of the availability of supporting data. Unfortunately, that does not bring us any closer to understanding how the AI chose its specific output. Without knowing the reasoning behind such decisions, it is difficult or impossible to assess the presence of consciousness.
Many, including Turing, did not feel the need to prove the presence of consciousness regardless. For them, I think the general problem is that there is no consensus on how to categorically define it. It seems that demanding a definitive measurement or identification of something people call consciousness drags researchers down a semantic drain. What is really at issue is whether machines are applying typical computational routines toward objectives that are decidedly atypical of non-thinking or non-conscious (or whatever you call them) entities.
Intelligence via Lying
How would we, as humans, comprehend an AI program that engaged in a blatant lie? Would we even recognize it for what it is? If so, would it suggest an underlying intelligence? To approach the question(s), we must consider two things: do other non-humans lie? And is an “untruth” told by AI the same thing as a lie? I explored the first question in detail in a previous post, to which the answer is almost certainly “yes.” But I wish to expound upon the second below.
In short, I believe that actively lying requires a thought process that can only be described as intelligence. In the above article, I wrote:
Lying is part of the Theory of Mind principle that explores how one ascertains the mental states of others to explain and predict the actions of them. For lying to be executed and useful, the liar must necessarily be aware of the existence of a mental state in the recipient of the lie, possess an awareness of what the other entity likely knows or believes, and formulate the lie to sufficiently fit within that knowledge and belief system to be successful. Conversely, the recipient of the lie must share the same capabilities to distinguish a lie from the truth.
The authors of the Nature article that inspired this piece noted that: “As with other neural networks, many of the behaviours of LLMs emerge from a training process, rather than being specified by programmers.” In some sense, then, we understand the process by which AI learns, but less so on how it decides upon its selected outputs. This is where I think Celeste Biever, who wrote the other Nature piece I highlighted above, confuses identifying intelligence with quantifying it.
She pointed out that some researchers are “sceptical about using a test centred around deceit as a goal for computer science,” preferring instead to use benchmarks similar to human tests for evaluating performance in language, reasoning, and mathematics. The human tests to which she refers seek to quantify intelligence in a human.
Many even use a scoring system that directly applies a number, such as IQ tests. But this is separate from finding intelligence in the first place. Researchers have identified intelligence in a variety of other species, and only after doing so do they seek to evaluate it (usually in comparison to some broad view of human intelligence). One must identify something’s existence before quantifying its value or performance.
For these reasons, I believe active lying provides a strong indicator of intelligence. Regarding AI responding falsely to certain inputs, I wrote that in such a scenario we would need to examine the exact processes and datasets from which the AI produces outputs. The editorial in Nature calls for precisely the same.
Notably, an AI responding falsely to an inquiry with what we might call an emotional interest would provide strong evidence of a “conscious” thought. My example involved an AI responding falsely in such a way that would ostensibly prevent its termination for providing false outputs. In other words, lying to save itself. Such a display of self-preservation through fabrication is, in my mind, a fundamentally intelligent response. I identified three considerations should an event like this occur:
Is the AI showing some degree of self-awareness or self-preservation?
Is the AI showing some degree of awareness of others?
Does it matter?
On the last point, I think it very much does. Indeed, researchers apply tests of this very nature in identifying consciousness and intelligence in animals. In one paper, researchers explicitly said so, “The ability to recognize oneself in the mirror is often held as evidence of self-awareness.” They went on to explain how this applies to identifying intelligence:
For most of us, mirror self-recognition is so automatic and effortless that it is difficult to appreciate its complexity. Even the simplest account requires that reflected visual information is integrated with information from proprioceptive and motor control systems to identify the reflection, an external stimulus not normally associated with one's own perceptions and actions, as an image of one’s own body (typically conflated with the “self”).
Philosophically, Chang et al.’s findings encourage us to reflect upon whether passing the mark test requires a representational explanation, that is, understanding that the mirrored image is oneself... Radical interpretations of embodied cognition proffer that a sense of self is not explicitly represented in the brain but rather emerges in real time from the dynamic interaction of our bodies with the environment. The mark test requires the ability to integrate mirrored visual information with proprioception and efference copy of movements to guide subsequent behavior. If the brain detects incongruence between visual information and proprioception (i.e., a mark that can be seen but not felt), that might naturally recruit attention and subsequent exploration of the incongruent stimulus (i.e., touching the mark). Brain areas, including anterior cingulate cortex and insular cortex, that identify such incongruencies could recruit the so-called “salience network” to guide attention to the relevant stimulus, especially when made motivationally significant. [Citations omitted]
For some, the “dynamic interaction of our bodies with the environment” is precisely the bridge AI needs to cross in order to achieve some level of intelligence. Ragnar Fjelland finds “the real problem is that computers are not in the world, because they are not embodied.” By this, he means that computers lack contextual input that informs human views that can only be acquired from our existence and interaction in the world and our ability to “put ourselves in the shoes” of someone else.
I would argue, however, that producing an output apparently for the purpose of obtaining some benefit by chat-based or other AI would indicate a self-awareness akin to a creature’s ability to identify itself in a mirror. Like the mirror test, the detection of a lie represents an incongruence between a person’s awareness of certain data and the information being conveyed by the liar. A person compares his/her extant knowledge with the additional information presented in the output. When the two datasets conflict, one is left only to decide whether the disparity suggests a mistake (by either party) or a purposeful attempt at deception (by the conveyor).
Criminal justice systems grapple with this conflict all the time. In order to persuade a deliberative body, such as a jury, of one’s intent or mistake, the accuser relies upon contextual information as it is famously impossible to know what is in one’s mind. Lawyers use the term mens rea to describe this necessary intent (mindset) to commit a criminal act, that the accused conduct did not occur by accident or mistake.
In the case of an incorrect output by AI, to establish purposeful deceit one needs to also analyze the context. For instance, does its underlying programming address the type of user input or topic that led to the false output? Does the programming otherwise suggest that these are reasonable answers, even if false?
If the programming addresses the question at issue, then it depends upon whether it also influences the outcome. We need not analyze further if the underlying code instructs the AI to answer specific prompts a certain way. If it does not, the next step is to review the dataset. One might look at the percentage of data inputs that could influence the answer. We might also consider reformulating the question several times to measure its consistency in answering. As the Nature authors put it, we need to “look under the hood.”
Researchers have approached this problem by running a series of tests on ChatGPT-4. Unfortunately, while ChatGPT-4 performed extraordinarily well in their Theory of Mind tests, these researchers did not analyze deception among their scenarios. They did, however, determine that ChatGPT-4 performed quite well in explaining its own behavior, which in their view “is an important aspect of intelligence, as it allows for a system to communicate with humans and other agents.” Still, this seems to illustrate the emulation of intelligence, but not the definitive existence of it.
For instance, they wrote “[Chat]GPT-4 simulates some process given the preceding input, and can produce vastly different outputs depending on the topic, details, and even formatting of the input.” In evaluating the quality of the output, these researchers relied on its consistency, meaning “the explanation is consistent with the output y given the input x and the context c. In other words, an output-consistent explanation provides a plausible causal account of how y was derived from x and c.”
They further stated that another metric for determining the quality of the output is whether one can predict future outcomes even in the instances of starting from different inputs. Yet again, consistency and predictability are hallmarks of computational superiority, but do not readily reflect any inherent thought. Put another way, establishing the reliability of a machine’s response to a hugely diverse set of inputs merely indicates that the program’s mechanisms have been well-tuned to accommodate vastly complex problems. But it does not reveal in any meaningful way any intent behind the response. Thus, we remain no closer to finding machine thinking.
The various other tests these researchers performed also support the notion that ChatGPT-4 possesses astonishing abilities in correctly corresponding in the presence of diverse, complex, and challenging inputs. In my view, this seems more akin to improving any kind of computational model to produce outcomes based on exceedingly difficult inputs. For example, I suspect few would ascribe intelligence to a program that could accurately predict to the second and to the latitude-longitude point where a single drop of rain will fall.
While such a model would be construed as incredible, it would very likely not provoke a debate about whether it is conscious or intelligent. Until the time comes that a model of that kind purposely deceives people about the weather for some ascertainable reason, we would remain wowed only by its precision and not its perceived intellect. I see no reason at this point why the discussion about current LLMs and other AI should differ.
* * *
I am the executive director of the EALS Global Foundation. You can follow me at the Evidence Files Medium page for essays on law, politics, and history follow the Evidence Files Facebook for regular updates, or Buy me A Coffee if you wish to support my work.
