In the beginning was the Word, and the Word was with Chatbot, and the Word was Chatbot

Given the profound importance of language and its various disciplines in technological developments, it is crucial to consider how chatbots function as products of advanced technology. This understanding contributes to recognising how chatbots learn through algorithmic cognition and how they respond effectively and accurately to diverse user queries, reflecting their systems in linguistic studies.

By introducing the argument above, there is little need to mention the importance of ‘the Word’ and, by extension, language and its specific disciplines, and what we humans have achieved over time through our enriched communication systems, especially in technological and diplomatic contexts, where words are an essential and powerful instrument. 

Since linguistics, especially nowadays, is an inseparable element of the realm of technology, it is absolutely legitimate to question how chatbots, the offshoots of the latest technology, work. In other words, it is legitimate to question how chatbots learn through digital (algorithmic) cognition and how they accurately and articulately express themselves in response to diverse queries and inputs.

What makes the human-like cognitive power of deep learning LLMs?

To understand AI and the epicentre of its evolution, chatbots, which interact with people by responding to most different prompts, we should delve into the branches of linguistics called semantics and syntax, and the process of learning and elaboration of most diverse and articulated info by chatbots. 

The complex understanding of language and how it is assimilated by humans – and in this case, by deep learning machines – was explained as far back as in some segments of language studies by Ferdinand de Saussure.

For that reason, we will explore the cognitive mechanisms underlying semantics and syntax in large language models (LLMs) such as ChatGPT, integrating the theoretical perspectives of one of the most renowned linguistic philosophers, Saussure. By synthesising linguistic theories with contemporary AI methodologies, the aim is to provide a comprehensive understanding of how LLMs process, understand, and generate natural language. What follows is a modest examination of the models’ training processes, data integration, and real-time interaction with users, highlighting the interplay between linguistic theories and AI language assimilation systems.

An overview of Saussure’s studies related to synta(x)gmatic relations and semantics 

Ferdinand de Saussure, one of the first linguistic scientists of the 20th century (along with Charles Sanders Peirce and Leonard Bloomfield), wrote an introduction to syntax and semantics in his Course in General Linguistics, where he depicts language as a scientific phenomenon, emphasising the synchronic study of language. This approach focuses on its current state rather than its historical evolution, in a structuralist view, with syntax and semantics as fundamental components of its structure.

Syntax

Syntax, within this framework, is a grammar discipline that represents and explains the systematic and linear arrangement of words and phrases to form meaningful sentences within a given language. Saussure views syntax as an essential aspect of language, an abstract system that encompasses grammar, vocabulary, and rules. He argues that syntax operates according to inherent principles and conventions established within a linguistic community, rather than being governed by individual speakers. His structuralist approach to linguistics highlights the interdependence between syntax and other linguistic elements, such as semantics, phonology, and morphology, within the overall structure of language.

Semantics

Semantics is a branch of linguistics and philosophy concerned with the study of meaning in language. It explores how words, phrases, sentences, and texts convey meaning and how interpretation is influenced by context, culture, and usage. Semantics covers various aspects, including the meaning of words (lexical semantics), the meaning of sentences (compositional semantics), and the role of context in understanding language (pragmatics).

However, one of Saussure’s biggest precepts within semantics posits that language is a system of signs composed of the signifier (sound/image) and the signified (concept). This dyadic structure is crucial for understanding how LLMs process the meaning of words and their possible ambiguity.

How do chatbots cognise semantics and syntax in linguistic processes?

Chatbots’ processing and understanding of language usage involves several key steps:

• Training on vast amounts of textual data from the internet to predict the next word in a sequence
• Tokenisation to divide the text into smaller units
• Learning relationships between words and phrases for semantic understanding
• Using vector representations to recognise similarities and generate contextually relevant responses
• Leveraging transformer architecture to efficiently process long contexts and complex linguistic structures

Although it does not learn in real time, the model is periodically updated with new data to improve performance, enabling it to generate coherent and useful responses to user queries.

As explained earlier, in LLMs, words and phrases are tokenised and transformed into vectors within a high-dimensional space. These vectors function similarly to Saussure’s signifiers, with their positions and relationships encoding meaning (the signified). Thus, within the process of ‘Tokenisation and Embedding’, LLMs tokenise text into discrete units (signifiers) and map them to embeddings that capture their meanings (signified). The model learns these embeddings by processing vast amounts of text, identifying patterns and relationships analogous to Saussure’s linguistic structures.

Chatbots’ ability to understand and generate text relies on their grasp of semantics (meaning) and syntax (structure). They process semantics through contextual word embeddings that capture meanings based on usage, an attention mechanism that weighs word importance in context, and layered contextual understanding that handles polysemy and synonymy. The model is pre-trained on general language patterns and fine-tuned on specific datasets for enhanced semantic comprehension. For syntax, it uses positional encoding to understand word order, attention mechanisms to maintain syntactic coherence, layered processing to build complex structures, and probabilistic grammar learning from vast text exposure. Tokenisation and sequence modelling help track dependencies and coherence, while the transformer model integrates syntax and semantics at each layer, ensuring that responses are both meaningful and grammatically correct. Training on diverse datasets further enhances its ability to generalise across various language uses, making the chatbot a powerful natural language processing tool.

An interesting invention

Recently, researchers in the Netherlands developed an AI platform capable of recognising sarcasm, which was presented at the Acoustical Society of America and Canadian Acoustical Association meeting. By training a neural network with the Multimodal Sarcasm Detection Dataset (MUStARD) using video clips and text from sitcoms like Friends and The Big Bang Theory, the large language model accurately detected sarcasm in about 75% of unlabelled exchanges.

Sarcasm generally takes the form of a, linguistically speaking, layered and ironic remark, often rooted in humour, that is intended to mock or satirise something. When a speaker is being sarcastic, they say something different from what they actually mean, and that’s why it is hard for a large language model to detect such nuances in someone’s speech.

This process leverages deep learning techniques that analyse both syntax and semantics, and the concepts of syntagma and idiom to understand the layered structure and meaning of language and how comprehensive the acquisition of human speech by an LLM is.

By integrating Saussure’s linguistic theories with the cognitive mechanisms of large language models, we gain a deeper understanding of how these models process and generate language. The interplay between structural rules, contextual usage, and the fluidity of meaning partially depicts the sophisticated performance of LLMs’ language generation. This synthesis not only illuminates the inner workings of contemporary AI systems but also reinforces the enduring relevance of classical linguistic theories in the age of AI.

The blog post was first published on DigWatch.