Welcome to the official page for our AAAI 2025 tutorial!

Overview

Outline: Introducing A Emerging Science of Language Models

As large language models (LMs) continue to transform natural language processing and AI applications, they have become the backbone of AI-driven systems, consuming substantial economic and research resources while taking on roles that significantly influence society. The efficient, guaranteed, and reliable development of future language models demands the establishment of scientific analysis frameworks for language models. While extensive engineering insights and empirical observations have been accumulated, the immense resource costs of model development make blind explorations unsustainable. This necessitates scientific perspectives, guiding principles, and theoretical frameworks for studying language models. Though a systematic foundation for this emergent protoscience is yet to be fully constructed, preliminary studies already illuminate innovative adaptations of language models. This tutorial covers groundbreaking advancements and emerging insights in this new science. It aims to provide LM developers with guidelines for model development, provide interdisciplinary researchers with tools for creative application across domains, offer LM users predictive laws and guarantees of model behavior, and present the public with a deeper understanding of LMs.

The tutorial starts with the motivations for establishing a systematic science of language models (LMs), highlighting the need for guiding principles to overcome the rising costs and limitations of empirical approaches. Participants will explore the key challenges this field seeks to address, including predicting model behavior, interpreting internal mechanisms, optimizing training processes, and resolving issues like scaling inefficiencies and representation limitations. By integrating foundational components of this emerging discipline, the tutorial examines internal LM mechanisms, scaling behaviors, and theoretical frameworks, offers insights into the physiology of LM representations, and showcases how internal components process information alongside groundbreaking innovations. This comprehensive overview bridges practical applications with a deeper theoretical foundation, ultimately aiming to improve model transparency and reliability.

While a basic understanding of NLP and LM concepts (e.g., tokenization, embeddings, attention-mechanism, LM architectures) and a general AI research background are beneficial, this tutorial is designed to be accessible to all. We will provide high-level explanations to ensure that all participants, regardless of their expertise level, can engage and learn from this session.

Speakers

Chi Han

Chi Han

Chi Han is a Ph.D. candidate in computer science at UIUC, advised by Prof. Heng Ji. His research focuses on understanding and adapting large language model (LLM) representations. He has first-authored papers in NeurIPS, ICLR, NAACL, and ACL, earning Outstanding Paper Awards at NAACL 2024 and ACL 2024.

Heng Ji

Heng Ji

Heng Ji, a Professor at UIUC and Amazon Scholar, directs the Amazon-Illinois Center on AI for Interactive Conversational Experiences. With expertise in NLP, multimedia multilingual information extraction, and knowledge-enhanced language models, she has earned numerous accolades, including IEEE’s “AI’s 10 to Watch,” NSF CAREER, and multiple Best Paper Awards.

Materials

To be updated.