Hire an Illini
Chi Han
- Advisor:
- Heng Ji
- Departments:
- Areas of Expertise:
- Representation Understanding
- Large Language Models
- Natural Language Processing
- Thesis Title:
- Mindreading and Surpassing Limits for Large Language Models
- Thesis abstract:
- Although displaying impressive generation powers, the internal representations of large language models (LLMs) remain mostly black-box, baffling an understanding of their properties and adapting LLMs beyond their limits. My work addresses this challenge by combining theoretical understanding and empirical adaptation of LLMs. My paper builds upon understanding the properties of language models' internal representations (mindreading) to develop methods to transcend their original boundaries (surpassing Limits). For example, LM-Infinite analyzes the properties of attention features, which inspires a method for zero-shot extreme length generalization for LLMs. Another work, LM-Switch reveals the theoretical structure of word embeddings to steer language models in an efficient, flexible, transparent, and transferrable way. A third work interprets in-context learning as kernel regression, which sheds light on a series of empirical phenomena during in-context learning.
- Downloads:
Contact information:
chihan3@illinois.edu