Yoav Artzi, "Sparks of New Pre-Training"

Abstract: 
Pre-training is the first stage of LLM training. This is when the fundamental qualities of the model are formed, and when most data is seen. It is largely considered settled, and beside scaling up, has changed little since GPT-2. This talk aims for fundamental changes in pre-training, with the goal of building models with new core qualities. We will cover two projects. First, we introduce the hypothesis that state and prediction representations, which are entangled in transformers, are better separated. We design a simple architectural modification that effectively separates them, and provides 2.6x data efficiency. The second project is focused on externalizing knowledge by pre-training an LLM to rely on an external knowledge base, while inducing this KB from the pre-training data. We introduce Limited Memory Language Models (LMLMs), a new class of LLMs. A tiny 360M parameter LMLM model outperforms models trained on 40x the data, and shows factuality performance on par with Claude Sonnet 4.5 and above OpenAI's 120B open model. All while providing a new slate of LMLM-unique advantages: knowledge control, provenance, editing, and factuality. Together, these techniques demonstrate two of many possible avenues to bring about fundamental change in LLMs through new pre-training paradigms.

Bio:
Yoav Artzi is an Associate Professor in the Department of Computer Science and Cornell Tech at Cornell University, and a visiting faculty research at Google DeepMind. His research focuses on language modeling and learning in interactive and situated scenarios. His work was acknowledged by awards and honorable mentions at ACL, EMNLP, NAACL, and IROS, as well as a TACL test-of-time award. Yoav holds a B.Sc. from Tel Aviv University and a Ph.D. from the University of Washington. He was previously and arXiv's associate faculty director, and co-founded COLM.