Colloquium - Jason Xinyi Liu, "Robotic Language Grounding"

Abstract: 
Natural language provides an intuitive and flexible way for humans to communicate with robots. However, understanding diverse, ambiguous language commands is challenging. Grounding language to structured task specifications enables autonomous robots to understand a broad range of natural language and solve long-horizon tasks with safety guarantees. Linear temporal logic (LTL) provides unambiguous semantics for language grounding, and its compositionality can induce skill transfer.

In this talk, I will first propose two language grounding systems. 1) Lang2LTL is a modular system that uses large language models (LLMs) to ground navigation commands with diverse temporal patterns to LTL task specifications in novel environments without retraining. 2) Improved upon its predecessor, Lang2LTL-2 uses LLMs and a pretrained vision-language model to ground spatiotemporal navigation commands. By translating language to LTL, both systems can detect infeasible task specifications and abort execution when necessary. 3) Next, I will introduce LTL-Transfer, a zero-shot transfer algorithm that leverages the compositionality of LTL to reuse learned skills to solve novel tasks without violating any safety constraints. In future work, I want to develop robotic systems that produce robust and verifiable behavior by integrating multimodal grounding and human-robot dialogue.

Relevant Papers:
https://arxiv.org/abs/2302.11649
https://ieeexplore.ieee.org/document/10802696
https://arxiv.org/abs/2206.05096
https://arxiv.org/abs/2405.13245 

Bio:
Jason Xinyu Liu is a postdoctoral researcher at MIT, working with Prof. Julie Shah. He received his Ph.D. in Computer Science at Brown University, advised by Prof. Stefanie Tellex. Jason is working towards developing autonomous robots that assist people. His research lies in the intersection of robotics, natural language processing, machine learning, and formal methods. His work has appeared at CoRL, ICRA, IROS, IJCAI, and AAAI Symposiums. He earned his Bachelor's degree in Electrical Engineering and Computer Sciences from UC Berkeley. His research has been generously funded by Amazon, the Office of Naval Research, the NSF Graduate Research Fellowship Program, and the Jack Kent Cooke Foundation Graduate Scholarship.