Sherry Yang (杨梦娇) Email: sherryy at {google.com, berkeley.edu}   I am a researcher at Google DeepMind. I work on machine learning and sequential decision making. Recently: Foundation models: I am interested in learning generative world models and agents from internet-scale video, language, and structured data. Decision making algorithms: I work on reinforcement learning (RL) algorithms that can leverage broad knowledge. Applications: I am interested in applications empowered by combining internet-scale knowledge with decision making algorithms. Examples include but are not limited to robot learning and materials discovery. I received my PhD in Computer Science from UC Berkeley advised by Pieter Abbeel, and B.S. and M.Eng. from MIT advised by Patrick Winston and Julian Shun.   Publications | Google Scholar | Twitter | Talks | Services | Teaching

News

• Our work UniSim: Learning Interactive Real-World Simulators received the Outstanding Paper Award at ICLR 2024.

• I talked about foundation models as real-world simulators at the CVPR Foundation Models for Autonomous Systems workshop [recording].

• I talked about our position paper, Video as the New Language for Real-World Decision Making, on the TWIML podcast.

Publications

• Video as the New Language for Real-World Decision Making

  Sherry Yang, Jacob Walker, Jack Parker-Holder, Yilun Du, Jake Bruce, Andre Barreto, Pieter Abbeel, Dale Schuurmans.

  International Conference on Machine Learning (ICML), 2024.




• Code as Reward: Empowering Reinforcement Learning with VLMs

  David Venuto, Sami Nur Islam, Martin Klissarov, Doina Precup, Sherry Yang, Ankit Anand.

  International Conference on Machine Learning (ICML), 2024.




• Learning Interactive Real-World Simulators [website]

  Sherry Yang, Yilun Du, Kamyar Ghasemipour, Jonathan Tompson, Leslie Kaelbling, Dale Schuurmans, Pieter Abbeel.

  International Conference on Learning Representations (ICLR), 2024. Outstanding Paper Award.

  NeurIPS workshop on Instruction Tuning and Instruction Following, 2023. Best Paper.




• Scalable Diffusion for Materials Generation [website]

  Sherry Yang, KwangHwan Cho, Amil Merchant, Pieter Abbeel, Dale Schuurmans, Igor Mordatch, Ekin Dogus Cubuk.

  International Conference on Learning Representations (ICLR), 2024.

  NeurIPS workshop on AI4Mat, 2023. Spotlight.




• Video Language Planning [website]

  Yilun Du, Sherry Yang, Pete Florence, Fei Xia, Ayzaan Wahid, Brian Ichter, Pierre Sermanet, Tianhe Yu, Pieter Abbeel, Joshua B. Tenenbaum, Leslie Kaelbling, Andy Zeng, Jonathan Tompson.

  International Conference on Learning Representations (ICLR), 2024.




• Probabilistic Adaptation of Text-to-Video Models [website]

  Sherry Yang*, Yilun Du*, Bo Dai, Dale Schuurmans, Joshua B. Tenenbaum, Pieter Abbeel.

  International Conference on Learning Representations (ICLR), 2024.




• Foundation Models for Decision Making: Problems, Methods, and Opportunities [website]

  Sherry Yang, Ofir Nachum, Yilun Du, Jason Wei, Pieter Abbeel, Dale Schuurmans.




• Learning Universal Policies via Text-Guided Video Generation [website] [slides] [recording]

  Yilun Du*, Sherry Yang*, Bo Dai, Hanjun Dai, Ofir Nachum, Joshua B. Tenenbaum, Dale Schuurmans, Pieter Abbeel.

  Advances in Neural Information Processing Systems (NeurIPS), 2023. Spotlight.




• Dichotomy of control: Separating what you can control from what you cannot [code] [slides] [recording]

  Sherry Yang, Dale Schuurmans, Pieter Abbeel, Ofir Nachum.

  International Conference on Learning Representations (ICLR), 2023. Oral.




• Multi-Environment Pretraining Enables Transfer to Action Limited Datasets

  David Venuto*, Sherry Yang*, Pieter Abbeel, Doina Precup, Igor Mordatch, Ofir Nachum.

  International Conference on Machine Learning (ICML), 2023.




• Chain of Thought Imitation Learning with Procedure Cloning [code] [slides] [poster] [website]

  Sherry Yang, Dale Schuurmans, Pieter Abbeel, Ofir Nachum.

  Advances in Neural Information Processing Systems (NeurIPS), 2022.




• Multi-Game Decision Transformers [code] [blog]

  Kuang-Huei Lee, Ofir Nachum, Sherry Yang, Lisa Lee, Daniel Freeman, Winnie Xu, Sergio Guadarrama, Ian Fischer, Eric Jang, Henryk Michalewski, Igor Mordatch.

  Advances in Neural Information Processing Systems (NeurIPS), 2022. Oral.




• Making linear mdps practical via contrastive representation learning

  Tianjun Zhang, Tongzheng Ren, Sherry Yang, Joseph Gonzalez, Dale Schuurmans, Bo Dai.

  International Conference on Machine Learning (ICML), 2022.




• Marginal Distribution Adaptation for Discrete Sets via Module-Oriented Divergence Minimization

  Hanjun Dai, Sherry Yang, Yuan Xue, Dale Schuurmans, Bo Dai.

  International Conference on Machine Learning (ICML), 2022.




• Offline RL for Natural Language Generation with Implicit Language Q Learning [code] [website]

  Charlie Snell, Ilya Kostrikov, Yi Su, Sherry Yang, Sergey Levine.

  International Conference on Learning Representations (ICLR), 2023.




• Offline Policy Selection under Uncertainty [code] [slides] [recording]

  Sherry Yang*, Bo Dai*, Ofir Nachum*, George Tucker, Dale Schuurmans.

  Artificial Intelligence and Statistics Conference (AISTATS), 2022.




• CHAI: A CHatbot AI for Task-Oriented Dialogue with Offline Reinforcement Learning [website]

  Siddharth Verma, Justin Fu, Sherry Yang, Sergey Levine.

  Findings of the Association for Computational Linguistics: NAACL, 2022. Oral.




• Context-Aware Language Modeling for Goal-Oriented Dialogue Systems [website]

  Charlie Snell, Sherry Yang, Justin Fu, Yi Su, Sergey Levine.

  Findings of the Association for Computational Linguistics: NAACL, 2022




• Combiner: Full Attention Transformer with Sparse Computation Cost [code]

  Hongyu Ren, Hanjun Dai, Zihang Dai, Sherry Yang, Jure Leskovec, Dale Schuurmans, Bo Dai.

  Advances in Neural Information Processing Systems (NeurIPS), 2021.




• Towards Automatic Evaluation of Dialog Systems: A Model-Free Off-Policy Evaluation Approach [code]

  Haoming Jiang, Bo Dai, Sherry Yang, Tuo Zhao, Wei Wei.

  Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing (EMNLP), 2021. Oral.




• Benchmarks for Deep Off-Policy Evaluation [code]

  Justin Fu, Mohammad Norouzi, Ofir Nachum, George Tucker, Ziyu Wang, Alexander Novikov, Sherry Yang, Michael R. Zhang, Yutian Chen, Aviral Kumar, Cosmin Paduraru, Sergey Levine, Thomas Paine.

  International Conference on Learning Representations (ICLR), 2021.




• TRAIL: Near-Optimal Imitation Learning with Suboptimal Data [code] [slides] [recording] [website]

  Sherry Yang, Sergey Levine, Ofir Nachum.

  International Conference on Learning Representations (ICLR), 2021.




• Provable Representation Learning for Imitation with Contrastive Fourier Features [code]

  Ofir Nachum, Sherry Yang.

  Advances in Neural Information Processing Systems (NeurIPS), 2021.




• Representation Matters: Offline Pretraining for Sequential Decision Making [code] [slides]

  Sherry Yang, Ofir Nachum.

  International Conference on Machine Learning (ICML), 2021.




• Off-Policy Evaluation via the Regularized Lagrangian [code] [slides]

  Sherry Yang*, Ofir Nachum*, Bo Dai*, Lihong Li, Dale Schuurmans.

  Advances in Neural Information Processing Systems (NeurIPS), 2020.




• Energy-Based Processes for Exchangeable Data [code] [slides] [recording]

  Sherry Yang*, Bo Dai*, Hanjun Dai, Dale Schuurmans.

  International Conference on Machine Learning (ICML), 2020.




• Benchmarking Attribution Methods with Relative Feature Importance [code] [poster]

  Sherry Yang, Been Kim.

  Workshop on Human-Centric Machine Learning, NeurIPS, 2019. Oral.




• GraphIt: A High-Performance Graph DSL [code] [slides] [website]

  Yunming Zhang, Sherry Yang, Riyadh Baghdadi, Shoaib Kamil, Julian Shun, Saman Amarasinghe.

  Object-oriented Programming, Systems, Languages, and Applications (OOPSLA), 2018.




• Cache and NUMA Optimizations in A Domain-Specific Language for Graph Processing [slides]

  Sherry Yang.

  M.Eng. Thesis, Massachusetts Institute of Technology, 2018.

Talks

• June 2024: Foundation Models as Real-World Simulators.

  • Invited Talk at CVPR 2024 Foundation Models for Autonomous Systems workshop [recording].

• May 2024: UniSim: Learning Interactive Real-World Simulators.

  • Oral presentation at ICLR 2024.

  • Generative Models for Decision Making workshop at ICLR 2024.

• March 2024 - April 2024: Decision Making with Internet-Scale Knowledge.

  • Columbia, Cornell, Cornell Tech, New York University, University of Pennsylvania, Johns Hopkins University, University of Toronto, University of California San Diego, University of Southern California.

• March 2024: Video as the New Language for Real-World Decision Making.

  • This Week in Machine Learning (TWIML) [recording].

• October 2023: Foundation Models for Decision Making.

  • Stanford Intelligent and Interactive Autonomous Systems group and Intelligence through Robotic Interaction group.

  • University of Washington Robotics and State Estimation group and Embodied Intelligence and Robotics Development group.

  • UT Austin RL reading group.

  • CMU Artificial Intelligence Seminar.

  • NYU Computational Intelligence, Vision, and Robotics lab.

  • Princeton Computer Vision group.

  • MIT Computational Sensorimotor Learning Seminar.

  • Georgia Tech Interactive Computing.

  • Beijing Academy of Artificial Intelligence.

Services

• Organizer of the 1st and 2nd Workshop on Foundation Models for Decision Making at NeurIPS 2022 and 2023.

• Reviewer for NeurIPS, ICML, ICLR, AAAI BayLearn, SIGKDD. Program Committee for IJCAI.

• Reviewer for Google Faculty Research Awards, 2022-2024.

Teaching

• Guest lecturer for CSCI 677: Advanced Computer Vision, Fall 2023, USC.

• Guest lecturer for CSE 6243: Advanced Topics in Machine Learning, Fall 2023, Georgia Tech.

• Graduate Student Instructor of CS 188: Introduction to Artificial Intelligence, Fall 2022, UC Berkeley.

• Graduate Teaching Assistant of 6.886: Graph Analytics, Spring 2018, MIT.