Lerrel Pinto

I am an Assistant Professor of Computer Science at NYU Courant working on problems in Robotics and Machine Learning. I am also affiliated with the Center for Data Science. Together with several wonderful colleagues I am part of the CILVR (Computational Intelligence, Learning, Vision and Robotics) group.

My lab’s goal is to get robots to generalize and adapt in the diverse world we live in. To this end, my research touches the areas of Robot Learning, Representation Learning, Reinforcement Learning, and Affordable Robotics. A talk on our recent robotics efforts is here.

Recent News

Recent Talks

Here are some public talks that covers my recent research:

Courses Taught at NYU

Selected Research and Publications


Teach a Robot to FISH: Versatile Imitation from One Minute of Demonstrations
Teach a Robot to FISH: Versatile Imitation from One Minute of Demonstrations
Siddhant Haldar, Jyothish Pari, Anant Rai, Lerrel Pinto
In RSS 2023 (Best Student Paper Award)
PDF Cite Code Project

CLIP-Fields: Weakly Supervised Semantic Fields for Robotic Memory
CLIP-Fields: Weakly Supervised Semantic Fields for Robotic Memory
Mahi Shafiullah, Chris Paxton, Lerrel Pinto, Soumith Chintala, Arthur Szlam
In RSS 2023 (Outstanding Paper Award at the LangRob workshop)
PDF Cite Code Project

From play to policy: Conditional behavior generation from uncurated robot data
From play to policy: Conditional behavior generation from uncurated robot data
Jeff Cui, Yibin Wang, Mahi Shafiullah, Lerrel Pinto
In ICLR 2023 (Notable Top-5% Paper / Oral Presentation)
PDF Cite Code Project

Holo-Dex: Teaching Dexterity with Immersive Mixed Reality
Holo-Dex: Teaching Dexterity with Immersive Mixed Reality
Sridhar Pandian Arunachalam, Irmak Güzey, Soumith Chintala, Lerrel Pinto
In ICRA 2023
PDF Cite Code Project

Behavior Transformers: Cloning k modes with one stone
Behavior Transformers: Cloning k modes with one stone
Mahi Shafiullah, Jeff Cui, Ariuntuya Altanzaya, Lerrel Pinto
In NeurIPS 2022 (Nominated for Outstanding Paper Award)
PDF Cite Code Project

Watch and Match: Supercharging Imitation with Regularized Optimal Transport
Watch and Match: Supercharging Imitation with Regularized Optimal Transport
Siddhant Haldar, Vaibhav Mathur, Denis Yarats, Lerrel Pinto
In CoRL 2022 (Best Paper Award Finalist)
PDF Cite Code Project Video

Dexterous Imitation Made Easy: A Learning-Based Framework for Efficient Dexterous Manipulation
Dexterous Imitation Made Easy: A Learning-Based Framework for Efficient Dexterous Manipulation
Sridhar Pandian Arunachalam, Sneha Silwal, Ben Evans, Lerrel Pinto
In ICRA 2023
PDF Cite Code Project Video

Don't Change the Algorithm, Change the Data: Exploratory Data for Offline Reinforcement Learning
Don't Change the Algorithm, Change the Data: Exploratory Data for Offline Reinforcement Learning
Denis Yarats, David Brandfonbrener, Hao Liu, Michael Laskin, Pieter Abbeel, Alessandro Lazaric, Lerrel Pinto
Preprint
PDF Cite Code

The Surprising Effectiveness of Representation Learning for Visual Imitation
The Surprising Effectiveness of Representation Learning for Visual Imitation
Jyothish Pari, Nur Muhammad Shafiullah, Sridhar Pandian Arunachalam, Lerrel Pinto
In RSS 2022
PDF Cite Code Project

Context is Everything: Implicit Identification for Dynamics Adaptation
Context is Everything: Implicit Identification for Dynamics Adaptation
Ben Evans, Abitha Thankaraj, Lerrel Pinto
In ICRA 2022
PDF Cite Code Project Video

Mastering Visual Continuous Control: Improved Data-Augmented Reinforcement Learning
Mastering Visual Continuous Control: Improved Data-Augmented Reinforcement Learning
Denis Yarats, Rob Fergus, Alessandro Lazaric, Lerrel Pinto
In ICLR 2022
PDF Cite Code

Reinforcement Learning with Prototypical Representations
Reinforcement Learning with Prototypical Representations
Denis Yarats, Rob Fergus, Alessandro Lazaric, Lerrel Pinto
In ICML 2021
PDF Cite Code

Learning Cross-Domain Correspondence for Control with Dynamics Cycle-Consistency
Learning Cross-Domain Correspondence for Control with Dynamics Cycle-Consistency
Qiang Zhang, Tete Xiao, Alexei A. Efros, Lerrel Pinto, Xiaolong Wang
In ICLR 2021 (Oral Presentation)
PDF Cite Code Project

Self-Supervised Policy Adaptation during Deployment
Self-Supervised Policy Adaptation during Deployment
Nicklas Hansen, Rishabh Jangir, Yu Sun, Guillem Alenya, Pieter Abbeel, Alexei A. Efros, Lerrel Pinto, Xiaolong Wang
In ICLR 2021 (Spotlight Presentation)
PDF Cite Code Project

Task-Agnostic Morphology Evolution
Task-Agnostic Morphology Evolution
Donald J. Hejna III, Pieter Abbeel, Lerrel Pinto
In ICLR 2021
PDF Cite Code Project

Visual Imitation Made Easy
Visual Imitation Made Easy
Sarah Young, Dhiraj Gandhi, Shubham Tulsiani, Abhinav Gupta, Pieter Abbeel, Lerrel Pinto
In CoRL 2020
PDF Cite Code Project

Learning Predictive Representations for Deformable Objects Using Contrastive Estimation
Learning Predictive Representations for Deformable Objects Using Contrastive Estimation
Wilson Yan, Ashwin Vangipuram,, Pieter Abbeel, Lerrel Pinto
In CoRL 2020
PDF Cite Code Project

Robust Policies via Mid-Level Visual Representations
Robust Policies via Mid-Level Visual Representations
Bryan Chen, Alexander Sax, Francis E. Lewis, Silvio Savarese, Jitendra Malik, Amir Zamir, Lerrel Pinto
In CoRL 2020
PDF Cite Code Project

Automatic Curriculum Learning through Value Disagreement
Automatic Curriculum Learning through Value Disagreement
Yunzhi Zhang, Pieter Abbeel, Lerrel Pinto
In NeurIPS 2020
PDF Cite Code Project

Generalized Hindsight for Reinforcement Learning
Generalized Hindsight for Reinforcement Learning
Alexander Li, Lerrel Pinto, Pieter Abbeel
In NeurIPS 2020
PDF Cite Project

Reinforcement Learning with Augmented Data
Reinforcement Learning with Augmented Data
Michael Laskin, Kimin Lee, Adam Stooke, Lerrel Pinto, Pieter Abbeel, Aravind Srinivas
In NeurIPS 2020 (Spotlight Presentation)
PDF Cite Code Project

Learning to Manipulate Deformable Objects without Demonstrations
Learning to Manipulate Deformable Objects without Demonstrations
Yilin Wu, Wilson Yan, Thanard Kurutach, Lerrel Pinto, Pieter Abbeel
In RSS 2020
PDF Cite Code Project Video

Swoosh! Rattle! Thump! - Actions that Sound
Swoosh! Rattle! Thump! - Actions that Sound
Dhiraj Gandhi, Abhinav Gupta, Lerrel Pinto
In RSS 2020
PDF Cite Project

Hierarchically Decoupled Imitation for Morphological Transfer
Hierarchically Decoupled Imitation for Morphological Transfer
Donald J. Hejna III, Pieter Abbeel, Lerrel Pinto
In ICML 2020
PDF Cite Code Project

PyRobot: An Open-source Robotics Framework for Research and Benchmarking
PyRobot: An Open-source Robotics Framework for Research and Benchmarking
Adithya Murali, Tao Chen, Kalyan Vasudev Alwala, Dhiraj Gandhi, Lerrel Pinto, Saurabh Gupta, Abhinav Gupta
Open source platform (https://www.pyrobot.org/)
PDF Cite Code

Multiple Interactions Made Easy (MIME): Large Scale Demonstrations Data for Imitation
Multiple Interactions Made Easy (MIME): Large Scale Demonstrations Data for Imitation
Pratyusha Sharma, Lekha Mohan, Lerrel Pinto, Abhinav Gupta
In CoRL 2018
PDF Cite

Robot Learning in Homes: Improving Generalization and Reducing Dataset Bias
Robot Learning in Homes: Improving Generalization and Reducing Dataset Bias
Abhinav Gupta, Adithya Murali, Dhiraj Gandhi, Lerrel Pinto
In NeurIPS 2018
PDF Cite Project

Asymmetric Actor Critic for Image-Based Robot Learning
Asymmetric Actor Critic for Image-Based Robot Learning
Lerrel Pinto, Marcin Andrychowicz, Peter Welinder, Wojciech Zaremba, Pieter Abbeel
In RSS 2018
PDF Cite Project

CASSL: Curriculum Accelerated Self-Supervised Learning
CASSL: Curriculum Accelerated Self-Supervised Learning
Adithyavairavan Murali, Lerrel Pinto, Dhiraj Gandhi, Abhinav Gupta
In ICRA 2018
PDF Cite Video

Learning to Fly by Crashing
Learning to Fly by Crashing
Dhiraj Gandhi, Lerrel Pinto, Abhinav Gupta
In IROS 2017
PDF Cite Video

Robust Adversarial Reinforcement Learning
Robust Adversarial Reinforcement Learning
Lerrel Pinto, James Davidson, Rahul Sukthankar, Abhinav Gupta
In ICML 2017
PDF Cite

Learning to Push by Grasping: Using multiple tasks for effective learning
Learning to Push by Grasping: Using multiple tasks for effective learning
Lerrel Pinto, Abhinav Gupta
In ICRA 2017
PDF Cite Video

Supervision via Competition: Robot Adversaries for Learning Tasks
Supervision via Competition: Robot Adversaries for Learning Tasks
Lerrel Pinto, James Davidson, Abhinav Gupta
In ICRA 2017
PDF Cite Code Video

The Curious Robot: Learning Visual Representations via Physical Interactions
The Curious Robot: Learning Visual Representations via Physical Interactions
Lerrel Pinto, Dhiraj Gandhi, Yuanfeng Han, Yong-Lae Park, Abhinav Gupta
In ECCV 2016 (Spotlight Presentation)
PDF Cite

Supersizing Self-supervision: Learning to Grasp from 50K Tries and 700 Robot Hours
Supersizing Self-supervision: Learning to Grasp from 50K Tries and 700 Robot Hours
Lerrel Pinto, Abhinav Gupta
In ICRA 2016 (Best Student Paper Award)
PDF Cite