RHOS
About
Hi, this is the website of RHOS team at MVIG. We study Human Activity Understanding, Visual Reasoning, and Embodied AI. We are building a knowledge-driven system that enables intelligent agents to perceive human activities, reason human behavior logics, learn skills from human activities, and interact with environment.
Research Interests:
(S) Embodied AI: how to make agents learn skills from humans and interact with human & scene & object.
(S-1) Human Activity Understanding: how to learn and ground complex/ambiguous human activity concepts (body motion, human-object/human/scene interaction) and object concepts from multi-modal information (2D-3D-4D).
(S-2) Visual Reasoning: how to mine, capture, and embed the logics and causal relations from human activities.
(S-3) General Multi-Modal Foundation Models: especially for human-centric perception tasks.
(S-4) Activity Understanding from A Cognitive Perspective: work with multidisciplinary researchers to study how the brain perceives activities.
(E) Human-Robot Interaction (e.g. for Smart Hospital): work with the healthcare team (doctors and engineers) in SJTU to develop intelligent robots to help people.
Contact
Email: yonglu_li[at]sjtu[dot]edu[dot]cn
Office: SEIEE-3-301
Shanghai Jiao Tong University
Recruitment
We are actively looking for self-motivated students (Master/PhD, 2024 spring & fall), interns / engineers / visitors (CV/ML/ROB/NLP/Math/Phys background, always welcome) to join us in Machine Vision and Intelligence Group (MVIG). If you share same/similar interests, feel free to drop me an email with your resume.
Click here for more details.
News and Olds
Projects
HAKE
Human Activity Knowledge Engine (HAKE) is a knowledge-driven system that aims at enabling intelligent agents to perceive human activities, reason human behavior logics, learn skills from human activities, and interact with objects and environments.
OCL
We propose a challenging Object Concept Learning (OCL) task to push the envelope of object understanding. It requires machines to reason out object affordances and simultaneously give the reason: what attributes make an object possesses these affordances.
Pangea
We design an action semantic space in view of verb taxonomy hierarchy and covering massive actions. Thus, we can gather multi-modal datasets into a unified database in a unified label system, i.e., bridging “isolated islands” into a “Pangea”. Accordingly, we propose a bidirectional mapping model between physical and semantic space to fully use Pangea.
EgoPCA
We rethinks and proposes a new framework as an infrastructure to advance Ego-HOI recognition by Probing, Curation and Adaption (EgoPCA). We contribute comprehensive pre-train sets, balanced test sets and a new baseline, which are complete with a training-finetuning strategy and several new and effective mechanisms and settings to advance further research.
Shadow-Teleop
A teleopration system for Shadow Dexterous Hand.
Check out some of our work
Publications
*=equal contribution
#=corresponding author
Symbol-LLM: Leverage Language Models for Symbolic System in Visual Human Activity Reasoning
Beyond Object Recognition: A New Benchmark towards Object Concept Learning
Discovering A Variety of Objects in Spatio-Temporal Human-Object Interactions
HAKE: Human Activity Knowledge Engine
| Main Repo: | HAKE Star | ||
| Sub-repos: | Torch Star | TF Star | HAKE-AVA Star |
| Halpe Star | HOI List Star |
Human Trajectory Prediction with Momentary Observation
HOI Analysis: Integrating and Decomposing Human-Object Interaction
PaStaNet: Toward Human Activity Knowledge Engine
Oral Talk: Compositionality in Computer Vision in CVPR 2020
