Recently, with the emergence of GPT, robot planning and reasoning systems based on basic large language models have been rapidly developed, but the current SOTA system (Robot Transformer-2, RoboAgent) already suffers from a number of shortcomings:
(a) Insufficient robustness to external disturbances: current mainstream models perform poorly in the face of external disturbances, are difficult to replan strategies in real time in dynamic environments, and often require manual intervention or are unable to automatically adapt to new situations, and thus are inefficient in the face of external changes.
(b) Data Efficiency Problem: Many SOTA systems require a large amount of data during training, which can lead to data inefficiency. Improving these systems to increase data efficiency is an important challenge.
(c) Insufficient scalability and migratability: current SOTA systems are usually only able to perform well in specific tasks or domains, and struggle to handle a wider range of tasks. Even for very similar environments, it is difficult to do policy migration.
Specifically, on dexterous hand reasoning and planning: current mainstream models perform well in executing strategies in static environments, but still lack robust and effective Re-Planning capabilities in the face of external disturbances. Our research goal is to address this problem by proposing a new planning approach with a dual-layer planner with replaceable strategies and goals at both coarse and fine grains to cope with changing environments; at the same time, we will introduce a planner and an executor capable of generating or replacing new strategies when planning errors are detected at both the vision and executor side to ensure that the target task is accomplished.