Robotics 67
☆ SPA: 3D Spatial-Awareness Enables Effective Embodied Representation
In this paper, we introduce SPA, a novel representation learning framework
that emphasizes the importance of 3D spatial awareness in embodied AI. Our
approach leverages differentiable neural rendering on multi-view images to
endow a vanilla Vision Transformer (ViT) with intrinsic spatial understanding.
We present the most comprehensive evaluation of embodied representation
learning to date, covering 268 tasks across 8 simulators with diverse policies
in both single-task and language-conditioned multi-task scenarios. The results
are compelling: SPA consistently outperforms more than 10 state-of-the-art
representation methods, including those specifically designed for embodied AI,
vision-centric tasks, and multi-modal applications, while using less training
data. Furthermore, we conduct a series of real-world experiments to confirm its
effectiveness in practical scenarios. These results highlight the critical role
of 3D spatial awareness for embodied representation learning. Our strongest
model takes more than 6000 GPU hours to train and we are committed to
open-sourcing all code and model weights to foster future research in embodied
representation learning. Project Page: https://haoyizhu.github.io/spa/.
☆ SG-Nav: Online 3D Scene Graph Prompting for LLM-based Zero-shot Object Navigation NeurIPS 2024
In this paper, we propose a new framework for zero-shot object navigation.
Existing zero-shot object navigation methods prompt LLM with the text of
spatially closed objects, which lacks enough scene context for in-depth
reasoning. To better preserve the information of environment and fully exploit
the reasoning ability of LLM, we propose to represent the observed scene with
3D scene graph. The scene graph encodes the relationships between objects,
groups and rooms with a LLM-friendly structure, for which we design a
hierarchical chain-of-thought prompt to help LLM reason the goal location
according to scene context by traversing the nodes and edges. Moreover, benefit
from the scene graph representation, we further design a re-perception
mechanism to empower the object navigation framework with the ability to
correct perception error. We conduct extensive experiments on MP3D, HM3D and
RoboTHOR environments, where SG-Nav surpasses previous state-of-the-art
zero-shot methods by more than 10% SR on all benchmarks, while the decision
process is explainable. To the best of our knowledge, SG-Nav is the first
zero-shot method that achieves even higher performance than supervised object
navigation methods on the challenging MP3D benchmark.
comment: Accepted to NeurIPS 2024. Project page:
https://bagh2178.github.io/SG-Nav/
☆ On the Evaluation of Generative Robotic Simulations
Due to the difficulty of acquiring extensive real-world data, robot
simulation has become crucial for parallel training and sim-to-real transfer,
highlighting the importance of scalable simulated robotic tasks. Foundation
models have demonstrated impressive capacities in autonomously generating
feasible robotic tasks. However, this new paradigm underscores the challenge of
adequately evaluating these autonomously generated tasks. To address this, we
propose a comprehensive evaluation framework tailored to generative
simulations. Our framework segments evaluation into three core aspects:
quality, diversity, and generalization. For single-task quality, we evaluate
the realism of the generated task and the completeness of the generated
trajectories using large language models and vision-language models. In terms
of diversity, we measure both task and data diversity through text similarity
of task descriptions and world model loss trained on collected task
trajectories. For task-level generalization, we assess the zero-shot
generalization ability on unseen tasks of a policy trained with multiple
generated tasks. Experiments conducted on three representative task generation
pipelines demonstrate that the results from our framework are highly consistent
with human evaluations, confirming the feasibility and validity of our
approach. The findings reveal that while metrics of quality and diversity can
be achieved through certain methods, no single approach excels across all
metrics, suggesting a need for greater focus on balancing these different
metrics. Additionally, our analysis further highlights the common challenge of
low generalization capability faced by current works. Our anonymous website:
https://sites.google.com/view/evaltasks.
comment: Project website: https://sites.google.com/view/evaltasks
☆ LiPO: LiDAR Inertial Odometry for ICP Comparison ICRA 2025
Darwin Mick, Taylor Pool, Madankumar Sathenahally Nagaraju, Michael Kaess, Howie Choset, Matt Travers
We introduce a LiDAR inertial odometry (LIO) framework, called LiPO, that
enables direct comparisons of different iterative closest point (ICP) point
cloud registration methods. The two common ICP methods we compare are
point-to-point (P2P) and point-to-feature (P2F). In our experience, within the
context of LIO, P2F-ICP results in less drift and improved mapping accuracy
when robots move aggressively through challenging environments when compared to
P2P-ICP. However, P2F-ICP methods require more hand-tuned hyper-parameters that
make P2F-ICP less general across all environments and motions. In real-world
field robotics applications where robots are used across different
environments, more general P2P-ICP methods may be preferred despite increased
drift. In this paper, we seek to better quantify the trade-off between P2P-ICP
and P2F-ICP to help inform when each method should be used. To explore this
trade-off, we use LiPO to directly compare ICP methods and test on relevant
benchmark datasets as well as on our custom unpiloted ground vehicle (UGV). We
find that overall, P2F-ICP has reduced drift and improved mapping accuracy,
but, P2P-ICP is more consistent across all environments and motions with
minimal drift increase.
comment: Submitted to ICRA 2025
☆ UW-SDF: Exploiting Hybrid Geometric Priors for Neural SDF Reconstruction from Underwater Multi-view Monocular Images IROS 2024
Due to the unique characteristics of underwater environments, accurate 3D
reconstruction of underwater objects poses a challenging problem in tasks such
as underwater exploration and mapping. Traditional methods that rely on
multiple sensor data for 3D reconstruction are time-consuming and face
challenges in data acquisition in underwater scenarios. We propose UW-SDF, a
framework for reconstructing target objects from multi-view underwater images
based on neural SDF. We introduce hybrid geometric priors to optimize the
reconstruction process, markedly enhancing the quality and efficiency of neural
SDF reconstruction. Additionally, to address the challenge of segmentation
consistency in multi-view images, we propose a novel few-shot multi-view target
segmentation strategy using the general-purpose segmentation model (SAM),
enabling rapid automatic segmentation of unseen objects. Through extensive
qualitative and quantitative experiments on diverse datasets, we demonstrate
that our proposed method outperforms the traditional underwater 3D
reconstruction method and other neural rendering approaches in the field of
underwater 3D reconstruction.
comment: 8 pages, 9 figures, presented at IROS 2024
☆ Dynamic Object Catching with Quadruped Robot Front Legs IROS 2024
This paper presents a framework for dynamic object catching using a quadruped
robot's front legs while it stands on its rear legs. The system integrates
computer vision, trajectory prediction, and leg control to enable the quadruped
to visually detect, track, and successfully catch a thrown object using an
onboard camera. Leveraging a fine-tuned YOLOv8 model for object detection and a
regression-based trajectory prediction module, the quadruped adapts its front
leg positions iteratively to anticipate and intercept the object. The catching
maneuver involves identifying the optimal catching position, controlling the
front legs with Cartesian PD control, and closing the legs together at the
right moment. We propose and validate three different methods for selecting the
optimal catching position: 1) intersecting the predicted trajectory with a
vertical plane, 2) selecting the point on the predicted trajectory with the
minimal distance to the center of the robot's legs in their nominal position,
and 3) selecting the point on the predicted trajectory with the highest
likelihood on a Gaussian Mixture Model (GMM) modelling the robot's reachable
space. Experimental results demonstrate robust catching capabilities across
various scenarios, with the GMM method achieving the best performance, leading
to an 80% catching success rate. A video demonstration of the system in action
can be found at https://youtu.be/sm7RdxRfIYg .
comment: Accepted to IROS 2024
☆ Probabilistic Satisfaction of Temporal Logic Constraints in Reinforcement Learning via Adaptive Policy-Switching
Constrained Reinforcement Learning (CRL) is a subset of machine learning that
introduces constraints into the traditional reinforcement learning (RL)
framework. Unlike conventional RL which aims solely to maximize cumulative
rewards, CRL incorporates additional constraints that represent specific
mission requirements or limitations that the agent must comply with during the
learning process. In this paper, we address a type of CRL problem where an
agent aims to learn the optimal policy to maximize reward while ensuring a
desired level of temporal logic constraint satisfaction throughout the learning
process. We propose a novel framework that relies on switching between pure
learning (reward maximization) and constraint satisfaction. This framework
estimates the probability of constraint satisfaction based on earlier trials
and properly adjusts the probability of switching between learning and
constraint satisfaction policies. We theoretically validate the correctness of
the proposed algorithm and demonstrate its performance and scalability through
comprehensive simulations.
☆ Towards Synergistic, Generalized, and Efficient Dual-System for Robotic Manipulation
The increasing demand for versatile robotic systems to operate in diverse and
dynamic environments has emphasized the importance of a generalist policy,
which leverages a large cross-embodiment data corpus to facilitate broad
adaptability and high-level reasoning. However, the generalist would struggle
with inefficient inference and cost-expensive training. The specialist policy,
instead, is curated for specific domain data and excels at task-level precision
with efficiency. Yet, it lacks the generalization capacity for a wide range of
applications. Inspired by these observations, we introduce RoboDual, a
synergistic dual-system that supplements the merits of both generalist and
specialist policy. A diffusion transformer-based specialist is devised for
multi-step action rollouts, exquisitely conditioned on the high-level task
understanding and discretized action output of a vision-language-action (VLA)
based generalist. Compared to OpenVLA, RoboDual achieves 26.7% improvement in
real-world setting and 12% gain on CALVIN by introducing a specialist policy
with merely 20M trainable parameters. It maintains strong performance with 5%
of demonstration data only, and enables a 3.8 times higher control frequency in
real-world deployment. Code would be made publicly available. Our project page
is hosted at: https://opendrivelab.com/RoboDual/
comment: Project page: https://opendrivelab.com/RoboDual/
☆ Fron CAD to URDF: Co-Design of a Jet-Powered Humanoid Robot Including CAD Geometry IROS 2024
Punith Reddy Vanteddu, Gabriele Nava, Fabio Bergonti, Giuseppe L'Erario, Antonello Paolino, Daniele PUcci
Co-design optimization strategies usually rely on simplified robot models
extracted from CAD. While these models are useful for optimizing geometrical
and inertial parameters for robot control, they might overlook important
details essential for prototyping the optimized mechanical design. For
instance, they may not account for mechanical stresses exerted on the optimized
geometries and the complexity of assembly-level design. In this paper, we
introduce a co-design framework aimed at improving both the control performance
and mechanical design of our robot. Specifically, we identify the robot links
that significantly influence control performance. The geometric characteristics
of these links are parameterized and optimized using a multi-objective
evolutionary algorithm to achieve optimal control performance. Additionally, an
automated Finite Element Method (FEM) analysis is integrated into the framework
to filter solutions not satisfying the required structural safety margin. We
validate the framework by applying it to enhance the mechanical design for
flight performance of the jet-powered humanoid robot iRonCub.
comment: IROS 2024
☆ Multimodal Perception System for Real Open Environment
This paper presents a novel multimodal perception system for a real open
environment. The proposed system includes an embedded computation platform,
cameras, ultrasonic sensors, GPS, and IMU devices. Unlike the traditional
frameworks, our system integrates multiple sensors with advanced computer
vision algorithms to help users walk outside reliably. The system can
efficiently complete various tasks, including navigating to specific locations,
passing through obstacle regions, and crossing intersections. Specifically, we
also use ultrasonic sensors and depth cameras to enhance obstacle avoidance
performance. The path planning module is designed to find the locally optimal
route based on various feedback and the user's current state. To evaluate the
performance of the proposed system, we design several experiments under
different scenarios. The results show that the system can help users walk
efficiently and independently in complex situations.
☆ Understanding Human Activity with Uncertainty Measure for Novelty in Graph Convolutional Networks
Understanding human activity is a crucial aspect of developing intelligent
robots, particularly in the domain of human-robot collaboration. Nevertheless,
existing systems encounter challenges such as over-segmentation, attributed to
errors in the up-sampling process of the decoder. In response, we introduce a
promising solution: the Temporal Fusion Graph Convolutional Network. This
innovative approach aims to rectify the inadequate boundary estimation of
individual actions within an activity stream and mitigate the issue of
over-segmentation in the temporal dimension.
Moreover, systems leveraging human activity recognition frameworks for
decision-making necessitate more than just the identification of actions. They
require a confidence value indicative of the certainty regarding the
correspondence between observations and training examples. This is crucial to
prevent overly confident responses to unforeseen scenarios that were not part
of the training data and may have resulted in mismatches due to weak similarity
measures within the system. To address this, we propose the incorporation of a
Spectral Normalized Residual connection aimed at enhancing efficient estimation
of novelty in observations. This innovative approach ensures the preservation
of input distance within the feature space by imposing constraints on the
maximum gradients of weight updates. By limiting these gradients, we promote a
more robust handling of novel situations, thereby mitigating the risks
associated with overconfidence. Our methodology involves the use of a Gaussian
process to quantify the distance in feature space.
comment: 15 pages, 10 figures, The International Journal of Robotics Research
☆ Understanding Spatio-Temporal Relations in Human-Object Interaction using Pyramid Graph Convolutional Network IROS 2022
Human activities recognition is an important task for an intelligent robot,
especially in the field of human-robot collaboration, it requires not only the
label of sub-activities but also the temporal structure of the activity. In
order to automatically recognize both the label and the temporal structure in
sequence of human-object interaction, we propose a novel Pyramid Graph
Convolutional Network (PGCN), which employs a pyramidal encoder-decoder
architecture consisting of an attention based graph convolution network and a
temporal pyramid pooling module for downsampling and upsampling interaction
sequence on the temporal axis, respectively. The system represents the 2D or 3D
spatial relation of human and objects from the detection results in video data
as a graph. To learn the human-object relations, a new attention graph
convolutional network is trained to extract condensed information from the
graph representation. To segment action into sub-actions, a novel temporal
pyramid pooling module is proposed, which upsamples compressed features back to
the original time scale and classifies actions per frame.
We explore various attention layers, namely spatial attention, temporal
attention and channel attention, and combine different upsampling decoders to
test the performance on action recognition and segmentation. We evaluate our
model on two challenging datasets in the field of human-object interaction
recognition, i.e. Bimanual Actions and IKEA Assembly datasets. We demonstrate
that our classifier significantly improves both framewise action recognition
and segmentation, e.g., F1 micro and F1@50 scores on Bimanual Actions dataset
are improved by $4.3\%$ and $8.5\%$ respectively.
comment: 7 pages, 6 figures, IROS 2022 conference
☆ Soothing Sensations: Enhancing Interactions with a Socially Assistive Robot through Vibrotactile Heartbeats
Physical interactions with socially assistive robots (SARs) positively affect
user wellbeing. However, haptic experiences when touching a SAR are typically
limited to perceiving the robot's movements or shell texture, while other
modalities that could enhance the touch experience with the robot, such as
vibrotactile stimulation, are under-explored. In this exploratory qualitative
study, we investigate the potential of enhancing human interaction with the
PARO robot through vibrotactile heartbeats, with the goal to regulate
subjective wellbeing during stressful situations. We conducted in-depth
one-on-one interviews with 30 participants, who watched three horror movie
clips alone, with PARO, and with a PARO that displayed a vibrotactile
heartbeat. Our findings show that PARO's presence and its interactive
capabilities can help users regulate emotions through attentional redeployment
from a stressor toward the robot. The vibrotactile heartbeat further reinforced
PARO's physical and social presence, enhancing the socio-emotional support
provided by the robot and its perceived life-likeness. We discuss the impact of
individual differences in user experience and implications for the future
design of life-like vibrotactile stimulation for SARs.
comment: 2024 33rd IEEE International Conference on Robot and Human
Interactive Communication (RO-MAN)
☆ Constrained Skill Discovery: Quadruped Locomotion with Unsupervised Reinforcement Learning
Representation learning and unsupervised skill discovery can allow robots to
acquire diverse and reusable behaviors without the need for task-specific
rewards. In this work, we use unsupervised reinforcement learning to learn a
latent representation by maximizing the mutual information between skills and
states subject to a distance constraint. Our method improves upon prior
constrained skill discovery methods by replacing the latent transition
maximization with a norm-matching objective. This not only results in a much a
richer state space coverage compared to baseline methods, but allows the robot
to learn more stable and easily controllable locomotive behaviors. We
successfully deploy the learned policy on a real ANYmal quadruped robot and
demonstrate that the robot can accurately reach arbitrary points of the
Cartesian state space in a zero-shot manner, using only an intrinsic skill
discovery and standard regularization rewards.
☆ L-VITeX: Light-weight Visual Intuition for Terrain Exploration
This paper presents L-VITeX, a lightweight visual intuition system for
terrain exploration designed for resource-constrained robots and swarms.
L-VITeX aims to provide a hint of Regions of Interest (RoIs) without
computationally expensive processing. By utilizing the Faster Objects, More
Objects (FOMO) tinyML architecture, the system achieves high accuracy (>99%) in
RoI detection while operating on minimal hardware resources (Peak RAM usage <
50 KB) with near real-time inference (<200 ms). The paper evaluates L-VITeX's
performance across various terrains, including mountainous areas, underwater
shipwreck debris regions, and Martian rocky surfaces. Additionally, it
demonstrates the system's application in 3D mapping using a small mobile robot
run by ESP32-Cam and Gaussian Splats (GS), showcasing its potential to enhance
exploration efficiency and decision-making.
☆ Synergizing Morphological Computation and Generative Design: Automatic Synthesis of Tendon-Driven Grippers
Robots' behavior and performance are determined both by hardware and
software. The design process of robotic systems is a complex journey that
involves multiple phases. Throughout this process, the aim is to tackle various
criteria simultaneously, even though they often contradict each other. The
ultimate goal is to uncover the optimal solution that resolves these
conflicting factors. Generative, computation or automatic designs are the
paradigms aimed at accelerating the whole design process. Within this paper we
propose a design methodology to generate linkage mechanisms for robots with
morphological computation. We use a graph grammar and a heuristic search
algorithm to create robot mechanism graphs that are converted into simulation
models for testing the design output. To verify the design methodology we have
applied it to a relatively simple quasi-static problem of object grasping. We
found a way to automatically design an underactuated tendon-driven gripper that
can grasp a wide range of objects. This is possible because of its structure,
not because of sophisticated planning or learning.
☆ RDT-1B: a Diffusion Foundation Model for Bimanual Manipulation
Songming Liu, Lingxuan Wu, Bangguo Li, Hengkai Tan, Huayu Chen, Zhengyi Wang, Ke Xu, Hang Su, Jun Zhu
Bimanual manipulation is essential in robotics, yet developing foundation
models is extremely challenging due to the inherent complexity of coordinating
two robot arms (leading to multi-modal action distributions) and the scarcity
of training data. In this paper, we present the Robotics Diffusion Transformer
(RDT), a pioneering diffusion foundation model for bimanual manipulation. RDT
builds on diffusion models to effectively represent multi-modality, with
innovative designs of a scalable Transformer to deal with the heterogeneity of
multi-modal inputs and to capture the nonlinearity and high frequency of
robotic data. To address data scarcity, we further introduce a Physically
Interpretable Unified Action Space, which can unify the action representations
of various robots while preserving the physical meanings of original actions,
facilitating learning transferrable physical knowledge. With these designs, we
managed to pre-train RDT on the largest collection of multi-robot datasets to
date and scaled it up to 1.2B parameters, which is the largest diffusion-based
foundation model for robotic manipulation. We finally fine-tuned RDT on a
self-created multi-task bimanual dataset with over 6K+ episodes to refine its
manipulation capabilities. Experiments on real robots demonstrate that RDT
significantly outperforms existing methods. It exhibits zero-shot
generalization to unseen objects and scenes, understands and follows language
instructions, learns new skills with just 1~5 demonstrations, and effectively
handles complex, dexterous tasks. We refer to
https://rdt-robotics.github.io/rdt-robotics/ for the code and videos.
comment: 10 pages, conference
☆ Online DNN-driven Nonlinear MPC for Stylistic Humanoid Robot Walking with Step Adjustment
Giulio Romualdi, Paolo Maria Viceconte, Lorenzo Moretti, Ines Sorrentino, Stefano Dafarra, Silvio Traversaro, Daniele Pucci
This paper presents a three-layered architecture that enables stylistic
locomotion with online contact location adjustment. Our method combines an
autoregressive Deep Neural Network (DNN) acting as a trajectory generation
layer with a model-based trajectory adjustment and trajectory control layers.
The DNN produces centroidal and postural references serving as an initial guess
and regularizer for the other layers. Being the DNN trained on human motion
capture data, the resulting robot motion exhibits locomotion patterns,
resembling a human walking style. The trajectory adjustment layer utilizes
non-linear optimization to ensure dynamically feasible center of mass (CoM)
motion while addressing step adjustments. We compare two implementations of the
trajectory adjustment layer: one as a receding horizon planner (RHP) and the
other as a model predictive controller (MPC). To enhance MPC performance, we
introduce a Kalman filter to reduce measurement noise. The filter parameters
are automatically tuned with a Genetic Algorithm. Experimental results on the
ergoCub humanoid robot demonstrate the system's ability to prevent falls,
replicate human walking styles, and withstand disturbances up to 68 Newton.
Website: https://sites.google.com/view/dnn-mpc-walking
Youtube video: https://www.youtube.com/watch?v=x3tzEfxO-xQ
comment: This paper has been accepted for publication at the 2024 IEEE-RAS
International Conference on Humanoid Robots,(Humanoids) Nancy, France, 2024
☆ SwarmPath: Drone Swarm Navigation through Cluttered Environments Leveraging Artificial Potential Field and Impedance Control
In the area of multi-drone systems, navigating through dynamic environments
from start to goal while providing collision-free trajectory and efficient path
planning is a significant challenge. To solve this problem, we propose a novel
SwarmPath technology that involves the integration of Artificial Potential
Field (APF) with Impedance Controller. The proposed approach provides a
solution based on collision free leader-follower behaviour where drones are
able to adapt themselves to the environment. Moreover, the leader is virtual
while drones are physical followers leveraging APF path planning approach to
find the smallest possible path to the target. Simultaneously, the drones
dynamically adjust impedance links, allowing themselves to create virtual links
with obstacles to avoid them. As compared to conventional APF, the proposed
SwarmPath system not only provides smooth collision-avoidance but also enable
agents to efficiently pass through narrow passages by reducing the total travel
time by 30% while ensuring safety in terms of drones connectivity. Lastly, the
results also illustrate that the discrepancies between simulated and real
environment, exhibit an average absolute percentage error (APE) of 6% of drone
trajectories. This underscores the reliability of our solution in real-world
scenarios.
comment: Manuscript accepted in IEEE International Conference on Robotics and
Biomimetics (IEEE ROBIO 2024)
☆ Autonomous Vehicles Path Planning under Temporal Logic Specifications
Path planning is an essential component of autonomous driving. A global
planner is responsible for the high-level planning. It basically performs a
shortest-path search on a known map, thereby defining waypoints used to control
the local (low-level) planner. Local planning is a runtime verification method
which is repeatedly run on the vehicle itself in real-time, so as to find the
optimal short-horizon path which leads to the desired waypoint in a way which
is both efficient and safe. The challenge is that the local planner has to take
into account repeatedly incoming updates about the information available of the
environment. In addition, it performs a complex task, as it has to take into
account a large variety of requirements, originating from the necessity of
collision avoidance with obstacles, respecting traffic rules, sticking to
regulatory requirements, and lastly to reach the next waypoint efficiently. In
this paper, we describe a logic-based specification mechanism which fulfills
all these requirements.
comment: 10 pages, 5 Figures, 1 Table, Accepted as a short paper at 27th
Brazilian Symposium on Formal Methods (SBMF 2024)
☆ LaB-CL: Localized and Balanced Contrastive Learning for improving parking slot detection
Parking slot detection is an essential technology in autonomous parking
systems. In general, the classification problem of parking slot detection
consists of two tasks, a task determining whether localized candidates are
junctions of parking slots or not, and the other that identifies a shape of
detected junctions. Both classification tasks can easily face biased learning
toward the majority class, degrading classification performances. Yet, the data
imbalance issue has been overlooked in parking slot detection. We propose the
first supervised contrastive learning framework for parking slot detection,
Localized and Balanced Contrastive Learning for improving parking slot
detection (LaB-CL). The proposed LaB-CL framework uses two main approaches.
First, we propose to include class prototypes to consider representations from
all classes in every mini batch, from the local perspective. Second, we propose
a new hard negative sampling scheme that selects local representations with
high prediction error. Experiments with the benchmark dataset demonstrate that
the proposed LaB-CL framework can outperform existing parking slot detection
methods.
comment: 7 pages, 6 figures
☆ Robotic framework for autonomous manipulation of laboratory equipment with different degrees of transparency via 6D pose estimation
Many modern robotic systems operate autonomously, however they often lack the
ability to accurately analyze the environment and adapt to changing external
conditions, while teleoperation systems often require special operator skills.
In the field of laboratory automation, the number of automated processes is
growing, however such systems are usually developed to perform specific tasks.
In addition, many of the objects used in this field are transparent, making it
difficult to analyze them using visual channels. The contributions of this work
include the development of a robotic framework with autonomous mode for
manipulating liquid-filled objects with different degrees of transparency in
complex pose combinations. The conducted experiments demonstrated the
robustness of the designed visual perception system to accurately estimate
object poses for autonomous manipulation, and confirmed the performance of the
algorithms in dexterous operations such as liquid dispensing. The proposed
robotic framework can be applied for laboratory automation, since it allows
solving the problem of performing non-trivial manipulation tasks with the
analysis of object poses of varying degrees of transparency and liquid levels,
requiring high accuracy and repeatability.
comment: Accepted to the 2024 IEEE International Conference on Robotics and
Biomimetics (IEEE ROBIO 2024), 8 pages, 11 figures
☆ Mastering Contact-rich Tasks by Combining Soft and Rigid Robotics with Imitation Learning
Mariano Ramírez Montero, Ebrahim Shahabi, Giovanni Franzese, Jens Kober, Barbara Mazzolai, Cosimo Della Santina
Soft robots have the potential to revolutionize the use of robotic systems
with their capability of establishing safe, robust, and adaptable interactions
with their environment, but their precise control remains challenging. In
contrast, traditional rigid robots offer high accuracy and repeatability but
lack the flexibility of soft robots. We argue that combining these
characteristics in a hybrid robotic platform can significantly enhance overall
capabilities. This work presents a novel hybrid robotic platform that
integrates a rigid manipulator with a fully developed soft arm. This system is
equipped with the intelligence necessary to perform flexible and generalizable
tasks through imitation learning autonomously. The physical softness and
machine learning enable our platform to achieve highly generalizable skills,
while the rigid components ensure precision and repeatability.
☆ Neural Semantic Map-Learning for Autonomous Vehicles IROS 2024
Autonomous vehicles demand detailed maps to maneuver reliably through
traffic, which need to be kept up-to-date to ensure a safe operation. A
promising way to adapt the maps to the ever-changing road-network is to use
crowd-sourced data from a fleet of vehicles. In this work, we present a mapping
system that fuses local submaps gathered from a fleet of vehicles at a central
instance to produce a coherent map of the road environment including drivable
area, lane markings, poles, obstacles and more as a 3D mesh. Each vehicle
contributes locally reconstructed submaps as lightweight meshes, making our
method applicable to a wide range of reconstruction methods and sensor
modalities. Our method jointly aligns and merges the noisy and incomplete local
submaps using a scene-specific Neural Signed Distance Field, which is
supervised using the submap meshes to predict a fused environment
representation. We leverage memory-efficient sparse feature-grids to scale to
large areas and introduce a confidence score to model uncertainty in scene
reconstruction. Our approach is evaluated on two datasets with different local
mapping methods, showing improved pose alignment and reconstruction over
existing methods. Additionally, we demonstrate the benefit of multi-session
mapping and examine the required amount of data to enable high-fidelity map
learning for autonomous vehicles.
comment: Accepted at 2024 IEEE/RSJ International Conference on Intelligent
Robots and Systems (IROS 2024)
☆ Learning Low-Level Causal Relations using a Simulated Robotic Arm ICANN
Causal learning allows humans to predict the effect of their actions on the
known environment and use this knowledge to plan the execution of more complex
actions. Such knowledge also captures the behaviour of the environment and can
be used for its analysis and the reasoning behind the behaviour. This type of
knowledge is also crucial in the design of intelligent robotic systems with
common sense. In this paper, we study causal relations by learning the forward
and inverse models based on data generated by a simulated robotic arm involved
in two sensorimotor tasks. As a next step, we investigate feature attribution
methods for the analysis of the forward model, which reveals the low-level
causal effects corresponding to individual features of the state vector related
to both the arm joints and the environment features. This type of analysis
provides solid ground for dimensionality reduction of the state
representations, as well as for the aggregation of knowledge towards the
explainability of causal effects at higher levels.
comment: 14 pages, 5 figures, 3 tables. Appeared in 2024 International
Conference on Artificial Neural Networks (ICANN) proceedings. Published
version copyrighted by Springer. This work was funded by the Horizon Europe
Twinning project TERAIS, G.A. number 101079338 and in part by the Slovak
Grant Agency for Science (VEGA), project 1/0373/23
☆ PHODCOS: Pythagorean Hodograph-based Differentiable Coordinate System
This paper presents PHODCOS, an algorithm that assigns a moving coordinate
system to a given curve. The parametric functions underlying the coordinate
system, i.e., the path function, the moving frame and its angular velocity, are
exact -- approximation free -- differentiable, and sufficiently continuous.
This allows for computing a coordinate system for highly nonlinear curves,
while remaining compliant with autonomous navigation algorithms that require
first and second order gradient information. In addition, the coordinate system
obtained by PHODCOS is fully defined by a finite number of coefficients, which
may then be used to compute additional geometric properties of the curve, such
as arc-length, curvature, torsion, etc. Therefore, PHODCOS presents an
appealing paradigm to enhance the geometrical awareness of existing guidance
and navigation on-orbit spacecraft maneuvers. The PHODCOS algorithm is
presented alongside an analysis of its error and approximation order, and thus,
it is guaranteed that the obtained coordinate system matches the given curve
within a desired tolerance. To demonstrate the applicability of the coordinate
system resulting from PHODCOS, we present numerical examples in the Near
Rectilinear Halo Orbit (NRHO) for the Lunar Gateway.
comment: Code: https://github.com/jonarriza96/phodcos
☆ Design Method of a Kangaroo Robot with High Power Legs and an Articulated Soft Tail IROS2023
Shunnosuke Yoshimura, Temma Suzuki, Masahiro Bando, Sota Yuzaki, Kento Kawaharazuka, Kei Okada, Masayuki Inaba
In this paper, we focus on the kangaroo, which has powerful legs capable of
jumping and a soft and strong tail. To incorporate these unique structure into
a robot for utilization, we propose a design method that takes into account
both the feasibility as a robot and the kangaroo-mimetic structure. Based on
the kangaroo's musculoskeletal structure, we determine the structure of the
robot that enables it to jump by analyzing the muscle arrangement and prior
verification in simulation. Also, to realize a tail capable of body support, we
use an articulated, elastic structure as a tail. In order to achieve both
softness and high power output, the robot is driven by a direct-drive,
high-power wire-winding mechanism, and weight of legs and the tail is reduced
by placing motors in the torso. The developed kangaroo robot can jump with its
hind legs, moving its tail, and supporting its body using its hind legs and
tail.
comment: accepted at IROS2023
☆ Lean Methodology for Garment Modernization
Lean Methodology for Garment Modernization. This article presents the lean
methodology for modernizing garment manufacturing, focusing on lean thinking,
lean practices, automation development, VSM, and CRP, and how to integrate them
effectively. While isolated automation of specific operations can improve
efficiency and reduce cycle time, it does not necessarily enhance overall
garment output and efficiency. To achieve these broader improvements, it is
essential to consider the entire production line and process using VSM and CRP
to optimize production and center balance. This approach can increase
efficiency, and reduce manufacturing costs, labor time, and lead time,
ultimately adding value to the company and factory.
comment: 11 pages,7 Figures
☆ Autonomous Driving in Unstructured Environments: How Far Have We Come?
Chen Min, Shubin Si, Xu Wang, Hanzhang Xue, Weizhong Jiang, Yang Liu, Juan Wang, Qingtian Zhu, Qi Zhu, Lun Luo, Fanjie Kong, Jinyu Miao, Xudong Cai, Shuai An, Wei Li, Jilin Mei, Tong Sun, Heng Zhai, Qifeng Liu, Fangzhou Zhao, Liang Chen, Shuai Wang, Erke Shang, Linzhi Shang, Kunlong Zhao, Fuyang Li, Hao Fu, Lei Jin, Jian Zhao, Fangyuan Mao, Zhipeng Xiao, Chengyang Li, Bin Dai, Dawei Zhao, Liang Xiao, Yiming Nie, Yu Hu
Research on autonomous driving in unstructured outdoor environments is less
advanced than in structured urban settings due to challenges like environmental
diversities and scene complexity. These environments-such as rural areas and
rugged terrains-pose unique obstacles that are not common in structured urban
areas. Despite these difficulties, autonomous driving in unstructured outdoor
environments is crucial for applications in agriculture, mining, and military
operations. Our survey reviews over 250 papers for autonomous driving in
unstructured outdoor environments, covering offline mapping, pose estimation,
environmental perception, path planning, end-to-end autonomous driving,
datasets, and relevant challenges. We also discuss emerging trends and future
research directions. This review aims to consolidate knowledge and encourage
further research for autonomous driving in unstructured environments. To
support ongoing work, we maintain an active repository with up-to-date
literature and open-source projects at:
https://github.com/chaytonmin/Survey-Autonomous-Driving-in-Unstructured-Environments.
comment: Survey paper; 38 pages
☆ A Visual Cooperative Localization Method for Airborne Magnetic Surveying Based on a Manifold Sensor Fusion Algorithm Using Lie Groups
Recent advancements in UAV technology have spurred interest in developing
multi-UAV aerial surveying systems for use in confined environments where GNSS
signals are blocked or jammed. This paper focuses airborne magnetic surveying
scenarios. To obtain clean magnetic measurements reflecting the Earth's
magnetic field, the magnetic sensor must be isolated from other electronic
devices, creating a significant localization challenge. We propose a visual
cooperative localization solution. The solution incorporates a visual
processing module and an improved manifold-based sensor fusion algorithm,
delivering reliable and accurate positioning information. Real flight
experiments validate the approach, demonstrating single-axis centimeter-level
accuracy and decimeter-level overall 3D positioning accuracy.
comment: 12 pages
☆ Toward a Better Understanding of Robot Energy Consumption in Agroecological Applications
In this paper, we present a comprehensive analysis and discussion of energy
consumption in agricultural robots. Robots are emerging as a promising solution
to address food production and agroecological challenges, offering potential
reductions in chemical use and the ability to perform strenuous tasks beyond
human capabilities. The automation of agricultural tasks introduces a
previously unattainable level of complexity, enabling robots to optimize
trajectories, control laws, and overall task planning. Consequently, automation
can lead to higher levels of energy optimization in agricultural tasks.
However, the energy consumption of robotic platforms is not fully understood,
and a deeper analysis of contributing factors is essential to optimize energy
use. We analyze the energy data of an automated agricultural tractor performing
tasks throughout the year, revealing nontrivial correlations between the
robot's velocity, the type of task performed, and energy consumption. This
suggests a tradeoff between task efficiency, time to completion, and energy
expenditure that can be harnessed to improve the energy efficiency of robotic
agricultural operations.
comment: 6 pages, 6 figures
☆ PokeFlex: A Real-World Dataset of Deformable Objects for Robotics
Jan Obrist, Miguel Zamora, Hehui Zheng, Ronan Hinchet, Firat Ozdemir, Juan Zarate, Robert K. Katzschmann, Stelian Coros
Data-driven methods have shown great potential in solving challenging
manipulation tasks, however, their application in the domain of deformable
objects has been constrained, in part, by the lack of data. To address this, we
propose PokeFlex, a dataset featuring real-world paired and annotated
multimodal data that includes 3D textured meshes, point clouds, RGB images, and
depth maps. Such data can be leveraged for several downstream tasks such as
online 3D mesh reconstruction, and it can potentially enable underexplored
applications such as the real-world deployment of traditional control methods
based on mesh simulations. To deal with the challenges posed by real-world 3D
mesh reconstruction, we leverage a professional volumetric capture system that
allows complete 360{\deg} reconstruction. PokeFlex consists of 18 deformable
objects with varying stiffness and shapes. Deformations are generated by
dropping objects onto a flat surface or by poking the objects with a robot arm.
Interaction forces and torques are also reported for the latter case. Using
different data modalities, we demonstrated a use case for the PokeFlex dataset
in online 3D mesh reconstruction. We refer the reader to our website (
https://pokeflex-dataset.github.io/ ) for demos and examples of our dataset.
☆ The Power of Input: Benchmarking Zero-Shot Sim-To-Real Transfer of Reinforcement Learning Control Policies for Quadrotor Control
In the last decade, data-driven approaches have become popular choices for
quadrotor control, thanks to their ability to facilitate the adaptation to
unknown or uncertain flight conditions. Among the different data-driven
paradigms, Deep Reinforcement Learning (DRL) is currently one of the most
explored. However, the design of DRL agents for Micro Aerial Vehicles (MAVs)
remains an open challenge. While some works have studied the output
configuration of these agents (i.e., what kind of control to compute), there is
no general consensus on the type of input data these approaches should employ.
Multiple works simply provide the DRL agent with full state information,
without questioning if this might be redundant and unnecessarily complicate the
learning process, or pose superfluous constraints on the availability of such
information in real platforms. In this work, we provide an in-depth benchmark
analysis of different configurations of the observation space. We optimize
multiple DRL agents in simulated environments with different input choices and
study their robustness and their sim-to-real transfer capabilities with
zero-shot adaptation. We believe that the outcomes and discussions presented in
this work supported by extensive experimental results could be an important
milestone in guiding future research on the development of DRL agents for
aerial robot tasks.
☆ Patterned Structure Muscle : Arbitrary Shaped Wire-driven Artificial Muscle Utilizing Anisotropic Flexible Structure for Musculoskeletal Robots IROS2024
Shunnosuke Yoshimura, Akihiro Miki, Kazuhiro Miyama, Yuta Sahara, Kento Kawaharazuka, Kei Okada, Masayuki Inaba
Muscles of the human body are composed of tiny actuators made up of myosin
and actin filaments. They can exert force in various shapes such as curved or
flat, under contact forces and deformations from the environment. On the other
hand, muscles in musculoskeletal robots so far have faced challenges in
generating force in such shapes and environments. To address this issue, we
propose Patterned Structure Muscle (PSM), artificial muscles for
musculoskeletal robots. PSM utilizes patterned structures with anisotropic
characteristics, wire-driven mechanisms, and is made of flexible material
Thermoplastic Polyurethane (TPU) using FDM 3D printing. This method enables the
creation of various shapes of muscles, such as simple 1 degree-of-freedom (DOF)
muscles, Multi-DOF wide area muscles, joint-covering muscles, and branched
muscles. We created an upper arm structure using these muscles to demonstrate
wide range of motion, lifting heavy objects, and movements through
environmental contact. These experiments show that the proposed PSM is capable
of operating in various shapes and environments, and is suitable for the
muscles of musculoskeletal robots.
comment: accepted at IROS2024
☆ Simplified POMDP Planning with an Alternative Observation Space and Formal Performance Guarantees
Online planning under uncertainty in partially observable domains is an
essential capability in robotics and AI. The partially observable Markov
decision process (POMDP) is a mathematically principled framework for
addressing decision-making problems in this challenging setting. However,
finding an optimal solution for POMDPs is computationally expensive and is
feasible only for small problems. In this work, we contribute a novel method to
simplify POMDPs by switching to an alternative, more compact, observation space
and simplified model to speedup planning with formal performance guarantees. We
introduce the notion of belief tree topology, which encodes the levels and
branches in the tree that use the original and alternative observation space
and models. Each belief tree topology comes with its own policy space and
planning performance. Our key contribution is to derive bounds between the
optimal Q-function of the original POMDP and the simplified tree defined by a
given topology with a corresponding simplified policy space. These bounds are
then used as an adaptation mechanism between different tree topologies until
the optimal action of the original POMDP can be determined. Further, we
consider a specific instantiation of our framework, where the alternative
observation space and model correspond to a setting where the state is fully
observable. We evaluate our approach in simulation, considering exact and
approximate POMDP solvers and demonstrating a significant speedup while
preserving solution quality. We believe this work opens new exciting avenues
for online POMDP planning with formal performance guarantees.
comment: Accepted to ISRR 2024
☆ Stop-N-Go: Search-based Conflict Resolution for Motion Planning of Multiple Robotic Manipulators
We address the motion planning problem for multiple robotic manipulators in
packed environments where shared workspace can result in goal positions
occupied or blocked by other robots unless those other robots move away to make
the goal positions free. While planning in a coupled configuration space
(C-space) is straightforward, it struggles to scale with the number of robots
and often fails to find solutions. Decoupled planning is faster but frequently
leads to conflicts between trajectories.
We propose a conflict resolution approach that inserts pauses into
individually planned trajectories using an A* search strategy to minimize the
makespan--the total time until all robots complete their tasks. This method
allows some robots to stop, enabling others to move without collisions, and
maintains short distances in the C-space. It also effectively handles cases
where goal positions are initially blocked by other robots. Experimental
results show that our method successfully solves challenging instances where
baseline methods fail to find feasible solutions.
☆ Imitation Learning with Limited Actions via Diffusion Planners and Deep Koopman Controllers
Recent advances in diffusion-based robot policies have demonstrated
significant potential in imitating multi-modal behaviors. However, these
approaches typically require large quantities of demonstration data paired with
corresponding robot action labels, creating a substantial data collection
burden. In this work, we propose a plan-then-control framework aimed at
improving the action-data efficiency of inverse dynamics controllers by
leveraging observational demonstration data. Specifically, we adopt a Deep
Koopman Operator framework to model the dynamical system and utilize
observation-only trajectories to learn a latent action representation. This
latent representation can then be effectively mapped to real high-dimensional
continuous actions using a linear action decoder, requiring minimal
action-labeled data. Through experiments on simulated robot manipulation tasks
and a real robot experiment with multi-modal expert demonstrations, we
demonstrate that our approach significantly enhances action-data efficiency and
achieves high task success rates with limited action data.
☆ Self-Supervised Meta-Learning for All-Layer DNN-Based Adaptive Control with Stability Guarantees
A critical goal of adaptive control is enabling robots to rapidly adapt in
dynamic environments. Recent studies have developed a meta-learning-based
adaptive control scheme, which uses meta-learning to extract nonlinear features
(represented by Deep Neural Networks (DNNs)) from offline data, and uses
adaptive control to update linear coefficients online. However, such a scheme
is fundamentally limited by the linear parameterization of uncertainties and
does not fully unleash the capability of DNNs. This paper introduces a novel
learning-based adaptive control framework that pretrains a DNN via
self-supervised meta-learning (SSML) from offline trajectories and online
adapts the full DNN via composite adaptation. In particular, the offline SSML
stage leverages the time consistency in trajectory data to train the DNN to
predict future disturbances from history, in a self-supervised manner without
environment condition labels. The online stage carefully designs a control law
and an adaptation law to update the full DNN with stability guarantees.
Empirically, the proposed framework significantly outperforms (19-39%) various
classic and learning-based adaptive control baselines, in challenging
real-world quadrotor tracking problems under large dynamic wind disturbance.
☆ Streamlined shape of cyborg cockroach promotes traversability in confined environments by gap negotiation
The centimeter-scale cyborg insects have a potential advantage for
application in narrow environments where humans cannot operate. To realize such
tasks, researchers have developed a small printed-circuit-board (PCB) which an
insect can carry and control it. The electronic components usually remain bare
on the board and the whole board is mounted on platform animals, resulting in
uneven morphology of whole cyborg with sharp edges. It is well known that
streamlined body shape in artificial vehicles or robots contributes to
effective locomotion by reducing drag force in media. However, little is known
how the entire body shape impacts on locomotor performance of cyborg insect.
Here, we developed a 10 mm by 10 mm board which provided electrical stimulation
via Sub-GHz communication and investigated the impact of physical arrangement
of the board using Madagascar hissing cockroach. We compared the success rate
of gap negotiation between the cyborg with mounted board and implanted board
and found the latter outperformed the former. We demonstrated our cyborg
cockroach with implanted board could follow faithfully to the locomotion
command via antennal or cercal stimulation and traverse a narrow gap like air
vent cover. In contrast to the conventional arrangement, our cyborg insects are
suitable for application in a concealed environment.
☆ Force-Centric Imitation Learning with Force-Motion Capture System for Contact-Rich Manipulation ICRA 2025
In most contact-rich manipulation tasks, humans apply time-varying forces to
the target object, compensating for inaccuracies in the vision-guided hand
trajectory. However, current robot learning algorithms primarily focus on
trajectory-based policy, with limited attention given to learning force-related
skills. To address this limitation, we introduce ForceMimic, a force-centric
robot learning system, providing a natural, force-aware and robot-free robotic
demonstration collection system, along with a hybrid force-motion imitation
learning algorithm for robust contact-rich manipulation. Using the proposed
ForceCapture system, an operator can peel a zucchini in 5 minutes, while
force-feedback teleoperation takes over 13 minutes and struggles with task
completion. With the collected data, we propose HybridIL to train a
force-centric imitation learning model, equipped with hybrid force-position
control primitive to fit the predicted wrench-position parameters during robot
execution. Experiments demonstrate that our approach enables the model to learn
a more robust policy under the contact-rich task of vegetable peeling,
increasing the success rates by 54.5% relatively compared to state-of-the-art
pure-vision-based imitation learning. Hardware, code, data and more results
would be open-sourced on the project website at https://forcemimic.github.io.
comment: 8 pages, 7 figures, submitted to ICRA 2025, project website at
https://forcemimic.github.io
☆ G$^{2}$TR: Generalized Grounded Temporal Reasoning for Robot Instruction Following by Combining Large Pre-trained Models
Riya Arora, Niveditha Narendranath, Aman Tambi, Sandeep S. Zachariah, Souvik Chakraborty, Rohan Paul
Consider the scenario where a human cleans a table and a robot observing the
scene is instructed with the task "Remove the cloth using which I wiped the
table". Instruction following with temporal reasoning requires the robot to
identify the relevant past object interaction, ground the object of interest in
the present scene, and execute the task according to the human's instruction.
Directly grounding utterances referencing past interactions to grounded objects
is challenging due to the multi-hop nature of references to past interactions
and large space of object groundings in a video stream observing the robot's
workspace. Our key insight is to factor the temporal reasoning task as (i)
estimating the video interval associated with event reference, (ii) performing
spatial reasoning over the interaction frames to infer the intended object
(iii) semantically track the object's location till the current scene to enable
future robot interactions. Our approach leverages existing large pre-trained
models (which possess inherent generalization capabilities) and combines them
appropriately for temporal grounding tasks. Evaluation on a video-language
corpus acquired with a robot manipulator displaying rich temporal interactions
in spatially-complex scenes displays an average accuracy of 70.10%. The
dataset, code, and videos are available at
https://reail-iitdelhi.github.io/temporalreasoning.github.io/ .
☆ Autonomous Robotic System with Optical Coherence Tomography Guidance for Vascular Anastomosis
Jesse Haworth, Rishi Biswas, Justin Opfermann, Michael Kam, Yaning Wang, Desire Pantalone, Francis X. Creighton, Robin Yang, Jin U. Kang, Axel Krieger
Vascular anastomosis, the surgical connection of blood vessels, is essential
in procedures such as organ transplants and reconstructive surgeries. The
precision required limits accessibility due to the extensive training needed,
with manual suturing leading to variable outcomes and revision rates up to
7.9%. Existing robotic systems, while promising, are either fully teleoperated
or lack the capabilities necessary for autonomous vascular anastomosis. We
present the Micro Smart Tissue Autonomous Robot (micro-STAR), an autonomous
robotic system designed to perform vascular anastomosis on small-diameter
vessels. The micro-STAR system integrates a novel suturing tool equipped with
Optical Coherence Tomography (OCT) fiber-optic sensor and a microcamera,
enabling real-time tissue detection and classification. Our system autonomously
places sutures and manipulates tissue with minimal human intervention. In an ex
vivo study, micro-STAR achieved outcomes competitive with experienced surgeons
in terms of leak pressure, lumen reduction, and suture placement variation,
completing 90% of sutures without human intervention. This represents the first
instance of a robotic system autonomously performing vascular anastomosis on
real tissue, offering significant potential for improving surgical precision
and expanding access to high-quality care.
comment: This paper was submitted to IEEE TMRB and is currently under review.
There are 9 pages, 9 figures, and 2 tables
♻ ☆ Towards Realistic UAV Vision-Language Navigation: Platform, Benchmark, and Methodology
Xiangyu Wang, Donglin Yang, Ziqin Wang, Hohin Kwan, Jinyu Chen, Wenjun Wu, Hongsheng Li, Yue Liao, Si Liu
Developing agents capable of navigating to a target location based on
language instructions and visual information, known as vision-language
navigation (VLN), has attracted widespread interest. Most research has focused
on ground-based agents, while UAV-based VLN remains relatively underexplored.
Recent efforts in UAV vision-language navigation predominantly adopt
ground-based VLN settings, relying on predefined discrete action spaces and
neglecting the inherent disparities in agent movement dynamics and the
complexity of navigation tasks between ground and aerial environments. To
address these disparities and challenges, we propose solutions from three
perspectives: platform, benchmark, and methodology. To enable realistic UAV
trajectory simulation in VLN tasks, we propose the OpenUAV platform, which
features diverse environments, realistic flight control, and extensive
algorithmic support. We further construct a target-oriented VLN dataset
consisting of approximately 12k trajectories on this platform, serving as the
first dataset specifically designed for realistic UAV VLN tasks. To tackle the
challenges posed by complex aerial environments, we propose an assistant-guided
UAV object search benchmark called UAV-Need-Help, which provides varying levels
of guidance information to help UAVs better accomplish realistic VLN tasks. We
also propose a UAV navigation LLM that, given multi-view images, task
descriptions, and assistant instructions, leverages the multimodal
understanding capabilities of the MLLM to jointly process visual and textual
information, and performs hierarchical trajectory generation. The evaluation
results of our method significantly outperform the baseline models, while there
remains a considerable gap between our results and those achieved by human
operators, underscoring the challenge presented by the UAV-Need-Help task.
♻ ☆ HBTP: Heuristic Behavior Tree Planning with Large Language Model Reasoning
Behavior Trees (BTs) are increasingly becoming a popular control structure in
robotics due to their modularity, reactivity, and robustness. In terms of BT
generation methods, BT planning shows promise for generating reliable BTs.
However, the scalability of BT planning is often constrained by prolonged
planning times in complex scenarios, largely due to a lack of domain knowledge.
In contrast, pre-trained Large Language Models (LLMs) have demonstrated task
reasoning capabilities across various domains, though the correctness and
safety of their planning remain uncertain. This paper proposes integrating BT
planning with LLM reasoning, introducing Heuristic Behavior Tree Planning
(HBTP)-a reliable and efficient framework for BT generation. The key idea in
HBTP is to leverage LLMs for task-specific reasoning to generate a heuristic
path, which BT planning can then follow to expand efficiently. We first
introduce the heuristic BT expansion process, along with two heuristic variants
designed for optimal planning and satisficing planning, respectively. Then, we
propose methods to address the inaccuracies of LLM reasoning, including action
space pruning and reflective feedback, to further enhance both reasoning
accuracy and planning efficiency. Experiments demonstrate the theoretical
bounds of HBTP, and results from four datasets confirm its practical
effectiveness in everyday service robot applications.
♻ ☆ Grounding Robot Policies with Visuomotor Language Guidance
Recent advances in the fields of natural language processing and computer
vision have shown great potential in understanding the underlying dynamics of
the world from large-scale internet data. However, translating this knowledge
into robotic systems remains an open challenge, given the scarcity of
human-robot interactions and the lack of large-scale datasets of real-world
robotic data. Previous robot learning approaches such as behavior cloning and
reinforcement learning have shown great capabilities in learning robotic skills
from human demonstrations or from scratch in specific environments. However,
these approaches often require task-specific demonstrations or designing
complex simulation environments, which limits the development of generalizable
and robust policies for new settings. Aiming to address these limitations, we
propose an agent-based framework for grounding robot policies to the current
context, considering the constraints of a current robot and its environment
using visuomotor-grounded language guidance. The proposed framework is composed
of a set of conversational agents designed for specific roles -- namely,
high-level advisor, visual grounding, monitoring, and robotic agents. Given a
base policy, the agents collectively generate guidance at run time to shift the
action distribution of the base policy towards more desirable future states. We
demonstrate that our approach can effectively guide manipulation policies to
achieve significantly higher success rates both in simulation and in real-world
experiments without the need for additional human demonstrations or extensive
exploration. Project videos at https://sites.google.com/view/motorcortex/home.
comment: 19 pages, 6 figures, 1 table
♻ ☆ Context-Aware Command Understanding for Tabletop Scenarios
This paper presents a novel hybrid algorithm designed to interpret natural
human commands in tabletop scenarios. By integrating multiple sources of
information, including speech, gestures, and scene context, the system extracts
actionable instructions for a robot, identifying relevant objects and actions.
The system operates in a zero-shot fashion, without reliance on predefined
object models, enabling flexible and adaptive use in various environments. We
assess the integration of multiple deep learning models, evaluating their
suitability for deployment in real-world robotic setups. Our algorithm performs
robustly across different tasks, combining language processing with visual
grounding. In addition, we release a small dataset of video recordings used to
evaluate the system. This dataset captures real-world interactions in which a
human provides instructions in natural language to a robot, a contribution to
future research on human-robot interaction. We discuss the strengths and
limitations of the system, with particular focus on how it handles multimodal
command interpretation, and its ability to be integrated into symbolic robotic
frameworks for safe and explainable decision-making.
♻ ☆ An Algorithm for Distributed Computation of Reachable Sets for Multi-Agent Systems
In this paper, we consider the problem of distributed reachable set
computation for multi-agent systems (MASs) interacting over an undirected,
stationary graph. A full state-feedback control input for such MASs depends no
only on the current agent's state, but also of its neighbors. However, in most
MAS applications, the dynamics are obscured by individual agents. This makes
reachable set computation, in a fully distributed manner, a challenging
problem. We utilize the ideas of polytopic reachable set approximation and
generalize it to a MAS setup. We formulate the resulting sub-problems in a
fully distributed manner and provide convergence guarantees for the associated
computations. The proposed algorithm's convergence is proved for two cases:
static MAS graphs, and time-varying graphs under certain restrictions.
comment: 10 pages, 4 figures, 1 algorithm float. Preprint submitted to ACC
2025 for review
♻ ☆ Demonstration Based Explainable AI for Learning from Demonstration Methods
Learning from Demonstration (LfD) is a powerful type of machine learning that
can allow novices to teach and program robots to complete various tasks.
However, the learning process for these systems may still be difficult for
novices to interpret and understand, making effective teaching challenging.
Explainable artificial intelligence (XAI) aims to address this challenge by
explaining a system to the user. In this work, we investigate XAI within LfD by
implementing an adaptive explanatory feedback system on an inverse
reinforcement learning (IRL) algorithm. The feedback is implemented by
demonstrating selected learnt trajectories to users. The system adapts to user
teaching by categorizing and then selectively sampling trajectories shown to a
user, to show a representative sample of both successful and unsuccessful
trajectories. The system was evaluated through a user study with 26
participants teaching a robot a navigation task. The results of the user study
demonstrated that the proposed explanatory feedback system can improve robot
performance, teaching efficiency and user understanding of the robot.
comment: 8 Pages, 9 Figures, 2 Tables
♻ ☆ A Planar-Symmetric SO(3) Representation for Learning Grasp Detection
Planar-symmetric hands, such as parallel grippers, are widely adopted in both
research and industrial fields. Their symmetry, however, introduces ambiguity
and discontinuity in the SO(3) representation, which hinders both the training
and inference of neural-network-based grasp detectors. We propose a novel SO(3)
representation that can parametrize a pair of planar-symmetric poses with a
single parameter set by leveraging the 2D Bingham distribution. We also detail
a grasp detector based on our representation, which provides a more consistent
rotation output. An intensive evaluation with multiple grippers and objects in
both the simulation and the real world quantitatively shows our approach's
contribution.
comment: Accepted by CoRL2024
♻ ☆ Theia: Distilling Diverse Vision Foundation Models for Robot Learning
Jinghuan Shang, Karl Schmeckpeper, Brandon B. May, Maria Vittoria Minniti, Tarik Kelestemur, David Watkins, Laura Herlant
Vision-based robot policy learning, which maps visual inputs to actions,
necessitates a holistic understanding of diverse visual tasks beyond
single-task needs like classification or segmentation. Inspired by this, we
introduce Theia, a vision foundation model for robot learning that distills
multiple off-the-shelf vision foundation models trained on varied vision tasks.
Theia's rich visual representations encode diverse visual knowledge, enhancing
downstream robot learning. Extensive experiments demonstrate that Theia
outperforms its teacher models and prior robot learning models using less
training data and smaller model sizes. Additionally, we quantify the quality of
pre-trained visual representations and hypothesize that higher entropy in
feature norm distributions leads to improved robot learning performance. Code,
models, and demo are available at https://theia.theaiinstitute.com.
comment: CoRL 2024
♻ ☆ Unpacking Failure Modes of Generative Policies: Runtime Monitoring of Consistency and Progress
Christopher Agia, Rohan Sinha, Jingyun Yang, Zi-ang Cao, Rika Antonova, Marco Pavone, Jeannette Bohg
Robot behavior policies trained via imitation learning are prone to failure
under conditions that deviate from their training data. Thus, algorithms that
monitor learned policies at test time and provide early warnings of failure are
necessary to facilitate scalable deployment. We propose Sentinel, a runtime
monitoring framework that splits the detection of failures into two
complementary categories: 1) Erratic failures, which we detect using
statistical measures of temporal action consistency, and 2) task progression
failures, where we use Vision Language Models (VLMs) to detect when the policy
confidently and consistently takes actions that do not solve the task. Our
approach has two key strengths. First, because learned policies exhibit diverse
failure modes, combining complementary detectors leads to significantly higher
accuracy at failure detection. Second, using a statistical temporal action
consistency measure ensures that we quickly detect when multimodal, generative
policies exhibit erratic behavior at negligible computational cost. In
contrast, we only use VLMs to detect failure modes that are less
time-sensitive. We demonstrate our approach in the context of diffusion
policies trained on robotic mobile manipulation domains in both simulation and
the real world. By unifying temporal consistency detection and VLM runtime
monitoring, Sentinel detects 18% more failures than using either of the two
detectors alone and significantly outperforms baselines, thus highlighting the
importance of assigning specialized detectors to complementary categories of
failure. Qualitative results are made available at
https://sites.google.com/stanford.edu/sentinel.
comment: Project page: https://sites.google.com/stanford.edu/sentinel. 35
pages, 9 figures. Accepted to the Conference on Robot Learning (CoRL) 2024
♻ ☆ Mapping the Unseen: Unified Promptable Panoptic Mapping with Dynamic Labeling using Foundation Models
In the field of robotics and computer vision, efficient and accurate semantic
mapping remains a significant challenge due to the growing demand for
intelligent machines that can comprehend and interact with complex
environments. Conventional panoptic mapping methods, however, are limited by
predefined semantic classes, thus making them ineffective for handling novel or
unforeseen objects. In response to this limitation, we introduce the Unified
Promptable Panoptic Mapping (UPPM) method. UPPM utilizes recent advances in
foundation models to enable real-time, on-demand label generation using natural
language prompts. By incorporating a dynamic labeling strategy into traditional
panoptic mapping techniques, UPPM provides significant improvements in
adaptability and versatility while maintaining high performance levels in map
reconstruction. We demonstrate our approach on real-world and simulated
datasets. Results show that UPPM can accurately reconstruct scenes and segment
objects while generating rich semantic labels through natural language
interactions. A series of ablation experiments validated the advantages of
foundation model-based labeling over fixed label sets.
comment: This paper is under consideration at Pattern Recognition Letters
♻ ☆ Improving Robotic Arms through Natural Language Processing, Computer Vision, and Edge Computing
This paper introduces a prototype for a new approach to assistive robotics,
integrating edge computing with Natural Language Processing (NLP) and computer
vision to enhance the interaction between humans and robotic systems. Our proof
of concept demonstrates the feasibility of using large language models (LLMs)
and vision systems in tandem for interpreting and executing complex commands
conveyed through natural language. This integration aims to improve the
intuitiveness and accessibility of assistive robotic systems, making them more
adaptable to the nuanced needs of users with disabilities. By leveraging the
capabilities of edge computing, our system has the potential to minimize
latency and support offline capability, enhancing the autonomy and
responsiveness of assistive robots. Experimental results from our
implementation on a robotic arm show promising outcomes in terms of accurate
intent interpretation and object manipulation based on verbal commands. This
research lays the groundwork for future developments in assistive robotics,
focusing on creating highly responsive, user-centric systems that can
significantly improve the quality of life for individuals with disabilities.
For video demonstrations and source code, please refer to:
https://tinyurl.com/EnhancedArmEdgeNLP.
♻ ☆ Deployment of Large Language Models to Control Mobile Robots at the Edge
Pascal Sikorski, Leendert Schrader, Kaleb Yu, Lucy Billadeau, Jinka Meenakshi, Naveena Mutharasan, Flavio Esposito, Hadi AliAkbarpour, Madi Babaiasl
This paper investigates the possibility of intuitive human-robot interaction
through the application of Natural Language Processing (NLP) and Large Language
Models (LLMs) in mobile robotics. This work aims to explore the feasibility of
using these technologies for edge-based deployment, where traditional cloud
dependencies are eliminated. The study specifically contrasts the performance
of GPT-4-Turbo, which requires cloud connectivity, with an offline-capable,
quantized version of LLaMA 2 (LLaMA 2-7B.Q5 K M). These results show that
GPT-4-Turbo delivers superior performance in interpreting and executing complex
commands accurately, whereas LLaMA 2 exhibits significant limitations in
consistency and reliability of command execution. Communication between the
control computer and the mobile robot is established via a Raspberry Pi Pico W,
which wirelessly receives commands from the computer without internet
dependency and transmits them through a wired connection to the robot's Arduino
controller. This study highlights the potential and challenges of implementing
LLMs and NLP at the edge, providing groundwork for future research into fully
autonomous and network-independent robotic systems. For video demonstrations
and source code, please refer to: https://tinyurl.com/MobileRobotGPT4LLaMA2024.
♻ ☆ AO-Grasp: Articulated Object Grasp Generation
Carlota Parés Morlans, Claire Chen, Yijia Weng, Michelle Yi, Yuying Huang, Nick Heppert, Linqi Zhou, Leonidas Guibas, Jeannette Bohg
We introduce AO-Grasp, a grasp proposal method that generates 6 DoF grasps
that enable robots to interact with articulated objects, such as opening and
closing cabinets and appliances. AO-Grasp consists of two main contributions:
the AO-Grasp Model and the AO-Grasp Dataset. Given a segmented partial point
cloud of a single articulated object, the AO-Grasp Model predicts the best
grasp points on the object with an Actionable Grasp Point Predictor. Then, it
finds corresponding grasp orientations for each of these points, resulting in
stable and actionable grasp proposals. We train the AO-Grasp Model on our new
AO-Grasp Dataset, which contains 78K actionable parallel-jaw grasps on
synthetic articulated objects. In simulation, AO-Grasp achieves a 45.0 % grasp
success rate, whereas the highest performing baseline achieves a 35.0% success
rate. Additionally, we evaluate AO-Grasp on 120 real-world scenes of objects
with varied geometries, articulation axes, and joint states, where AO-Grasp
produces successful grasps on 67.5% of scenes, while the baseline only produces
successful grasps on 33.3% of scenes. To the best of our knowledge, AO-Grasp is
the first method for generating 6 DoF grasps on articulated objects directly
from partial point clouds without requiring part detection or hand-designed
grasp heuristics. Project website: https://stanford-iprl-lab.github.io/ao-grasp
comment: Project website: https://stanford-iprl-lab.github.io/ao-grasp
♻ ☆ Large Language Models for Orchestrating Bimanual Robots
Although there has been rapid progress in endowing robots with the ability to
solve complex manipulation tasks, generating control policies for bimanual
robots to solve tasks involving two hands is still challenging because of the
difficulties in effective temporal and spatial coordination. With emergent
abilities in terms of step-by-step reasoning and in-context learning, Large
Language Models (LLMs) have demonstrated promising potential in a variety of
robotic tasks. However, the nature of language communication via a single
sequence of discrete symbols makes LLM-based coordination in continuous space a
particular challenge for bimanual tasks. To tackle this challenge, we present
LAnguage-model-based Bimanual ORchestration (LABOR), an agent utilizing an LLM
to analyze task configurations and devise coordination control policies for
addressing long-horizon bimanual tasks. We evaluate our method through
simulated experiments involving two classes of long-horizon tasks using the
NICOL humanoid robot. Our results demonstrate that our method outperforms the
baseline in terms of success rate. Additionally, we thoroughly analyze failure
cases, offering insights into LLM-based approaches in bimanual robotic control
and revealing future research trends. The project website can be found at
http://labor-agent.github.io.
comment: Accepted in Humanoids 2024. The project website can be found at
http://labor-agent.github.io
♻ ☆ Hybrid Gripper with Passive Pneumatic Soft Joints for Grasping Deformable Thin Objects
Grasping a variety of objects remains a key challenge in the development of
versatile robotic systems. The human hand is remarkably dexterous, capable of
grasping and manipulating objects with diverse shapes, mechanical properties,
and textures. Inspired by how humans use two fingers to pick up thin and large
objects such as fabric or sheets of paper, we aim to develop a gripper
optimized for grasping such deformable objects. Observing how the soft and
flexible fingertip joints of the hand approach and grasp thin materials, a
hybrid gripper design that incorporates both soft and rigid components was
proposed. The gripper utilizes a soft pneumatic ring wrapped around a rigid
revolute joint to create a flexible two-fingered gripper. Experiments were
conducted to characterize and evaluate the gripper performance in handling
sheets of paper and other objects. Compared to rigid grippers, the proposed
design improves grasping efficiency and reduces the gripping distance by up to
eightfold.
♻ ☆ DragTraffic: Interactive and Controllable Traffic Scene Generation for Autonomous Driving
Evaluating and training autonomous driving systems require diverse and
scalable corner cases. However, most existing scene generation methods lack
controllability, accuracy, and versatility, resulting in unsatisfactory
generation results. Inspired by DragGAN in image generation, we propose
DragTraffic, a generalized, interactive, and controllable traffic scene
generation framework based on conditional diffusion. DragTraffic enables
non-experts to generate a variety of realistic driving scenarios for different
types of traffic agents through an adaptive mixture expert architecture. We
employ a regression model to provide a general initial solution and a
refinement process based on the conditional diffusion model to ensure
diversity. User-customized context is introduced through cross-attention to
ensure high controllability. Experiments on a real-world driving dataset show
that DragTraffic outperforms existing methods in terms of authenticity,
diversity, and freedom. Demo videos and code are available at
https://chantsss.github.io/Dragtraffic/.
♻ ☆ Safe Task Planning for Language-Instructed Multi-Robot Systems using Conformal Prediction
This paper addresses task planning problems for language-instructed robot
teams. Tasks are expressed in natural language (NL), requiring the robots to
apply their capabilities at various locations and semantic objects. Several
recent works have addressed similar planning problems by leveraging pre-trained
Large Language Models (LLMs) to design effective multi-robot plans. However,
these approaches lack mission completion guarantees. To address this challenge,
we introduce a new distributed LLM-based planner, called S-ATLAS for Safe
plAnning for Teams of Language-instructed AgentS, that is capable of achieving
user-defined mission success rates. This is accomplished by leveraging
conformal prediction (CP), a distribution-free uncertainty quantification tool
in black-box models. CP allows the proposed multi-robot planner to reason about
its inherent uncertainty in a distributed fashion, enabling robots to make
individual decisions when they are sufficiently certain and seek help
otherwise. We show, both theoretically and empirically, that the proposed
planner can achieve user-specified task success rates while minimizing the
overall number of help requests. We provide comparative experiments against
related works showing that our method is significantly more computational
efficient and achieves lower help rates. The advantage of our algorithm over
baselines becomes more pronounced with increasing robot team size.
♻ ☆ Semantic Region Aware Autonomous Exploration for Multi-Type Map Construction in Unknown Indoor Environments
Mainstream autonomous exploration methods usually perform
excessively-repeated explorations for the same region, leading to long
exploration time and exploration trajectory in complex scenes. To handle this
issue, we propose a novel semantic region aware autonomous exploration method,
the core idea of which is considering the information of semantic regions to
optimize the autonomous navigation strategy. Our method enables the mobile
robot to fully explore the current semantic region before moving to the next
region, contributing to avoid excessively-repeated explorations and accelerate
the exploration speed. In addition, compared with existing au?tonomous
exploration methods that usually construct the single-type map, our method
allows to construct four types of maps including point cloud map, occupancy
grid map, topological map, and semantic map. The experiment results demonstrate
that our method achieves the highest 50.7% exploration time reduction and 48.1%
exploration trajectory length reduction while maintaining >98% exploration rate
when comparing with the classical RRT (Rapid-exploration Random Tree) based
autonomous exploration method.
♻ ☆ Co-Design Optimisation of Morphing Topology and Control of Winged Drones
Fabio Bergonti, Gabriele Nava, Valentin Wüest, Antonello Paolino, Giuseppe L'Erario, Daniele Pucci, Dario Floreano
The design and control of winged aircraft and drones is an iterative process
aimed at identifying a compromise of mission-specific costs and constraints.
When agility is required, shape-shifting (morphing) drones represent an
efficient solution. However, morphing drones require the addition of actuated
joints that increase the topology and control coupling, making the design
process more complex. We propose a co-design optimisation method that assists
the engineers by proposing a morphing drone's conceptual design that includes
topology, actuation, morphing strategy, and controller parameters. The method
consists of applying multi-objective constraint-based optimisation to a
multi-body winged drone with trajectory optimisation to solve the motion
intelligence problem under diverse flight mission requirements, such as energy
consumption and mission completion time. We show that co-designed morphing
drones outperform fixed-winged drones in terms of energy efficiency and mission
time, suggesting that the proposed co-design method could be a useful addition
to the aircraft engineering toolbox.
♻ ☆ C$^3$P-VoxelMap: Compact, Cumulative and Coalescible Probabilistic Voxel Mapping
This work presents a compact, cumulative and coalescible probabilistic voxel
mapping method to enhance performance, accuracy and memory efficiency in LiDAR
odometry. Probabilistic voxel mapping requires storing past point clouds and
re-iterating on them to update the uncertainty every iteration, which consumes
large memory space and CPU cycles. To solve this problem, we propose a
two-folded strategy. First, we introduce a compact point-free representation
for probabilistic voxels and derive a cumulative update of the planar
uncertainty without caching original point clouds. Our voxel structure only
keeps track of a predetermined set of statistics for points that lie inside it.
This method reduces the runtime complexity from $O(MN)$ to $O(N)$ and the space
complexity from $O(N)$ to $O(1)$ where $M$ is the number of iterations and $N$
is the number of points. Second, to further minimize memory usage and enhance
mapping accuracy, we provide a strategy to dynamically merge voxels associated
with the same physical planes by taking advantage of the geometric features in
the real world. Rather than scanning for these coalescible voxels constantly at
every iteration, our merging strategy accumulates voxels in a
locality-sensitive hash and triggers merging lazily. On-demand merging not only
reduces memory footprint with minimal computational overhead but also improves
localization accuracy thanks to cross-voxel denoising. Experiments exhibit 20%
higher accuracy, 20% faster performance and 70% lower memory consumption than
the state-of-the-art.
♻ ☆ Learning to Plan Maneuverable and Agile Flight Trajectory with Optimization Embedded Networks
In recent times, an increasing number of researchers have been devoted to
utilizing deep neural networks for end-to-end flight navigation. This approach
has gained traction due to its ability to bridge the gap between perception and
planning that exists in traditional methods, thereby eliminating delays between
modules. However, the practice of replacing original modules with neural
networks in a black-box manner diminishes the overall system's robustness and
stability. It lacks principled explanations and often fails to consistently
generate high-quality motion trajectories. Furthermore, such methods often
struggle to rigorously account for the robot's kinematic constraints, resulting
in the generation of trajectories that cannot be executed satisfactorily. In
this work, we combine the advantages of traditional methods and neural networks
by proposing an optimization-embedded neural network. This network can learn
high-quality trajectories directly from visual inputs without the need of
mapping, while ensuring dynamic feasibility. Here, the deep neural network is
employed to directly extract environment safety regions from depth images.
Subsequently, we employ a model-based approach to represent these regions as
safety constraints in trajectory optimization. Leveraging the availability of
highly efficient optimization algorithms, our method robustly converges to
feasible and optimal solutions that satisfy various user-defined constraints.
Moreover, we differentiate the optimization process, allowing it to be trained
as a layer within the neural network. This approach facilitates the direct
interaction between perception and planning, enabling the network to focus more
on the spatial regions where optimal solutions exist. As a result, it further
enhances the quality and stability of the generated trajectories.
comment: Some statements in the introduction may be controversial
♻ ☆ Open-Vocabulary Action Localization with Iterative Visual Prompting
Video action localization aims to find the timings of specific actions from a
long video. Although existing learning-based approaches have been successful,
they require annotating videos, which comes with a considerable labor cost.
This paper proposes a learning-free, open-vocabulary approach based on emerging
off-the-shelf vision-language models (VLMs). The challenge stems from the fact
that VLMs are neither designed to process long videos nor tailored for finding
actions. We overcome these problems by extending an iterative visual prompting
technique. Specifically, we sample video frames and create a concatenated image
with frame index labels, making a VLM guess a frame that is considered to be
closest to the start and end of the action. Iterating this process by narrowing
a sampling time window results in finding the specific frames corresponding to
the start and end of an action. We demonstrate that this technique yields
reasonable performance, achieving results comparable to state-of-the-art
zero-shot action localization. These results illustrate a practical extension
of VLMs for understanding videos. A sample code is available at
https://microsoft.github.io/VLM-Video-Action-Localization/.
comment: 9 pages, 5 figures, 6 tables. Last updated on October 9th, 2024
♻ ☆ SE(3) Linear Parameter Varying Dynamical Systems for Globally Asymptotically Stable End-Effector Control
Linear Parameter Varying Dynamical Systems (LPV-DS) encode trajectories into
an autonomous first-order DS that enables reactive responses to perturbations,
while ensuring globally asymptotic stability at the target. However, the
current LPV-DS framework is established on Euclidean data only and has not been
applicable to broader robotic applications requiring pose control. In this
paper we present an extension to the current LPV-DS framework, named
Quaternion-DS, which efficiently learns a DS-based motion policy for
orientation. Leveraging techniques from differential geometry and Riemannian
statistics, our approach properly handles the non-Euclidean orientation data in
quaternion space, enabling the integration with positional control, namely
SE(3) LPV-DS, so that the synergistic behaviour within the full SE(3) pose is
preserved. Through simulation and real robot experiments, we validate our
method, demonstrating its ability to efficiently and accurately reproduce the
original SE(3) trajectory while exhibiting strong robustness to perturbations
in task space.
♻ ☆ Cross-Embodied Affordance Transfer through Learning Affordance Equivalences
Affordances represent the inherent effect and action possibilities that
objects offer to the agents within a given context. From a theoretical
viewpoint, affordances bridge the gap between effect and action, providing a
functional understanding of the connections between the actions of an agent and
its environment in terms of the effects it can cause. In this study, we propose
a deep neural network model that unifies objects, actions, and effects into a
single latent vector in a common latent space that we call the affordance
space. Using the affordance space, our system can generate effect trajectories
when action and object are given and can generate action trajectories when
effect trajectories and objects are given. Our model does not learn the
behavior of individual objects acted upon by a single agent. Still, rather, it
forms a `shared affordance representation' spanning multiple agents and
objects, which we call Affordance Equivalence. Affordance Equivalence
facilitates not only action generalization over objects but also Cross
Embodiment transfer linking actions of different robots. In addition to the
simulation experiments that demonstrate the proposed model's range of
capabilities, we also showcase that our model can be used for direct imitation
in real-world settings.
comment: 10 pages, 9 figures, Submitted to IEEE Transactions on Cognitive and
Developmental Systems
♻ ☆ CitDet: A Benchmark Dataset for Citrus Fruit Detection
Jordan A. James, Heather K. Manching, Matthew R. Mattia, Kim D. Bowman, Amanda M. Hulse-Kemp, William J. Beksi
In this letter, we present a new dataset to advance the state of the art in
detecting citrus fruit and accurately estimate yield on trees affected by the
Huanglongbing (HLB) disease in orchard environments via imaging. Despite the
fact that significant progress has been made in solving the fruit detection
problem, the lack of publicly available datasets has complicated direct
comparison of results. For instance, citrus detection has long been of interest
to the agricultural research community, yet there is an absence of work,
particularly involving public datasets of citrus affected by HLB. To address
this issue, we enhance state-of-the-art object detection methods for use in
typical orchard settings. Concretely, we provide high-resolution images of
citrus trees located in an area known to be highly affected by HLB, along with
high-quality bounding box annotations of citrus fruit. Fruit on both the trees
and the ground are labeled to allow for identification of fruit location, which
contributes to advancements in yield estimation and potential measure of HLB
impact via fruit drop. The dataset consists of over 32,000 bounding box
annotations for fruit instances contained in 579 high-resolution images. In
summary, our contributions are the following: (i) we introduce a novel dataset
along with baseline performance benchmarks on multiple contemporary object
detection algorithms, (ii) we show the ability to accurately capture fruit
location on tree or on ground, and finally (ii) we present a correlation of our
results with yield estimations.
comment: To be published in IEEE Robotics and Automation Letters (RA-L)