Raphael Memmesheimer

My long-term research goal is to develop autonomous service robots that assist people in everyday environments: robots that perceive, understand, and learn from human behavior using multi-modal sensory information, and that prove themselves outside the lab. My work pursues this goal along three threads:

• Multi-modal perception and action understanding. Recognizing and segmenting human actions in time-series data such as videos, inertial measurement units (IMUs), Wi-Fi CSI fingerprints, and skeleton sequences, so that robots understand and respond to human activities robustly and naturally.
• Learning from human behavior. Enabling robots to acquire new skills from human demonstrations and multi-modal observations rather than explicit programming.
• Integrated service robots in the real world. Building complete domestic service robots that combine perception, task planning, mobile manipulation, and human-robot interaction, and validating them in international competitions as real-world benchmarks.

I am a Postdoctoral Researcher at the Autonomous Intelligent Systems (AIS) group of Prof. Dr. Sven Behnke at the University of Bonn, where I re-initiated and lead the NimbRo@Home team. I am also a founding member of the Center for Robotics at the University of Bonn. I completed my PhD in the Active Vision Group under Prof. Dr. Dietrich Paulus at the University of Koblenz, with a thesis on multi-modal action recognition in time-series data.

Robot competitions serve as rigorous, real-world benchmarks for this research: as team leader, I have guided teams at two universities to multiple championship titles, including winning the RoboCup@Home World Championship five times (with three consecutive victories) and top placements in the European Robotics League and the World Robot Summit. As a member of the NimbRo Avatar team, I contributed to winning the ANA Avatar XPRIZE challenge with a grand prize of 5 million USD.

• GitHub
• Scholar
• YouTube
•   ORCID

News

• 07/2026: Best Paper Award for the paper: Efficient Image Annotation via Semi-Supervised Object Segmentation with Label Propagation, at the RoboCup Symposium.
• 07/2026: We were placed 3rd out of 24 teams at the RoboCup@Home World Championship in Seoul (South Korea). More information here.
• 03/2026: The NimbRo@Home team won the RoboCup German Open 2026 in the @Home League. More information on our website.
• 07/2025: We ended up second at RoboCup World Championship in Salvador (Brazil). More information here.
• 03/2025: Our team NimbRo won the RoboCup German Open 2025 in the @Home League. Here you find more information.
• 12/2024: The scientific magazine Bild der Wissenschaft wrote an article about our robot competition activities.
• 09/2024: We were featured by the local broadcasting channel WDR in their news Lokalzeit.
• 07/2024: We won the RoboCup@Home World Championship in the Open Platform League! Find more Info here.
• 07/2024: Open Vocabulary 6D pose estimation project funded.
• 04/2024: Won the RoboCup German Open in the @Home League.

Selected Publications

• 
  Efficient Image Annotation via Semi-Supervised Object Segmentation with Label Propagation
  Tutevych, Vitalii; Memmesheimer, Raphael; Eichler, Luca; Pavlichenko, Dmytro; Schilke, Fynn; Krudewig, Rodja; Behnke, Sven
  RoboCup 2026: Robot World Cup (Accepted for publication) (2026) Best Paper Award, RoboCup Symposium 2026
  [arxiv] [github]
• 
  Adaptive Domestic Service Robotics through Foundation Models for Perception, Interaction, and Action
  Memmesheimer, Raphael; Pavlichenko, Dmytro; Kruzhkov, Evgenii; Bode, Jonas; Schilke, Fynn; Tutevych, Vitalii; Lenz, Christian; Schreiber, Michael; Behnke, Sven
  Proceedings of the 2nd German Robotics Conference (GRC) (Accepted for presentation) (2026)
  
• 
  OMCL: Open-vocabulary Monte Carlo Localization
  Kruzhkov, Evgenii; Memmesheimer, Raphael; Behnke, Sven
  IEEE Robotics and Automation Letters (RAL) (2026)
  [DOI] [arxiv] [github]
• 
  NimbRo Wins ANA Avatar XPRIZE Immersive Telepresence Competition: Human-Centric Evaluation and Lessons Learned
  Lenz, Christian; Schwarz, Max; Rochow, Andre; Pätzold, Bastian; Memmesheimer, Raphael; Schreiber, Michael; Behnke, Sven
  Int. J. Soc. Robotics (2025)
  [DOI] [video] [project_page] [arxiv]
• 
  RoboCup@Home 2024 OPL Winner NimbRo: Anthropomorphic Service Robots Using Foundation Models for Perception and Planning
  Memmesheimer, Raphael; Nogga, Jan; Pätzold, Bastian; Kruzhkov, Evgenii; Bultmann, Simon; Schreiber, Michael; Bode, Jonas; Karacora, Bertan; Park, Juhui; Savinykh, Alena; Behnke, Sven
  RoboCup 2024: Robot World Cup XXVII Springer (2024)
  [DOI] [project_page] [arxiv]
• 
  External Camera-Based Mobile Robot Pose Estimation for Collaborative Perception with Smart Edge Sensors
  Bultmann, Simon; Memmesheimer, Raphael; Behnke, Sven
  IEEE International Conference on Robotics and Automation, ICRA 2023, London, UK, May 29 - June 2, 2023 IEEE (2023)
  [video][arxiv] [DOI]
• 
  Skeleton-DML: Deep Metric Learning for Skeleton-Based One-Shot Action Recognition
  Memmesheimer, Raphael; Häring, Simon; Theisen, Nick; Paulus, Dietrich
  IEEE/CVF Winter Conference on Applications of Computer Vision, WACV 2022, Waikoloa, HI, USA, January 3-8, 2022 IEEE (2022)
  [arxiv][video][github] [DOI]
• 
  SL-DML: Signal Level Deep Metric Learning for Multimodal One-Shot Action Recognition
  Memmesheimer, Raphael; Theisen, Nick; Paulus, Dietrich
  25th International Conference on Pattern Recognition, ICPR 2020, Virtual Event / Milan, Italy, January 10-15, 2021 IEEE (2020)

  Abstract:

  Recognizing an activity with a single reference sample using metric learning approaches is a promising research field. The majority of few-shot methods focus on object recognition or face-identification. We propose a metric learning approach to reduce the action recognition problem to a nearest neighbor search in embedding space. We encode signals into images and extract features using a deep residual CNN. Using triplet loss, we learn a feature embedding. The resulting encoder transforms features into an embedding space in which closer distances encode similar actions while higher distances encode different actions. Our approach is based on a signal level formulation and remains flexible across a variety of modalities. It further outperforms the baseline on the large scale NTU RGB+D 120 dataset for the One-Shot action recognition protocol by 5.6%. With just 60% of the training data, our approach still outperforms the baseline approach by 3.7%. With 40% of the training data, our approach performs comparably well to the second follow up. Further, we show that our approach generalizes well in experiments on the UTD-MHAD dataset for inertial, skeleton and fused data and the Simitate dataset for motion capturing data. Furthermore, our inter-joint and inter-sensor experiments suggest good capabilities on previously unseen setups.

  [arxiv][video] [DOI]
• 
  Gimme Signals: Discriminative signal encoding for multimodal activity recognition
  Memmesheimer, Raphael; Theisen, Nick; Paulus, Dietrich
  IEEE/RSJ International Conference on Intelligent Robots and Systems, IROS 2020, Las Vegas, NV, USA, October 24, 2020 - January 24, 2021 IEEE (2020)

  Abstract:

  We present a simple, yet effective and flexible method for action recognition supporting multiple sensor modalities. Multivariate signal sequences are encoded in an image and are then classified using a recently proposed EfficientNet CNN architecture. Our focus was to find an approach that generalizes well across different sensor modalities without specific adaptions while still achieving good results. We apply our method to 4 action recognition datasets containing skeleton sequences, inertial and motion capturing measurements as well as \wifi fingerprints that range up to 120 action classes. Our method defines the current best CNN-based approach on the NTU RGB+D 120 dataset, lifts the state of the art on the ARIL Wi-Fi dataset by +6.78%, improves the UTD-MHAD inertial baseline by +14.4%, the UTD-MHAD skeleton baseline by 1.13% and achieves 96.11% on the Simitate motion capturing data (80/20 split). We further demonstrate experiments on both, modality fusion on a signal level and signal reduction to prevent the representation from overloading.

  [arxiv][video] [DOI]
• 
  Simitate: A Hybrid Imitation Learning Benchmark
  Memmesheimer, Raphael; Kramer, Ivanna; Seib, Viktor; Paulus, Dietrich
  2019 IEEE/RSJ International Conference on Intelligent Robots and Systems, IROS 2019, Macau, SAR, China, November 3-8, 2019 IEEE (2019)

  Abstract:

  We present Simitate, a hybrid benchmarking suite targeting the evaluation of approaches for imitation learning. A dataset containing 1938 sequences where humans perform daily activities in a realistic environment is presented. The dataset is strongly coupled with an integration into a simulator. RGB and depth streams with a resolution of $960 times 540$ at 30Hz and accurate ground truth poses for the demonstrator’s hand, as well as the object in 6 DOF at 120Hz are provided. Along with our dataset we provide the 3D model of the used environment, labeled object images and pre-trained models. A benchmarking suite that aims at fostering comparability and reproducibility supports the development of imitation learning approaches. Further, we propose and integrate evaluation metrics on assessing the quality of effect and trajectory of the imitation performed in simulation. Simitate is available on our project website: https://agas.uni-koblenz.de/simitate/.

  [arxiv][video][project-page] [DOI]

All publications →

Contact

University of Bonn
Autonomous Intelligent Systems
Friedrich-Hirzebruch-Allee 8
53115 Bonn
Room 0.055
Phone: +49 228-73-60805


Location