Raphael Memmesheimer

Research Topics

Multi-Modal Perception and Action Understanding teaser

Multi-Modal Perception and Action Understanding

Recognizing and segmenting human actions in time-series data such as videos, inertial measurements, Wi-Fi CSI fingerprints, and skeleton sequences, so that robots understand and respond to human activities robustly and naturally.

SL-DML (ICPR 2020) · Gimme Signals (IROS 2020) · Skeleton-DML (WACV 2022)

Learning from Human Behavior teaser

Learning from Human Behavior

Enabling robots to acquire new skills from human demonstrations and multi-modal observations rather than explicit programming, so that non-experts can teach robots new tasks.

Simitate (IROS 2019) · Robotic Imitation by Markerless Visual Observation (ICARSC 2020)

Integrated Service Robots in the Real World teaser

Integrated Service Robots in the Real World

Building complete domestic service robots that combine perception, task planning, mobile manipulation, and human-robot interaction, validated in international competitions such as RoboCup@Home and the ANA Avatar XPRIZE as real-world benchmarks.

Adaptive Domestic Service Robotics (GRC 2026) · RoboCup@Home 2024 OPL Winner NimbRo (RoboCup 2024)

Datasets

Haptic Object Recognition teaser

Haptic Object Recognition

A Prototypical Full-Hand Tactile Skin Grasp Dataset with an Allegro Hand

A dataset of in-hand grasps recorded with an Allegro Hand fully covered by a XELA uSkin tactile skin: 368 tri-axial magnetic taxels on 18 sensor pads spanning fingertips, phalanges, and palm, sampled at ~100 Hz. It contains 14 grasps of 9 YCB objects across two recording sessions on different days, each synchronized with hand proprioception, RGB video, and scripted grasp-phase annotations. The release adds a machine-readable estimate of the taxel-to-pad mapping, a leakage-free frame-level object-recognition benchmark with cross-session and cross-pose protocols, an interactive web viewer, and the full bag-to-HDF5 extraction pipeline.

Dataset & Viewer · Download (ZIP, 106 MB)

LaserAct teaser

LaserAct

A Multi-LiDAR and RGB-D Dataset for Human Action Recognition

The first human action recognition dataset in which the same actions are observed simultaneously by four LiDARs of different scanning principles (spinning, rosette, micro-motion, and MEMS raster) together with an RGB-D-IR camera and estimated 2D poses. It contains 2,342 curated sequences across 49 action classes performed by 9 subjects, with full extrinsic calibration and a common clock, and defines cross-subject, cross-take, and cross-sensor evaluation protocols. It further ships a person re-identification benchmark that quantifies the residual privacy of camera-free LiDAR sensing.

Project Page · Download

AtHome-2026 teaser

AtHome-2026

A Multi-Venue Dataset for Household Object Instance Segmentation at RoboCup@Home

A collection of 61 household-object instance-segmentation datasets in COCO format, self-captured with domestic service robots at RoboCup@Home competition venues and the home lab. Data from 8 sources (Bonn, Bordeaux, Cologne, Eindhoven, Incheon, Kassel, Nürnberg, Salvador) is unified into one release with a single canonical label space.

Paper (RoboCup 2026) · Zenodo (under review)

LiDAR Person Action Detection teaser

LiDAR Person Action Detection

A Hemisphere Field of View LiDAR Dataset for Person Segmentation and Action Classification

A dataset of 3D scans recorded with an Ouster OSDome-64 hemisphere field of view LiDAR sensor, consisting of annotated scenes where persons perform three different actions: walking, waving, and sitting. It accompanies a MaskDINO-based method that detects and segments persons and recognizes their action states from combined spherical projected multi-channel representations of the LiDAR data with an additional positional encoding. The dataset is released in COCO format together with the trained models and code.

Dataset & Code (GitHub) · Paper (SII 2025) · arXiv

Roughness Sensing teaser

Roughness Sensing

An Audio-Based Surface Roughness Dataset for Haptic Perception in Telepresence

Audio recordings for learning-based surface roughness sensing, captured as a robot fingertip moves across surfaces of interest: two microphones record sound coupled directly through the fingertip and through the air. A learning-based detector analyzes the data in real time and estimates roughness with high temporal resolution and low latency, rendered to the human operator through an audio-based vibrational actuator. The dataset used for training and evaluation is released together with the trained models. The system was part of the winning entry in the ANA Avatar XPRIZE competition finals, where briefly trained judges solved a roughness-based selection task even without additional vision feedback.

Dataset & Code (GitHub) · Paper (SMC 2023) · arXiv

Simitate teaser

Simitate

A Hybrid Imitation Learning Benchmark

A benchmarking suite for imitation learning containing 1938 RGB-D sequences of humans performing daily activities in a realistic environment, with ground-truth 6 DOF hand and object poses and a coupled simulation for evaluating effect and trajectory quality.

Project Page · Paper (IROS 2019) · arXiv · Video

Funding & Grants

Competitive Research Funding

Prize Money & Sponsoring

Academic Service

Reviewer for KI (2026), IROS (2020–2026), ICRA (2019–2026), RO-MAN (2019–2024), RA-L (2019–2026), IEEE SPL (2023), IJCV (2023), Humanoids (2023–2024), SII (2024), THRI (2024), Avatar Workshop (2023–2024).