Dataset Description

Puzzle-STAMPS — A Multimodal Motion–Physiology–Speech Dataset

Experimental Setup

The experimental environment was designed to create a collaborative escape-room experience within a controlled laboratory setting. Teams were given no predefined roles and were free to determine their own collaborative dynamics throughout the experiment. The setup comprises a primary puzzle box, a secondary toolbox, and a hint-delivery system, all integrated with a unified recording pipeline.

The study comprised thirteen unique puzzles organized into eight timed segments. The third segment featured five independent sub-puzzles whose solutions unlocked a final integrating puzzle, while the other seven segments contained a single puzzle each. The tasks demanded diverse coordination skills: shared visual search, spatial reasoning (e.g., path-finding and jigsaw puzzle assembly), optional parallel workflows in segment 3, and real-time teamwork, such as navigating an electronic maze where the display and controls sit on opposite sides of the puzzle box. Some puzzles also allowed brute-force solutions. This variety was designed to elicit diverse leadership and coordination behaviors.

Puzzle Box

The primary task centered on a modified Deadly Hallows electronic puzzle box (Indestroom, HU). The box presents interactive puzzles across its four sides and top surface, combining digital displays (screens, LED indicators) with physical mechanisms (buttons, dials, rotating disks, locks). The box’s Unity-based control application was extended to interface with the recording pipeline, enabling automatic logging of puzzle state transitions (start, solved, failed).

Toolbox

To expand the problem-solving space, the setup was augmented with a secondary toolbox containing seven numbered compartments, each secured by one to three four-digit combination locks. As teams solved specific puzzles on the main box, motorized trapdoors opened automatically to reveal strings of letters, numbers, and emojis taped to their interior. Participants had to communicate or memorize these codes to unlock the corresponding compartments in the toolbox, which contained essential physical artifacts needed for subsequent puzzle stages.

Hint & Timer System

A monitor near the toolbox displayed a synchronized countdown for the current segment and provided time-triggered written hints, delivered at three escalating levels from directional nudges to near-explicit guidance. An audible signal played from a dedicated speaker whenever new information or hints appeared on the screen. If a segment timer reached zero, the system automatically skipped any unsolved puzzles to ensure that the session did not exceed the 180-minute limit. The puzzle box’s Unity-based control application was extended to interface with our recording pipeline, enabling automatic logging of state transitions and skip events with timestamps.

Laboratory Layout

The experiment took place in a laboratory at the Max Planck Institute for Intelligent Systems, Stuttgart. The room was arranged to host one team of four participants at a time to gather around the puzzle box, which was positioned centrally.

Room Configuration

The room was configured as follows:

• Puzzle box: Centrally located; participants stood or sat around it during the session.
• OptiTrack Prime 17W cameras: Eight cameras mounted at ceiling height, providing full-room motion capture at 360 Hz via Motive software.
• Room video cameras: Five fixed Reolink RLC-511 CCTV cameras capturing complementary exocentric viewpoints at 25 fps.
• Experimenter station: Located outside the main interaction area, hidden behind a curtain.

Sensor Placement on Participants

Each participant was simultaneously instrumented with the following sensor systems prior to the session.

Movement & Localization

Physiological Garments

Each participant wore a Healer R2 smart garment (L.I.F.E. Italia S.r.l.), a clinical-grade wearable providing cardio-respiratory signals comparable to standard medical equipment. The sleeveless close-fitting garments are machine-washable and come in a wide range of sizes for men and women to ensure stable sensor-to-skin contact without hindering physical movement. Each garment has its own on-body logger, which synchronized its internal clock to the local network. These physiological channels enable the study of interpersonal synchrony, stress reactivity, and arousal dynamics in collaboration and leadership scenarios.

Audio

Video

Synchronization & Recording Pipeline

All sensing systems were isolated from the internet and synchronized to the main acquisition computer, which served as the local NTP server. The main computer captured both the OptiTrack streams and the transmitted lapel microphone audio, while a dedicated Raspberry Pi 5 recorded the puzzle-box game-state logs and the Pozyx RTLS data. Each physiological garment has its own on-body logger, which synchronized its internal clock to the local network. The CCTV streams were captured continuously on a dedicated Synology DiskStation DS1520+ NAS.

---

Next: Experimental Protocol →