React dataset · action processing

The missing half of the action:
putting force back into pose-only demonstrations

React's actions are computed from OptiTrack sensor poses — like all UMI-style data, the demonstrated pose is the achieved pose. A stiffness controller exerts F = k·(target − actual), so replaying these actions reproduces the motion but presses with zero intended force. This page shows how each action is transformed, on real episode data, and what the transformation is worth.

36 episodes · 72 sensor-sides no F/T sensor involved ↖ full methods & validation 中文版

How an action is processed

pose p_tOptiTrack, 30 Hz

GelSight frame640×480 RGB

→

F̂_ndepth → Winkler,
FEATS-calibrated

→

n̂ = R_t·a_geldual-ball calibrated axis

→

action targetvirtual, past the surface

p_target = p_observed + ( F̂_n / k ) · n̂ k = 1500 N/m

The action stays a pose. In free space F̂_n = 0 and the target is the observed pose — nothing changes. In contact, the target moves past the surface along the gel normal by exactly the displacement an impedance controller at stiffness k needs to exert the demonstrated force. No new action dimension, no force interface at deployment — a DexForce-style transform (arXiv:2501.10356) driven by tactile-estimated rather than measured force.

Live on real data — drag across the trace

90 s of motherboard/2026-05-10/episode_000 (left sensor), centred on the strongest press. Top: estimated normal force. Bottom: the transform's entire effect on the action — the target's offset from the observed pose along the gel normal. The cyan zero-line is the original action; the sensor itself sweeps ±170 mm through this window, which is why the offset is drawn on its own millimetre scale.

F̂ normal [N] observed pose = zero offset (action before) target offset F̂/k along n̂ (action after)

–

F̂ normal

–

target offset F̂/k

–

state

–

drag / hover to scrub · data is the actual per-row output, not a mock-up

The effect, measured

free-space invariance

0.0e+00 m

max |target − pose| when F̂=0, all 72 sides

round-trip error

9e-14 N

k·‖target−pose‖ vs F̂ — machine precision

penetration in contact

1.0 mm median

max 15.2 mm at the hardest ~23 N press

Why this matters for training. Policies trained on raw poses learn "touch the surface and stop": the label says the fingertip halts at the contact plane, so at deployment the controller exerts whatever residual force tracking error happens to produce. With force-informed targets the label itself encodes how hard — DexForce measured near-zero task success without this correction and 76% with it, on kinesthetic demonstrations with measured forces; here the same transform runs from tactile-estimated force, with the estimator validated against FEA ground truth (ρ=0.70 pooled, 0.85 on unseen indenter shapes — see the main page).

the force signal driving the action transform, live under the tactile stream

Using it

# per-episode force estimates ship as npz next to the release
import numpy as np
from force_recovery.dexforce import force_informed_targets, gel_axis
from force_recovery.evaluate import median3_fresh

z = np.load("force_recovery/motherboard/2026-05-10/episode_000_left.npz")
force = median3_fresh(z["force_normal_n"], is_new)      # de-spike on fresh frames
act = force_informed_targets(pose, force, gel_axis("motherboard", "left"))
train_targets = act.target_pos                          # (T,3) — drop-in pose labels

Caveats, stated plainly: absolute newtons carry the FEATS-calibration uncertainty (cross-sensor scale drifts 2–4×; within-episode relative force is the reliable part), shear-dominant contact is a blind spot of the normal-force estimator, and legacy recordings update tactile at ~8.5 fps — the tactile_*_is_new flags mark which rows carry fresh force evidence.