Recovering Force-Related Actions for the React Tactile Dataset

React records GelSight-Mini images and OptiTrack poses — but, like all UMI-style data, no applied force: demonstrated pose equals achieved pose, so replaying the actions presses with zero intended force. Two methods put it back: tactile-estimated normal force, and DexForce-style force-informed action targets.

GelSight Mini (markerless gel) no F/T sensorno force ground truth dataset: yxma/React → method in one page → action processing deep-dive

What was tried first — and why it failed

FEATS transfer: a negative result

The plan matched FARM (arXiv:2510.13324): run FEATS (U-net, GelSight-Mini images → FEA-supervised force distributions, MAE < 1 N) offline over the recordings. It does not transfer: FEATS is trained on marker-dot gel, and the React sensors use markerless gel — different texture, different illumination profile.

training sensor vs our sensor

Left: FEATS training sensor (63 marker dots). Right: React sensor, markerless. On this input the network returns its no-contact output for every frame — the strongest-contact frame of pushT/episode_000 predicts −0.040 N total, identical to a free frame (−0.042 N). Dot-based per-sensor calibration is impossible for the same reason.

Method 1 — Photometric-stereo depth → Winkler normal force

GelSight frame
640×480 RGB
per-pixel MLP
(gsrobotics, markerless)
surface normals Poisson (DCT)
integration
background plane
removal (robust fit)
indentation δ(x,y) [mm] Winkler foundation
F = c · Σδ · dA

The gel pad is modelled as a bed of independent springs (thin-elastic-layer approximation); the constant c is fitted on FEATS ground truth rather than assumed (see the external-validation section). Per-sensor calibration uses the 15 lowest-intensity frames of each episode: median depth becomes the zero map and the MAD of the plane-removed residuals sets a 5σ contact threshold — a fixed threshold cannot work because reconstruction noise differs per sensor (measured ~10 µm after plane removal). Each frame additionally gets a robust background-plane fit, which removes the frame-to-frame illumination tilt that otherwise dominates integrated depth.

raw | diff | depth

Strongest contact of motherboard/episode_000 (left sensor): raw frame | difference vs reference | reconstructed depth. Indentation localises exactly where the image difference shows contact.

depth panels pushT depth panels motherboard

Evaluation (no ground truth → falsifiable properties instead)

E1.1 specificity — no-contact rows must read ≈0 N; SNR = median force on high-intensity rows / p95 on reference rows. E1.2 correlation — Spearman ρ against contact intensity, an independent statistic of the same images. E1.3 spikes — single-frame excursions are solver noise; a median-3 over fresh frames must remove them. E1.4 calibration stability across episodes.

per-episode table — 72 sensor-sides (click to expand)
episodesidefresh rowsmax F [N]SNRρ vs intensityspike raw→filt [%]threshold [µm]
2026-05-10/episode_000left20107.632732732.80.911.4 → 0.010
2026-05-10/episode_000right20076.3952601.40.831.1 → 0.010
2026-05-10/episode_001left21579.46625.60.831.4 → 0.010
2026-05-10/episode_001right21558.191295176.40.750.9 → 0.010
2026-05-10/episode_002left30379.92750.60.771.0 → 0.010
2026-05-10/episode_002right302514.557284.30.791.4 → 0.010
2026-05-10/episode_003left26295.22868.30.810.1 → 0.010
2026-05-10/episode_003right26293.202.00.820.5 → 0.010
2026-05-10/episode_004left6903.680.60.680.3 → 0.010
2026-05-10/episode_004right6924.222.10.890.3 → 0.010
2026-05-10/episode_005left2202.851.40.800.5 → 0.010
2026-05-10/episode_005right2213.4717.30.750.0 → 0.010
2026-05-10/episode_006left11532.5963549.30.870.1 → 0.010
2026-05-10/episode_006right11524.371608350.80.940.0 → 0.010
2026-05-10/episode_007left5562.250.80.680.2 → 0.010
2026-05-10/episode_007right5563.051334479.90.900.0 → 0.010
2026-05-10/episode_008left3762.94235.50.920.0 → 0.010
2026-05-10/episode_008right3772.313.00.830.3 → 0.010
2026-05-10/episode_009left5234.30108.20.880.8 → 0.010
2026-05-10/episode_009right5266.293279941.60.921.1 → 0.010
2026-05-10/episode_010left3004.135.20.900.7 → 0.010
2026-05-10/episode_010right2992.553.10.732.0 → 0.010
2026-05-10/episode_011left6359.0867.10.880.0 → 0.010
2026-05-10/episode_011right6362.214213.50.870.2 → 0.010
2026-05-11/episode_003left27969.8939.60.811.8 → 0.010
2026-05-11/episode_003right280414.902818249.90.792.0 → 0.010
2026-05-11/episode_004left73511.18766.60.761.5 → 0.010
2026-05-11/episode_004right73610.162624701.50.781.1 → 0.010
2026-05-11/episode_005left403810.273623773.10.871.1 → 0.010
2026-05-11/episode_005right403610.442970277.30.831.1 → 0.010
2026-05-11/episode_006left336111.1717.80.860.7 → 0.010
2026-05-11/episode_006right336412.862791.20.821.0 → 0.010
2026-05-11/episode_007left19239.298665.40.820.4 → 0.010
2026-05-11/episode_007right19238.2612200.40.841.4 → 0.110
2026-05-11/episode_008left382713.6736170.00.791.1 → 0.110
2026-05-11/episode_008right382413.063973363.40.851.4 → 0.010
2026-05-11/episode_009left12138.481427.00.840.8 → 0.010
2026-05-11/episode_009right121212.9881585.40.911.6 → 0.010
2026-05-11/episode_010left19034.661923.10.760.8 → 0.010
2026-05-11/episode_010right189617.104043420.80.871.8 → 0.110
2026-05-11/episode_011left21338.5117.80.830.8 → 0.010
2026-05-11/episode_011right212710.013374461.20.861.9 → 0.010
2026-05-11/episode_012left413111.213452.40.881.6 → 0.010
2026-05-11/episode_012right413011.363865536.70.871.6 → 0.010
2026-05-11/episode_013left34991.842.10.710.2 → 0.010
2026-05-11/episode_013right35082.65691976.00.850.8 → 0.010
2026-05-11/episode_014left6751.57394078.50.870.3 → 0.010
2026-05-11/episode_014right6792.53224.00.860.6 → 0.010
2026-05-11/episode_015left16501.80468833.10.850.3 → 0.010
2026-05-11/episode_015right16475.031496452.70.883.0 → 0.010
2026-05-11/episode_016left32692.2270.20.830.9 → 0.010
2026-05-11/episode_016right32524.411317693.80.841.9 → 0.010
2026-05-11/episode_017left398021.552568.40.811.5 → 0.110
2026-05-11/episode_017right395422.7688.90.881.4 → 0.012
2026-05-19/episode_000left5187.579.90.943.1 → 0.210
2026-05-19/episode_000right50611.466680776.60.924.0 → 0.210
2026-05-19/episode_001left2517.861.80.914.0 → 0.026
2026-05-19/episode_001right24610.061.50.922.0 → 0.010
2026-05-19/episode_002left5907.091385.50.945.3 → 0.010
2026-05-19/episode_002right5949.8020.60.914.6 → 0.210
2026-05-19/episode_003left294.561.10.180.0 → 0.027
2026-05-19/episode_003right314.924724913.10.9210.3 → 3.470
2026-05-19/episode_004left354.001.20.793.0 → 0.026
2026-05-19/episode_004right415.621.70.942.6 → 0.010
2026-06-18/episode_000left6891.84152355.90.750.4 → 0.110
2026-06-18/episode_000right6933.72772015.00.861.4 → 0.010
2026-06-18/episode_001left40997.10304962.00.800.5 → 0.010
2026-06-18/episode_001right41535.81556445.10.780.7 → 0.010
2026-06-18/episode_002left39564.20315086.30.790.5 → 0.010
2026-06-18/episode_002right39945.58731457.70.831.0 → 0.010
2026-06-18/episode_003left40213.79191250.30.760.2 → 0.010
2026-06-18/episode_003right40655.12583764.90.830.7 → 0.010
force timeline pushT force timeline motherboard

12 s around each episode's force peak: live force bar under the tactile stream (median-3 filtered). Left: pushT (light, intermittent contact). Right: motherboard (sustained presses).

External validation — FEATS ground truth

React has no force ground truth, but FEATS (arXiv:2411.03315) does: a GelSight Mini pressed by a CNC machine against 24 indenters, with FEA-derived force labels. Running our estimator on its val split is a transfer test — marker-dot gel, different pad, single global no-contact reference — i.e. strictly harder than the React setting, where reference frames come from the same episode seconds away.

FEATS validation scatter FoTa validation strip plot three estimators vs F/T ground truth (cnc_Mini) GlowTact F/T ground truth comparison
metricvaluescope
Spearman ρ (pooled)0.74 normal loading, ≤30 N, n=150
Pearson r (pooled)0.65
MAE after scale calibration4.2 N
ρ within one indenter family0.66–0.94zelda 0.94, cylinder 0.75, triangle 0.70
ρ on unseen indenter shapes0.85test_unknown_indenters split, n=100 — shape generalization holds
ρ vs press depth on FoTa (T3)0.43 / 0.2461 captures, third-party Panda rig, household objects; markerless / marker gel medians, 84% positive. FoTa has no force GT, so pose press-depth is the (attenuated) monotone proxy
ρ vs F/T ground truth — FoTa cnc_Mini0.43 / 0.65400 frames, CNC gantry + F/T sensor, markerless Mini (pooled / edge-filtered, after flat-field optimization). FeelAnyForce on identical frames: 0.83 / 0.92; FEATS U-net: 0.07
ρ vs F/T ground truth — GlowTact0.63600 of 14,715 cleaned markerless-Mini presses, 0–20 N; per-family up to 0.93. FeelAnyForce: 0.90. Third independent scale estimate (2.61 N/mm³) inside the 2–4× band
scale stability across splits2–4×fitted N/mm³ varies 1.9–7.2 between sessions/sensors: within-episode relative force is reliable, cross-sensor absolute scale is not
ρ on shear-loaded captures−0.15excluded & documented: shear churns the image without adding indentation volume
fitted scale2.01 N/mm³ 43× the assumed Winkler constant — the fit absorbs the depth-unit and gel-constant assumptions; React forces are reported in FEATS-calibrated newtons

What the validation bought. Chasing the initial ρ = 0.42 exposed two pipeline defects that also affected React inference: a global illumination tilt dominating the integrated depth (fixed by a robust background-plane fit per frame — React contact thresholds tightened from 50 µm to 10 µm) and marker-dot imprints from the SDK's mask-and-interpolate path (fixed by inpainting the dots before the network). A quadratic background model was also tried and rejected — it absorbed real contact (ρ 0.61→0.50). Sanity check on scale: the strongest motherboard press reads ~0.2 N under the theoretical constant but ~7 N calibrated — the latter is what pressing a connector home actually takes.

Method 2 — DexForce-style force-informed position targets

Rather than adding force as a new action dimension, the estimated normal force becomes a virtual position target past the contact surface (DexForce, arXiv:2501.10356):

ptarget = pobserved + (F̂n / k) · n̂,    k = 1500 N/m (deployment impedance)

The action stays a pose — it composes with the existing 30 Hz pose actions, needs only an impedance controller at deployment, and in free space reduces exactly to the observed pose. The pressing direction n̂ is not a guessed axis: it is the rig's dual-ball calibrated gel_axis_in_rigid (pose-to-pose consistency ≈1°), rotated into world frame per row. The naive [0,0,1] guess produced negative approach alignment at force onsets; the calibrated axis makes it positive.

virtual target offsets pushT virtual target offsets motherboard

Evaluation

E2.1 free-space invariance — zero force must leave the action exactly the observed pose. E2.2 boundedness — F/k must stay millimetre-scale to be a safe action. E2.3 roundtrip — k·‖target − pose‖ must reproduce the input force (pure algebra). E2.4 geometry — motion should align more with n̂ while pressing than in free motion.

per-episode table — 72 sensor-sides (click to expand)
episodesidefree-space max offset [m]penetration p50 / max [mm]roundtrip err [N]v·n̂ pressing / free
2026-05-10/episode_000left0.0e+001.01 / 5.096.3e-14+0.127 / -0.004
2026-05-10/episode_000right0.0e+000.93 / 4.265.5e-14+0.038 / +0.025
2026-05-10/episode_001left0.0e+001.24 / 6.316.9e-14+0.147 / +0.079
2026-05-10/episode_001right0.0e+001.10 / 5.465.9e-14+0.220 / +0.082
2026-05-10/episode_002left0.0e+000.91 / 6.624.6e-14+0.162 / +0.076
2026-05-10/episode_002right0.0e+001.11 / 9.706.0e-14+0.074 / +0.082
2026-05-10/episode_003left0.0e+000.57 / 3.484.0e-14-0.025 / +0.073
2026-05-10/episode_003right0.0e+000.38 / 2.136.3e-14+0.028 / -0.001
2026-05-10/episode_004left0.0e+000.66 / 2.452.1e-14-0.017 / +0.040
2026-05-10/episode_004right0.0e+000.63 / 2.824.3e-14-0.028 / -0.178
2026-05-10/episode_005left0.0e+000.86 / 1.902.1e-14-0.063 / -0.015
2026-05-10/episode_005right0.0e+000.67 / 2.313.3e-14-0.033 / +0.007
2026-05-10/episode_006left0.0e+000.68 / 1.732.9e-14-0.024 / +0.093
2026-05-10/episode_006right0.0e+000.52 / 2.913.3e-14+0.034 / -0.009
2026-05-10/episode_007left0.0e+000.53 / 1.503.4e-14-0.011 / +0.189
2026-05-10/episode_007right0.0e+000.66 / 2.032.9e-14-0.006 / -0.052
2026-05-10/episode_008left0.0e+000.33 / 1.962.9e-14-0.026 / +0.202
2026-05-10/episode_008right0.0e+000.51 / 1.542.7e-14+0.047 / -0.053
2026-05-10/episode_009left0.0e+000.78 / 2.874.2e-14-0.066 / +0.106
2026-05-10/episode_009right0.0e+001.33 / 4.194.7e-14+0.012 / +0.065
2026-05-10/episode_010left0.0e+000.65 / 2.752.7e-14-0.031 / +0.117
2026-05-10/episode_010right0.0e+000.74 / 1.702.5e-14+0.059 / +0.083
2026-05-10/episode_011left0.0e+000.45 / 6.053.0e-14-0.014 / +0.077
2026-05-10/episode_011right0.0e+000.54 / 1.474.6e-14+0.002 / +0.027
2026-05-11/episode_003left0.0e+001.31 / 6.594.3e-14+0.115 / +0.103
2026-05-11/episode_003right0.0e+001.67 / 9.936.7e-14+0.080 / +0.073
2026-05-11/episode_004left0.0e+002.30 / 7.455.0e-14+0.146 / +0.127
2026-05-11/episode_004right0.0e+001.82 / 6.785.6e-14+0.076 / +0.106
2026-05-11/episode_005left0.0e+001.64 / 6.856.0e-14+0.075 / +0.021
2026-05-11/episode_005right0.0e+001.55 / 6.968.4e-14+0.036 / -0.011
2026-05-11/episode_006left0.0e+001.89 / 7.454.5e-14+0.084 / +0.109
2026-05-11/episode_006right0.0e+002.03 / 8.588.5e-14+0.066 / +0.118
2026-05-11/episode_007left0.0e+001.55 / 6.193.1e-14+0.008 / +0.120
2026-05-11/episode_007right0.0e+001.08 / 5.507.2e-14+0.040 / +0.096
2026-05-11/episode_008left0.0e+001.96 / 9.113.5e-14+0.051 / +0.123
2026-05-11/episode_008right0.0e+001.99 / 8.716.8e-14+0.051 / +0.084
2026-05-11/episode_009left0.0e+000.77 / 5.652.9e-14+0.049 / +0.065
2026-05-11/episode_009right0.0e+001.76 / 8.653.6e-14-0.015 / +0.072
2026-05-11/episode_010left0.0e+000.72 / 3.113.9e-14+0.141 / +0.068
2026-05-11/episode_010right0.0e+001.88 / 11.404.8e-14+0.074 / +0.010
2026-05-11/episode_011left0.0e+001.05 / 5.673.8e-14+0.144 / +0.095
2026-05-11/episode_011right0.0e+001.81 / 6.674.2e-14+0.061 / +0.008
2026-05-11/episode_012left0.0e+001.04 / 7.483.6e-14+0.089 / +0.090
2026-05-11/episode_012right0.0e+001.72 / 7.575.9e-14+0.013 / +0.037
2026-05-11/episode_013left0.0e+000.25 / 1.234.2e-14+0.005 / +0.057
2026-05-11/episode_013right0.0e+000.39 / 1.777.3e-14+0.061 / +0.079
2026-05-11/episode_014left0.0e+000.22 / 1.053.1e-14-0.011 / +0.043
2026-05-11/episode_014right0.0e+000.29 / 1.698.1e-14+0.069 / +0.069
2026-05-11/episode_015left0.0e+000.27 / 1.203.5e-14-0.014 / +0.039
2026-05-11/episode_015right0.0e+000.70 / 3.353.7e-14+0.065 / +0.112
2026-05-11/episode_016left0.0e+000.30 / 1.483.9e-14+0.017 / +0.073
2026-05-11/episode_016right0.0e+000.61 / 2.944.8e-14+0.010 / +0.125
2026-05-11/episode_017left0.0e+003.95 / 14.375.0e-14+0.044 / +0.154
2026-05-11/episode_017right0.0e+004.07 / 15.177.8e-14-0.006 / +0.123
2026-05-19/episode_000left0.0e+001.56 / 5.043.0e-14+0.019 / -0.004
2026-05-19/episode_000right0.0e+003.27 / 7.643.4e-14+0.036 / -0.018
2026-05-19/episode_001left0.0e+001.70 / 5.243.2e-14-0.009 / +0.132
2026-05-19/episode_001right0.0e+002.97 / 6.712.9e-14+0.050 / +0.007
2026-05-19/episode_002left0.0e+001.67 / 4.733.1e-14+0.048 / -0.077
2026-05-19/episode_002right0.0e+001.77 / 6.544.4e-14+0.054 / +0.008
2026-05-19/episode_003left0.0e+001.48 / 3.041.6e-14-0.087 / +0.151
2026-05-19/episode_003right0.0e+002.15 / 3.282.4e-14+0.023 / +0.106
2026-05-19/episode_004left0.0e+001.88 / 2.671.9e-14+0.004 / +0.397
2026-05-19/episode_004right0.0e+002.78 / 3.742.6e-14-0.213 / +0.278
2026-06-18/episode_000left0.0e+000.31 / 1.224.3e-14-0.048 / +0.067
2026-06-18/episode_000right0.0e+000.38 / 2.486.5e-14+0.079 / -0.001
2026-06-18/episode_001left0.0e+000.30 / 4.744.7e-14+0.050 / +0.087
2026-06-18/episode_001right0.0e+000.43 / 3.878.4e-14+0.067 / +0.051
2026-06-18/episode_002left0.0e+000.30 / 2.804.8e-14+0.078 / +0.065
2026-06-18/episode_002right0.0e+000.46 / 3.729.0e-14+0.114 / +0.017
2026-06-18/episode_003left0.0e+000.28 / 2.535.3e-14+0.106 / +0.101
2026-06-18/episode_003right0.0e+000.40 / 3.427.2e-14+0.081 / +0.040

Across all episodes: free-space invariance 0.0e+00 m, roundtrip error 9.0e-14 N (machine precision), max penetration 15.2 mm.

What the numbers say

Debug log (what actually went wrong)

foundfix
FEATS returns its no-contact output on every markerless frame switched force estimation to photometric-stereo depth (negative result kept above)
fixed dot-threshold 55 detects 0 dots (our dots bottom out at gray 56) percentile-based threshold — then made moot by the markerless finding
1.1 mm phantom depth at the image border (Poisson/Neumann edge artifact) exclude a 12/16-px margin from force integration
contact threshold exploded 30× when a reference frame was lightly touching median zero-map + MAD threshold instead of mean + std
4% single-frame force spikes (bad Poisson solves) median-3 over fresh frames only — row-wise filtering would see each duplicated value 3× and keep it
assumed gel normal [0,0,1] gave negative approach alignment at onsets use the rig's dual-ball calibrated gel_axis_in_rigid; sign verified against approach kinematics
fed the full camera view to a depth MLP trained on the SDK's 15%-cropped view same 1/7 border crop for React frames and FEATS images — geometry now identical between inference and validation
FEATS transfer started at ρ = 0.42: illumination tilt dwarfed real indentation, dot imprints leaked into depth per-frame robust background-plane removal + marker inpainting → ρ = 0.70; quadratic background tried and rejected (absorbed contact)
shear-loaded captures anti-correlate (ρ = −0.15) scoped the estimator to normal loading; documented as a limitation (React pushing involves shear — treat high-shear force estimates with caution)
theoretical Winkler scale off by ~40× (depth-unit + gel-constant assumptions) absolute scale fitted on FEATS ground truth; React forces in FEATS-calibrated newtons, uncertainty stated

Data: yxma/React · code ships with the dataset (preprocess/, toolbox/) · references: FARM (2510.13324), DexForce (2501.10356), FEATS (2411.03315), ACP (2410.09309), gsrobotics SDK.