GelSight image → normal force, in one page
Markerless gel, no F/T sensor, no training data from our rig. A physics pipeline with exactly one fitted number.
↖ results matrix overview action transform 中文The pipeline — six steps
nnmini): three-color illumination makes color→normal invertiblePost-processing: a 3-tap median over fresh tactile frames only (duplicated rows would let a row-wise filter count bad values three times). Cuts single-frame spikes from 4–8% to ≈0.
Validation
The pipeline is validated on three force-labeled datasets (FEATS, FoTa cnc_Mini, GlowTact) and cross-checked against two neural estimators on identical frames — every predicted-vs-ground-truth scatter, per dataset, lives on the results page. Short version: physics 0.43-0.74 everywhere; each network 0.90+ in its own gel domain and collapsing outside it.
Optimizing against ground truth
The cnc_Mini force labels turned the pipeline's weak spots into measurable defects, fixed in order (each step verified on held-out data):
| step | evidence that drove it | ρ (held-out val) |
|---|---|---|
| baseline (volume, per-episode zeroing) | — | 0.34 pooled |
| + median zero map over scattered presses | only 4 of 2686 frames are truly contact-free; lowest-force references carried 1.7 N of baked-in contact | ≈ same (zero map wasn't the bottleneck) |
| + flat-field illumination normalization | force-fit residuals correlated with contact position (|ρ| up to 0.6); probe×quadrant conditioning raised ρ 0.45→0.55 — the vignette modulates the depth MLP's gain | 0.44 pooled |
| + edge filtering (contact centroid > 3 mm from border) | border presses sit where the vignette is steepest and imprints clip the sensor edge | 0.65 (probe-median 0.64) |
Volume or max depth? Settled empirically: the volume family wins (vol1.5-weighted 0.65, plain volume 0.63) over max depth (0.63) and clearly over contact area (0.35); a tiny 3-feature linear model matches ρ but halves MAE (0.61 N). The ceiling matters too: even the CNC's own commanded press depth only reaches ρ 0.78–0.88 within a probe — force at equal depth genuinely varies with texture and position.
Gallery
20 image samples (raw | indentation | 3D reconstruction | predicted vs ground-truth force) and 10 React episode clips (tactile | live depth | force trace): browse the full gallery.
NN vs model-based on the GelSight Mini — who has compared them?
No published head-to-head that we could find. The literature runs in two camps that cite but don't benchmark each other. Model-based: marker displacement × elasticity (Yuan 2017), photometric-stereo height + polynomial fit, inverse FEM (GelSlim, Ma 2019). Learned, on the Mini specifically: CANFnet (F/T-labeled, normal only), FEATS (FEA-labeled, 3D distributions), FeelAnyForce (200K ATI-labeled). Each motivates NN over physics qualitatively — FEA too slow for real time, linear elasticity misses elastomer nonlinearity — but their reported baselines are other networks, not the physics pipeline.
Our FEATS experiment is therefore one of the few direct data points: the model-based pipeline reaches ρ 0.70–0.85 with one fitted scalar and zero training frames, where the NNs earn sub-newton MAE in-domain but die outside their gel (FEATS on our markerless gel: no response at all). The trade is portability vs in-domain accuracy — and which one you need depends on whether you can collect labels on your own sensor.