17-EFT.WP.Methods.Imaging v1.0 | Chapter 3 Device and Mode Binding (Sensors / Modalities)

Chapter 3 Device and Mode Binding (Sensors / Modalities)

One-Sentence Goal
Decode heterogeneous imaging devices and modalities into the unified standard schema SRef.img, ensuring metrological, time-base, and path consistency so that downstream processing and audits are fully traceable.

I. Scope & Objects

1. Covered devices & modalities
   • Area/line CMOS/CCD; CFA/monochrome/three-chip; multispectral/hyperspectral; ToF (phase/pulse/gated); polarization cameras; HDR multi-exposure; rolling/global shutter; analog/digital gain chains.
   • Inputs: raw_blob, sensor_meta, optics_meta, timing, env, calib_assets.
   • Outputs:
     SRef.img = { DN, I_lin, masks, mode_axes, tau_map, manifest.imaging.device }.
2. Boundaries
   • Strongly time-varying PSF, non-standard in-house packs, or proprietary compression must supply a decoder and a statement of deviations.
   • Irreversibly processed streams from external ISPs are ingested as “already processed”; applicable steps in this chapter are skipped and the omission is recorded.

II. Terms & Variables

1. Metrology & encoding: DN, bpp, SatMax, black_level, gain, offset, packing ∈ {RAW10, RAW12, RAW14, ...}, endianness ∈ {LE, BE}.
2. Spatial & pixel layout: H, W, C, pitch, orientation ∈ {0,90,180,270}, flip_x, flip_y.
3. CFA & color: cfa_pattern ∈ {RGGB, BGGR, GRBG, GBRG}, S_cfa.
4. Timing & sync: exposure_time, line_time, frame_time, ts, tau_mono, offset/skew/J, rolling/global.
5. Modality specifics
   • ToF: I_cos, I_sin, phi, f_mod, n_eff, T_arr, d_range.
   • Polarization: I0,I45,I90,I135, S0,S1,S2,S3, DoLP, AoLP.
   • Multi/hyperspectral: λ_k, Δλ, I(λ_k).
6. Masks & quality: mask_bad, mask_hot, mask_sat, q_score.
7. Traceability: TraceID, hash_sha256(blob), signature.

*III. Axioms P203- **

• P203-1 (Sufficient metadata): sensor_meta / optics_meta / timing provide the minimal keys; dimensions are checkable; missing items are flagged by m ∈ {0,1}.
• P203-2 (Reversible packing): Bit-packed raw_blob with packing/endianness can be losslessly unpacked to DN ∈ [0, 2^bpp-1].
• P203-3 (Canonical orientation): A deterministic mapping reduces the physical readout orientation to canonical coordinates with orientation=0, flip_x=false, flip_y=false.
• P203-4 (Stable CFA): cfa_pattern is constant during capture and aligned to the pixel origin.
• P203-5 (Linearization convention): I_lin = gain * ( DN - black_level ) is valid in the radiometric linear domain with unit(I_lin)="e-".
• P203-6 (Mappable time base): Rolling-shutter row times satisfy non_decreasing(ts_row); evaluate internally on tau_mono, publish on ts.
• P203-7 (ToF scene coherence): Modulation frequency f_mod, modulation scheme, and demodulation algorithm are uniform within a data block; round-trip propagation assumptions hold.
• P203-8 (Polar sampling orthogonality): Polarization pixels or TDM samples are taken at 0/45/90/135°; if S3 exists, record how it is obtained.
• P203-9 (Monotone spectral axis): λ_k is non-decreasing with Δλ > 0; provide calibration curves and units.
• P203-10 (Two-form arrival in parallel): For ToF, T_arr must include both forms with delta_form ≤ tol_Tarr.
• P203-11 (Traceable): All binding parameters are written to the manifest and signed.

*IV. Minimal Equations S203- **

• S203-1 (Decode): DN = unpack( raw_blob ; packing, endianness ).
• S203-2 (Orientation canonicalization): DN' = orient( DN ; orientation, flip_x, flip_y ).
• S203-3 (Linearization): I_lin = gain * ( DN' - black_level ), mask_sat = [ DN' ≥ SatMax ].
• S203-4 (Rolling-shutter timing map): ts_row = ts_frame + row_idx * line_time; ts = map_tau_to_ts( tau_mono ; offset, skew, J ).
• S203-5 (ToF phase & range): phi = atan2( I_sin, I_cos ), T_arr = phi / ( 2 * π * f_mod ), d_range = ( c_ref / ( 2 * n_eff ) ) * T_arr.
• S203-6 (Two-form arrival gap):
  delta_form = | ( 1 / c_ref ) * ( ∫ n_eff d ell ) - ( ∫ ( n_eff / c_ref ) d ell ) |.
• S203-7 (Polarization Stokes):
  S0 = I0 + I90, S1 = I0 - I90, S2 = I45 - I135,
  DoLP = sqrt( S1^2 + S2^2 ) / S0, AoLP = 0.5 * atan2( S2, S1 ).
• S203-8 (Spectral to tristimulus):
  X = ∑_k I(λ_k) * xbar(λ_k) * Δλ, and likewise for Y,Z.
• S203-9 (Units & dimension check):
  check_dim( I_lin - gain*(DN'-black_level) ) = true.

*V. Mode-Binding Process M30- **

1. M30-1 Device registration: Resolve sensor_id / optics_id / firmware from a registry; complete/validate sensor_meta and units.
2. M30-2 Bitstream decode: Run S203-1 to produce DN and a bpp report.
3. M30-3 Orientation & coordinate canonicalization: Run S203-2 to normalize coordinates; validate a chessboard or equivalent test pattern orientation.
4. M30-4 CFA / modality detection: Read cfa_pattern / mode; for non-CFA modalities, register mode_axes.
5. M30-5 Linearization & saturation mask: Run S203-3; record the provenance and uncertainty of gain / black_level / SatMax.
6. M30-6 Mask construction: Merge bad_pixel_map / hot_pixel_map with mask_sat to obtain mask_bad / mask_hot / mask_sat.
7. M30-7 Time-base mapping: For rolling shutter, compute ts_row and map to ts; persist offset/skew/J.
8. M30-8 Modality-specific derivations:
   • ToF: run S203-5 / S203-6 to obtain phi / T_arr / d_range / delta_form.
   • Polarization: run S203-7 to obtain S0,S1,S2,DoLP,AoLP.
   • Spectral: register λ_k / Δλ / I(λ_k) and optionally run S203-8.
9. M30-9 Validation & signing: Execute the contract library; generate assert_report; write manifest.imaging.device and sign.
10. M30-10 Binding complete: Emit SRef.img for downstream metrology / demosaic / color / HDR pipelines.

VI. Contracts & Assertions

• Metadata & units
  required(sensor_meta.*) ⊆ keys(meta); repair_units(meta) = pass.
• Bitstream & value range
  min(DN) ≥ 0, max(DN) ≤ 2^bpp-1; packing_ok = true.
• Orientation & CFA
  orientation_check(pass) = true; cfa_pattern ∈ {RGGB,BGGR,GRBG,GBRG}.
• Linearization
  check_dim( I_lin - gain*(DN'-black_level) ) = true; mask_sat coverage ≤ sat_cov_max.
• Time base
  non_decreasing(ts_row) = true; | jitter | ≤ J_max.
• ToF
  delta_form ≤ tol_Tarr; d_range ≥ 0 and d_range ≤ d_max_spec.
• Polarization
  S0 ≥ 0; 0 ≤ DoLP ≤ 1.
• Spectral
  non_decreasing(λ_k) = true; Δλ > 0.
• Traceability
  manifest_signed = true and hash_sha256(blob) consistent.

*VII. Implementation Bindings I30- **

• I30-1 decode_raw_blob(raw_blob, packing, endianness, bpp) -> DN
• I30-2 canonicalize_orientation(DN, orientation, flip_x, flip_y) -> DN'
• I30-3 detect_mode_and_cfa(sensor_meta) -> { mode, cfa_pattern, mode_axes }
• I30-4 linearize(DN', gain, black_level, SatMax) -> { I_lin, mask_sat }
• I30-5 build_masks(I_lin, bad_pixel_map, hot_pixel_map, mask_sat) -> { mask_bad, mask_hot, mask_sat }
• I30-6 map_timebase_rs(meta, line_time, frame_time, exposure_time, tau_ref) -> { ts_row, ts, offset, skew, J }
• I30-7 tof_phase_range(I_cos, I_sin, f_mod, n_eff, c_ref) -> { phi, T_arr_forms, d_range, delta_form }
• I30-8 polar_compute(I0, I45, I90, I135) -> { S0, S1, S2, DoLP, AoLP }
• I30-9 bind_spectral_axis(cube, λ_k, Δλ) -> { I(λ_k), λ_axis }
• I30-10 emit_manifest_device(meta, hashes, contracts) -> manifest
• I30-11 assert_contract(ds, tests) -> report

VIII. Cross-References

• Modeling baseline & noise/color/HDR: see this volume’s Chapter 2 (P202-* / S202-* / M20-*).
• Standard inputs & binding; units/dimensions; time axis & arrival: see Methods.Cleaning v1.0, Chapters 3/4/5/6.
• Normalization & probability: see Methods.Cleaning v1.0, Chapter 13.
• Data schemas & keys: see EFT.WP.Core.DataSpec v1.0.
• Streaming execution & SLI/SLO: see EFT.WP.Core.Threads v1.0.

IX. Quality Metrics & Risk Control

• Key metrics
  decode_fail_rate, mask_hot_density, sat_cov, orientation_mismatch, cfa_mismatch, J, delta_form, DoLP_clip_rate.
• Risk handling
  On threshold breaches, roll back to the latest freeze_release(tag_prev), quarantine affected samples, and emit audit records with a postmortem.

Summary
This chapter standardizes device- and modality-level binding: from bitstream decode, orientation canonicalization, linearization and masking, to time-base mapping and modality-specific derivations—paired with contracts and implementation prototypes—so SRef.img can be reliably consumed by downstream optics, color, HDR, and geometry pipelines.