17-EFT.WP.Methods.Imaging v1.0 | Chapter 11 HDR and Exposure Fusion

Chapter 11 HDR and Exposure Fusion

One-Sentence Goal
On a traceable linear radiometric baseline, robustly merge multi-exposure/multi-gain observations—under alignment and de-ghosting constraints—into a high-dynamic-range, scene-referred representation, then map it to the target transfer system (PQ/HLG/legacy OETF) for publication.

I. Scope & Targets

1. Inputs
   • Multi-frame raw observations: { I_k_raw }, k = 1..K (possibly Bayer/multispectral, with differing t_k, ISO_k, ND_k, a_k).
   • Calibration & response: camera response f or inverse LUT LUT_inv, black level D, flat-field and PRNU/DSNU, saturation threshold S_max.
   • Temporal & geometric: ts | tau_mono, rolling/global shutter mode, reference frame index k_ref, alignment model W_k.
   • Noise & gain: read noise sigma_r, electronic gain G_k, net exposure factor K_k = t_k * a_k * G_k * ND_k.
   • Rendering target: dst_cs and OETF ∈ { sRGB, PQ, HLG, GammaX }, display peak or target luminance L_peak.
2. Outputs
   • Scene-referred HDR: E_hat or L_hat (linear irradiance/radiance), optionally OpenEXR/float16.
   • Display-referred image: RGB_out (after tone mapping and OETF), plus optional HDR10/HLG metadata.
   • Artifacts & manifest: hdr_profile.v1, ghost_mask, sat_mask_k, manifest.imaging.hdr, hash_sha256(profile), signature.
3. Applicability
   • Supports exposure bracketing, dual-gain readout, alternating ISO, multi-camera fusion; cross-modality fusion requires prior Chapter 10 color binding.
   • For fast motion and flicker sources, enable de-ghosting and exposure equalization by default; on failure, fall back to the best single frame.

II. Terms & Variables

1. Imaging & physics
   • E(x,y): irradiance; L: radiance; f: camera response; D: black level; S_max: saturation level.
   • K_k = t_k * a_k * G_k * ND_k: net exposure factor; y_k: pixel value; n_k: noise.
2. Estimation & fusion
   • E_hat_k = ( f^{-1}( y_k ) - D ) / K_k: per-frame irradiance estimate.
   • w_k(x,y): weight; sat_mask_k = 1{ y_k ≥ S_thr }; ghost_mask: motion/occlusion mask.
   • W_k: warp from frame k to k_ref; Ω_valid: valid domain after warping.
3. Metrics
   • DR_scene = log2( max(E_hat) / min_pos(E_hat) ); DR_gain = DR_out - DR_ref.
   • ghost_rate = |ghost_mask| / |Ω_valid|; halo_score: edge-halo metric; banding_rate.
   • SNR_k ≈ E_hat_k / sqrt( sigma_r^2 / K_k^2 + sigma_s * E_hat_k / K_k ) (with sigma_s the photon-noise factor).

III. Axioms P211- (HDR Fusion Baseline)*

• P211-1 (Linear-domain fusion): Perform HDR fusion in the linear radiometric domain, after black/flat/PRNU/DSNU correction and f^{-1} linearization.
• P211-2 (Trustworthy metadata): K_k, G_k, t_k, a_k, ND_k must be auditable; when errors exceed thresholds, reject fusion or self-calibrate.
• P211-3 (Alignment first): Align frames on tau_mono; enter fusion only after geometric registration meets thresholds; for rolling shutter, apply row-time correction.
• P211-4 (Saturation exclusion): Pixels with sat_mask_k = 1 are excluded from radiance estimation; prefer short exposures as substitutes.
• P211-5 (De-ghosting): Where ghost_mask = 1, switch to single-frame strategies or robust weighting to preserve causality.
• P211-6 (Noise-aware): Weights and thresholds must explicitly depend on SNR_k; in low-light/high-ISO, cap noise amplification.
• P211-7 (Rendering separation): Tone mapping and OETF are display-referred steps and must not feed back into scene-referred HDR.
• P211-8 (Traceable publication): Merge models, parameters, and masks are shipped in hdr_profile.v1 with signature.

IV. Minimal Equations S211-*

• S211-1 (Observation model)
  y_k = f( K_k * E + n_k ), where n_k includes read and signal-dependent noise.
• S211-2 (Linearization & calibration)
  E_hat_k = ( f^{-1}( y_k ) - D ) / K_k; if RAW is linear, f^{-1} is identity.
• S211-3 (SNR & variance)
  var(E_hat_k) ≈ ( sigma_r^2 + sigma_s * K_k * E ) / K_k^2, hence define w_k ∝ 1 / var(E_hat_k).
• S211-4 (Classical weight families)
  Triangular: w_tri(z) = min(z, Z_max - z); confidence: w_conf = 1 / ( var + eps );
  combined: w_k = (1 - sat_mask_k) * (1 - ghost_mask) * clamp( w_conf, 0, w_max ).
• S211-5 (Robust fusion)
  E_hat = ( ∑_k w_k * E_hat_k ) / ( ∑_k w_k ), with M-estimators (Huber/Tukey) substituting squared loss if needed.
• S211-6 (Alignment & domain)
  I_k' = W_k( I_k ), E_hat_k' = W_k( E_hat_k ), fusion domain Ω = ⋂_k dom(I_k').
• S211-7 (Ghost detection)
  ghost_mask = 1{ MAD( {E_hat_k'} ) > tol_ghost } ∨ 1{ |∇I_k' - ∇I_ref| > tol_edge }.
• S211-8 (Global tone-mapping exemplar)
  T_global(L) = L / ( 1 + L / L_white );
  RGB_out = OETF( M_dst^{-1} * CAT * M_ccm * wb * T_global(E_hat) ).
• S211-9 (Local tone mapping with regularization)
  T_local = L / ( 1 + B_σ * L ), where B_σ is an edge-preserving low-pass (guided/bilateral); regularize with ∥∇T_local - ∇L∥ to avoid halos.

V. Pipeline & Operational Flow M110-*

• M110-1 Readiness: Load { I_k_raw }, metadata, and calibration; run check_dim; compute K_k and saturation threshold S_thr.
• M110-2 Linearize & correct: I_k_lin = flat( prnu( dsnu( f^{-1}( I_k_raw ) ) ) ) - D; preserve pixel phase per CFA.
• M110-3 Temporal & geometric alignment: Align on tau_mono; estimate W_k (pyramidal optical flow / feature matching + RANSAC); apply row-time correction for rolling shutter.
• M110-4 Ghost candidates: From luminance and gradient consistency derive ghost_mask_seed; refine via spatiotemporal connectivity.
• M110-5 Per-frame irradiance: E_hat_k = ( I_k_lin - D ) / K_k; estimate var(E_hat_k) and SNR_k.
• M110-6 Weighting: w_k = g(SNR_k) * (1 - sat_mask_k) * (1 - ghost_mask), clipped to [0, w_max].
• M110-7 Fusion: E_hat = sum( w_k * W_k(E_hat_k) ) / sum( w_k ); suppress outliers with M-estimators.
• M110-8 White balance & color path: In the scene-referred domain, follow Chapter 10 wb → M_ccm → CAT, staying linear.
• M110-9 Tone mapping & OETF: Choose T_global / T_local per target system, then apply OETF ∈ { PQ, HLG, sRGB }.
• M110-10 Quality assessment: Compute DR_scene, ghost_rate, halo_score, banding_rate, and edge-detail metrics.
• M110-11 Contracts & fallback: If ghost_rate > tol_ghost or halo_score > tol_halo, downgrade to single-frame or local-enhancement-only.
• M110-12 Persistence & signing: Output E_hat / RGB_out, ghost_mask, sat_mask_k, hdr_profile.v1, and manifest.imaging.hdr.

VI. Contracts & Assertions

• assert linear_merge_domain: f^{-1} and flat/PRNU/DSNU are done; unit(E_hat_k)="a.u."; dim(E_hat_k) consistent.
• assert meta_consistency: var(K_k)/mean(K_k) ≤ tol_exposure_var; otherwise self-calibrate or reject.
• assert align_quality: EPE ≤ tol_align_epe and inlier_ratio ≥ tol_inlier.
• assert saturation_exclusion: sat_mask_k sufficiently covers highlights; spill_rate ≤ tol_spill.
• assert ghosting: ghost_rate ≤ tol_ghost.
• assert halo_banding: halo_score ≤ tol_halo and banding_rate ≤ tol_banding.
• assert dr_gain: DR_gain ≥ tol_dr_gain, or record cause (intrinsically low-DR scene).
• assert render_binding: OETF, T, M_ccm/CAT/M_dst hashes match hdr_profile.v1.

VII. Implementation Bindings I110-*

• I110-1 inv_response(raw, LUT_inv) -> lin
• I110-2 calibrate_flat_prnu_dsnu(lin, calib) -> lin'
• I110-3 build_exposure_factor(meta) -> { K_k }
• I110-4 align_multi_exposure(frames, mode) -> { W_k }, metrics
• I110-5 estimate_noise_params(frames, meta) -> { sigma_r, sigma_s }
• I110-6 estimate_irradiance(frame_lin, K_k, D) -> E_hat_k
• I110-7 make_weights({ E_hat_k }, { sat_mask_k }, ghost_mask, SNR) -> { w_k }
• I110-8 robust_merge({ E_hat_k }, { w_k }, { W_k }) -> E_hat
• I110-9 tone_map(E_hat, mode, params) -> E_tone
• I110-10 encode_oetf(E_tone, oetf, L_peak) -> RGB_out
• I110-11 evaluate_hdr_metrics(E_hat, masks) -> { DR_scene, ghost_rate, halo_score, banding_rate }
• I110-12 emit_hdr_profile(params, hashes) -> hdr_profile.v1
• I110-13 bind_manifest_hdr(profile, metrics, masks) -> manifest.imaging.hdr

VIII. Cross-References

• Radiometric metrology & calibration harmonization: Chapter 4—ordering and units for f^{-1}, flat, PRNU/DSNU.
• Noise modeling & denoising: Chapter 7—priors for w_k and robust merge.
• Flat/dark & fixed-pattern noise: Chapter 8—avoid stacking pattern residues across exposures.
• Geometric calibration & registration: Chapter 9—foundation for W_k and ghost_mask.
• Color management: Chapter 10—perform HDR white balance and color-space transforms in the scene-referred domain.
• Time axis & sync cleansing: Methods.Cleaning v1.0, Chapter 5—align multi-exposure stacks on tau_mono and record jitter.
• Data spec: EFT.WP.Core.DataSpec v1.0—field definitions and signatures for hdr_profile.v1 and manifest.imaging.hdr.

IX. Quality Metrics & Risk Control

1. Indicators & thresholds
   • DR_scene, DR_gain, ghost_rate, halo_score, banding_rate, edge_acutance_change, artifact_rate.
   • Runtime monitoring: within window Delta_t, track drift_exposure_meta, drift in w_k distributions, spikes in ghost_rate.
2. Key risks & playbooks
   • Flicker/LED PWM: exposure inconsistency across frames → exposure-equalization regression or short-exposure-only fallback.
   • Fast motion/occlusion: ghosting and mismatches → strengthen ghost_mask, locally replace with single frames.
   • Highlight spill & reverse interpolation: bright-edge artifacts → sub-pixel alignment + guided-filter tone mapping.
   • Response mismatch: f^{-1} bias → online CRF refinement or gray-card self-calibration.
   • Resources & latency: in streaming, constrain thr/chan flow (see Threads); if needed, down-order weights and process in tiles.

Summary
This chapter delivers an executable HDR fusion loop: inv_response → calibrate → align/ghost → per-frame irradiance → SNR-aware weights → robust merge → color binding → tone/OETF. With DR_scene / ghost_rate / halo_score as core contracts and explicit fallback paths, the pipeline—published with hdr_profile.v1 and manifest.imaging.hdr—ensures consistent, auditable HDR across devices and scenes.