45-EFT.WP.Data.Pipeline v1.0 | Chapter 9 Sampling, Splits & Distribution

Chapter 9 Sampling, Splits & Distribution

I. Chapter Purpose & Scope

: split definitions & ratios, frozen indices & consistency, stratification and re-sampling/re-weighting, leakage prevention & audits, package building and mirrors/rate/region compliance, integrity checks and the export manifest; ensure consistency with data contracts, Dataset Card frozen splits, Model Card evaluation protocol, the Metrology chapter, and citation anchors.distribution, and splits, samplingFix pipeline specifications for

II. Terminology & Dependencies

III. Fields & Structure (Normative)

stage:

name: "split.package|split.export"

type: "export.splits"

impl: "I16-5.split_package"

inputs: ["<Σ_in_feat>"]

outputs: ["train_pkg","val_pkg","test_pkg"]

splits:

train: {ratio: 0.8, count: null}

validation: {ratio: 0.1, count: null}

test: {ratio: 0.1, count: null}

policy:

sampling:

strategy: "random|stratified|time-based|spatial-tiles|systematic"

strata: [{by:"class|region|snr_bin", buckets: {"A":100,"B":200}}]

weights: {class:"inverse_freq|none"}

leakage_guard: ["per-object","per-timewindow","per-scene"]

freeze_indices: true

distribution:

packaging:

format: "tgz|zip|parquet|zarr"

shard_bytes: 134217728

layout: ["train","validation","test"]

mirrors: ["https://mirror-a.example/foo/","s3://bucket/foo/"]

rate_limit: {mbps: 50}

regional_compliance: ["EU-GDPR"]

checksums:

package: {sha256: "<hex>"}

shards:

- {path:"train-000.tgz", sha256:"<hex>"}

- {path:"train-001.tgz", sha256:"<hex>"}

on_fail: "block|quarantine|skip"

timeout_s: 1800

IV. Sampling Strategies & Stratification Posture

V. Split Definitions & Frozen Consistency

VI. Leakage Prevention & Audit

VII. Distribution & Integrity

VIII. Metrology & Units (SI)

  1. Performance: QPS (1/s), T_inf (ms with {p50,p95,p99}), utilization ρ (—); network net_mbps; package size size_bytes.
  2. metrology:{units:"SI", check_dim:true} is mandatory; normalize units first before composition/aggregation.
  3. If distribution/splits involve path quantities (e.g., T_arr), register delta_form, path="gamma(ell)", measure="d ell", use one of:
    • T_arr = ( 1 / c_ref ) * ( ∫ n_eff d ell )
    • T_arr = ( ∫ ( n_eff / c_ref ) d ell ),
      and pass check_dim.

IX. Machine-Readable Fragment (Drop-in)

layers:

- name: "export"

stages:

- name: "split.package"

type: "export.splits"

impl: "I16-5.split_package"

inputs: ["feat_rows"]

outputs: ["train_pkg","val_pkg","test_pkg"]

splits:

train: {ratio: 0.8}

validation: {ratio: 0.1}

test: {ratio: 0.1}

policy:

sampling:

strategy: "stratified"

strata: [{by:"class", buckets: {"A":520,"B":2100,"C":12380}}]

leakage_guard: ["per-object","per-timewindow"]

freeze_indices: true

distribution:

packaging: {format:"tgz", shard_bytes:134217728, layout:["train","validation","test"]}

mirrors: ["https://mirror-a.example/datasets/foo/","s3://bucket/foo/"]

rate_limit: {mbps: 50}

checksums:

package: {sha256: "…"}

shards:

- {path:"train-000.tgz", sha256:"…"}

- {path:"train-001.tgz", sha256:"…"}

on_fail: "block"

timeout_s: 1800

X. Lint Rules (Excerpt, Normative)

lint_rules:

- id: SPLIT.RATIO_SUM

when: "$.layers[*].stages[?(@.type=='export.splits')].splits"

assert: "abs(train.ratio + validation.ratio + test.ratio - 1) <= 1e-6"

level: error

- id: SPLIT.FREEZE_REQUIRED

when: "$.layers[*].stages[?(@.type=='export.splits')].policy.freeze_indices"

assert: "value == true"

level: error

- id: SPLIT.LEAKAGE_GUARDS

when: "$.layers[*].stages[?(@.type=='export.splits')].policy.leakage_guard"

assert: "contains_any(['per-object','per-timewindow','per-scene'])"

level: error

- id: DIST.PACKAGING_ALLOWED

when: "$.layers[*].stages[?(@.type=='export.splits')].distribution.packaging.format"

assert: "value in ['tgz','zip','parquet','zarr']"

level: error

- id: DIST.CHECKSUMS_PRESENT

when: "$.layers[*].stages[?(@.type=='export.splits')].checksums"

assert: "has_key('package') and len(shards) >= 1"

level: error

- id: METROLOGY.SI_AND_CHECKDIM

when: "$.pipeline.metrology"

assert: "units == 'SI' and check_dim == true"

level: error

XI. Export Manifest & Audit

export_manifest:

version: "v1.0"

artifacts:

- {path:"splits/train.index", sha256:"..."}

- {path:"splits/validation.index", sha256:"..."}

- {path:"splits/test.index", sha256:"..."}

- {path:"packages/train-000.tgz", sha256:"..."}

- {path:"packages/train-001.tgz", sha256:"..."}

- {path:"splits/leakage_report.csv", sha256:"..."}

references:

- "EFT.WP.Core.DataSpec v1.0:EXPORT"

- "EFT.WP.Core.Metrology v1.0:check_dim"

- "EFT.WP.Data.DatasetCards v1.0:Ch.11"

- "EFT.WP.Data.ModelCards v1.0:Ch.11"

XII. Chapter Compliance Checklist