45-EFT.WP.Data.Pipeline v1.0 | Chapter 17 Implementation Binding & Execution API

Chapter 17 Implementation Binding & Execution API

I. Chapter Purpose & Scope

II. Service Surface (Normative)

services:

pipelines.v1:

# Validation & checks

- POST /api/v1/pipelines/validate # schema/topology/cross-volume checks (blocking)

- POST /api/v1/pipelines/lint # run lint rules

- POST /api/v1/pipelines/check_units # metrology/unit consistency

- POST /api/v1/pipelines/verify_references # citation anchor format & reachability

# Planning & execution

- POST /api/v1/pipelines/plan # DAG planning & resource estimation

- POST /api/v1/pipelines/run # trigger execution (sync/async)

- POST /api/v1/pipelines/metrics # runtime metrics (QPS/p99/ρ/cost)

- POST /api/v1/pipelines/lineage # build/query lineage graph

- POST /api/v1/pipelines/replay # replay via inputs_lock

# Artifacts & release

- POST /api/v1/pipelines/hash_artifact # artifact hashing

- POST /api/v1/pipelines/sign_artifact # sign/verify artifacts

- POST /api/v1/pipelines/publish_release # publish

- POST /api/v1/pipelines/revoke_release # revoke

III. Common Request/Response & Auth

request_envelope:

headers:

Authorization: "Bearer <oidc-token> | HMAC <key>:<sig>"

x-eift-idempotency: "<uuid>" # idempotency key (valid ≥24h)

content-type: "application/json"

body:

spec?: { ... } # pipeline spec (YAML/JSON)

options?: {dry_run?: true, strict?: true}

filters?: {run_id?: "<id>", since?: "<ISO8601>", until?: "<ISO8601>"}

artifacts?: [{path, bytes_b64?, sha256?}]

response_envelope:

status: "ok" | "warn" | "error"

errors: [{code, message, path?, see?}]

warnings:[{code, message, path?, see?}]

metrics: { ... } # planning/execution/metrology/cost stats

data?: { ... } # structured payload (DAG/lineage/run details)

version: "pipelines.v1"

security:

auth: "OIDC bearer | HMAC"

tls: "TLS1.2+"

scope: ["validate","execute","metrics","lineage","publish","admin"]

rate_limits:

per_key_per_minute: 120

burst: 60

IV. Normative OpenAPI Excerpt

openapi: 3.0.3

info: {title: "EFT Pipelines API", version: "v1"}

paths:

/api/v1/pipelines/validate:

post:

summary: Validate pipeline spec against schema, topology, and cross-volume constraints

requestBody: {required: true, content: {"application/json": {schema: {$ref: "#/components/schemas/SpecEnvelope"}}}}

responses:

"200": {description: "Result", content: {"application/json": {schema: {$ref: "#/components/schemas/Result"}}}}

/api/v1/pipelines/run:

post:

summary: Execute pipeline (sync/async)

requestBody: {required: true, content: {"application/json": {schema: {$ref: "#/components/schemas/RunRequest"}}}}

responses:

"200": {description: "Run accepted", content: {"application/json": {schema: {$ref: "#/components/schemas/RunResult"}}}}

components:

schemas:

SpecEnvelope: {type: object, properties: {spec: {}, options: {type: object}}}

Result:

type: object

properties:

status: {type: string, enum: [ok, warn, error]}

errors: {type: array, items: {$ref: "#/components/schemas/Issue"}}

warnings:{type: array, items: {$ref: "#/components/schemas/Issue"}}

metrics: {type: object}

data: {type: object}

RunRequest:

type: object

properties:

spec: {}

options: {type: object, properties:{mode:{type:string, enum:["sync","async"]}}}

RunResult:

type: object

properties:

run_id: {type: string}

state: {type: string, enum: ["queued","running","succeeded","failed"]}

dag: {type: object}

Issue:

type: object

properties:

code: {type: string}

message: {type: string}

path: {type: string}

see: {type: array, items: {type: string}}

V. Endpoint Semantics (Essentials)

1. /pipelines/validate (blocking)

• Schema/regex, anchor dependencies, metrology.units="SI"&check_dim=true, layers/edges topology, Σ_out→Σ_in compatibility, frozen splits & leakage guardrails, minimal security (credentials_ref).

2. /pipelines/lint

• Run Chapter 16 lint: topology/splits/idempotency/feature-space/security & compliance minima; return rule-level locations and fix hints.

3. /pipelines/plan

• Produce executable DAG and resource/cost estimation: concurrency/queues/affinity & quotas, estimated QPS/p99/ρ, usd_per_kqps.

4. /pipelines/run

• options.mode:"sync|async"; returns run_id and DAG; async pairs with /metrics polling. Failures include compensation/rollback suggestions.

5. /pipelines/metrics

• Query runtime metrics: qps/latency_ms.{p50,p95,p99}/ρ/net_mbps/size_bytes/cost_breakdown and SLO posture.

6. /pipelines/lineage

• Build lineage.graph (nodes carry versions/hashes) from spec/run history; support diffs and “up/downstream impact”.

7. /pipelines/replay

• Requires inputs_lock; policy:"strict|lenient"; returns byte-level or tolerance-domain conformance and a diff report path.

8. /pipelines/hash_artifact|/sign_artifact

• Compute sha256 and sign/verify; reconcile with export_manifest.artifacts[].

9. /pipelines/publish_release|/revoke_release

• Follow versioning/notice policy: semver, stability_line, notice.type; revoke writes tombstones and updates mirrors/indexes.

VI. Error Codes (Normative)

errors:

- {code:"ESCHEMA001", message:"missing required field", path:"$.pipeline.id", see:["EFT.WP.Core.DataSpec v1.0:EXPORT"]}

- {code:"ESEMVER001", message:"invalid semver", path:"$.pipeline.version"}

- {code:"EDIM001", message:"units must be SI and check_dim=true",path:"$.metrology", see:["EFT.WP.Core.Metrology v1.0:check_dim"]}

- {code:"ETOPO001", message:"Σ_out -> Σ_in schema incompatible", path:"$.pipeline.edges[*]"}

- {code:"ESPLIT001", message:"split ratios must sum to 1±1e-6", path:"$..stages[*].splits"}

- {code:"ELEAK000", message:"cross-split leakage detected", path:"$..policy.leakage_guard"}

- {code:"ESEC001", message:"credentials_ref required", path:"$..stages[?(@.type^='source.')].params"}

- {code:"EPUB001", message:"publish gate not met", path:"$.export_manifest"}

VII. Idempotency, Versioning & Compatibility

idempotency:

header: "x-eift-idempotency"

window_hours: 24

versioning:

api: "pipelines.v1" # breaking change → bump MAJOR

minor: "backward-compatible additions"

compatibility:

request_backward: "minor+patch"

response_fields: "additive only; no removals"

VIII. Security, Audit & Compliance

• Auth: OIDC/HMAC; Transport: TLS1.2+; least privilege by scope.
• Audit: record request_id, idempotency_key, caller, timestamps, summary; logs feed the compliance module and are listed in exports.
• Compliance: regional limits and data-subject rights per Chapter 14; publish/revoke per versioning policy.

IX. Machine-Readable Implementation Snippets (Ixx-? Prototypes)

def validate_pipeline(spec: dict) -> dict: ...

def lint_pipeline(spec: dict, rules: dict) -> dict: ...

def check_units(spec: dict) -> dict: ... # uses Core.Metrology v1.0:check_dim

def verify_references(spec: dict) -> dict: ... # regex + anchor reachability

def plan(spec: dict, resources: dict | None = None) -> dict: ...

def run(spec: dict, mode: str = "async") -> dict: ...

def metrics(run_id: str, since: str | None = None, until: str | None = None) -> dict: ...

def lineage(spec: dict | None = None, run_id: str | None = None) -> dict: ...

def replay(run_id: str, policy: str = "strict") -> dict: ...

def hash_artifact(path: str | bytes) -> dict: ...

def sign_artifact(path: str | bytes, key_id: str) -> dict: ...

def publish_release(spec: dict) -> dict: ...

def revoke_release(tag: str, reason: str) -> dict: ...

X. Example Invocations (Ready-to-use)

# Structure + cross-volume validation

curl -s -X POST https://api.eift.org/api/v1/pipelines/validate \

-H "Authorization: Bearer <token>" \

-H "x-eift-idempotency: 7b7a0b1e-0a21-4f3f-9d0b-3b1e9b1f3c22" \

-H "Content-Type: application/json" \

-d @pipeline.json

# Planning & execution

curl -s -X POST https://api.eift.org/api/v1/pipelines/plan -d @pipeline.json

curl -s -X POST https://api.eift.org/api/v1/pipelines/run -d '{"spec": {...}, "options":{"mode":"async"}}'

# Metrics & lineage

curl -s -X POST https://api.eift.org/api/v1/pipelines/metrics -d '{"filters":{"run_id":"RUN-123"}}'

curl -s -X POST https://api.eift.org/api/v1/pipelines/lineage -d '{"filters":{"run_id":"RUN-123"}}'

XI. Coupling with Export Manifest (Normative)

export_manifest:

artifacts:

- {path:"api/openapi.yaml", sha256:"..."}

- {path:"api/clients/python.tar.gz", sha256:"..."}

- {path:"runs/RUN-123/metrics.json", sha256:"..."}

- {path:"runs/RUN-123/lineage.graph", sha256:"..."}

references:

- "EFT.WP.Core.DataSpec v1.0:EXPORT"

- "EFT.WP.Core.Metrology v1.0:check_dim"

- "EFT.WP.Data.Pipeline v1.0:Ch.11"

XII. Chapter Compliance Checklist

• Blocking endpoints (validate|check_units|verify_references|plan|run) implemented with auth/idempotency/rate limiting.
• Citations use “Volume vX.Y:Anchor” and appear in export_manifest.references[]; no shortcodes or versionless cites.
• Metrology checks active (units="SI", check_dim=true); topology/splits/leakage guardrails/security minima pass lint.
• Publish/revoke comply with versioning policy; OpenAPI/SDK and runtime metrics/lineage artifacts are listed in the export manifest and verifiable.