Cleaning Policy Reference

A cleaning policy is written preset@version, for example default@2026-13. The preset decides which changes may be applied automatically; the version decides which rules and thresholds run. This page lists every step of each version in execution order, with its parameters, and explains every decision id and issue code in the cleaning report. The report's field layout and how to accept or decline decisions are in Cleaning Presets & Report.

1. Policies at a glance

raw does not go through the cleaning service, and its structural check does not depend on the version; see section 7.

2. The versioning promise

• Dated versions are frozen. A released version's rules, thresholds and default parameters are never edited.
• Same input, same output. With the same data, options and policy id, the cleaning result and the decision ids are the same. The one model-dependent step is the minimum-length check, which depends on the selected model's minimum context.
• Rule changes ship as a new version. When the platform default moves to a new version, it is announced; the version it replaces is first marked deprecated and can still be selected explicitly (the response carries a cleaning_version_deprecated warning) until it is removed after the announced date.
• Defaults resolve by layer. The preset and the version are each taken from the first layer that sets them: request > key default > organization default > platform default. The platform default is currently default@2026-13; 2026-10, 2026-11 and 2026-12 are deprecated and will be removed after notice. Key and organization defaults are configured by the platform; there is no self-service setting yet.
• The effective policy is always echoed. Successful responses carry effective_policy with id, preset, version and source (request, key, org or platform: the highest layer that set the preset or the version). Record effective_policy.id to reproduce the same cleaning.

3. Selecting a version and the 2026-11 parameters

Pass version in options.cleaning only when you need a version other than the default 2026-13. The seven parameters below exist from 2026-11 on, so they work under the default too, and also go in options.cleaning:

{
"model": "hora-1.0",
"h": 24,
"freq": "1h",
"series": [
  {"unique_id": "pv-01", "ds": "2026-06-01T00:00:00Z", "y": 0.0},
  {"unique_id": "pv-01", "ds": "2026-06-01T01:00:00Z", "y": null}
],
"options": {
  "cleaning": {
    "preset": "default",
    "impute_method": "auto",
    "nonnegative": "true",
    "bounds": {"lower": 0, "upper": 500},
    "structural_zero": "auto",
    "business_days": "off"
  }
}
}

(series is shortened; a real request needs at least the model's minimum context.)

Long gaps: fill or mask

Cleaning always fills every gap so the report and the cleaned series are complete. For long interior gaps, long_gap decides what the model reads:

• fill (default): the model reads the filled values, exactly as shown in the report. Keep it when the filled values are a reasonable stand-in, for example when the series follows a stable seasonal pattern through the gap.
• mask: the model reads those positions as missing and forecasts from the observed values around them. Consider it when a long gap is an outage, a shutdown or another period whose real values are unknown and should not guide the forecast. mask is mainly meant for long gaps well before your most recent data; when a long gap ends close to the last observed value, keep fill unless a backtest on your data shows otherwise.

Short gaps (up to the short-gap limit) are filled for every model either way. With mask, each entity's model_input[] entry reports long_gap_points (masked long-gap positions in the context handed to the model) and long_gap_handling (how the model actually read them). SequenxAI Omni (hora-1.0), the model the API serves, supports missing-value masking: those positions reach it as missing values and long_gap_handling is "mask". "fill" applies only to a model that cannot mask missing values; it then reads the filled values instead (long_gap_handling: "fill", long_gap_reason: "model_without_mask", and a long_gap_mask_unavailable warning) and the request does not fail. The model the API serves today never takes this path. Billing is the same for both values: masked positions were never observed, so they are not billed input points. The choice is echoed in cleaning_report.policy.long_gap when you send it. Try both on your own data with a backtest before you settle on one.

Validation:

• Under 2026-10, any of these seven parameters (except nonnegative: "auto") returns 422 CLEANING_OPTION_UNSUPPORTED; they are never silently ignored:

{
"error": {
  "code": "CLEANING_OPTION_UNSUPPORTED",
  "message": "Cleaning options require policy version 2026-11",
  "details": {
    "options": ["business_days"],
    "policy": "default@2026-10",
    "requires_version": "2026-11"
  }
}
}

• nonnegative: "true" or "false" under 2026-10 also returns CLEANING_OPTION_UNSUPPORTED, with options and supported in details.
• Invalid values (for example lower greater than upper) return 422 VALIDATION_ERROR.
• A field not listed for options.cleaning returns 422 VALIDATION_ERROR (details.errors[].field names it); it is never ignored.
• An unknown version returns 422 CLEANING_OPTION_UNSUPPORTED; details.allowed lists the versions.
• The raw preset accepts none of these parameters, and no accept / decline.
• freq: "B" is valid only from 2026-11 on; 2026-10 returns FREQ_UNSUPPORTED, and raw returns RAW_INPUT_INVALID.

From 2026-11 on, cleaning_report.policy echoes the resolved parameters: impute_method, structural_zero and business_days echo auto when omitted; seasonal_period, bounds and long_gap appear only when sent. Under 2026-10 the policy object keeps its previous shape. You can preview the parameters for free with /v1/clean.

4. Steps in default@2026-10

Each entity (unique_id) is processed independently, in this order. Position lists in the report hold at most 50 entries; beyond that truncated is true.

4.1 Parse timestamps and time zones

• Accepted timestamp formats are listed in Timestamp formats. When the day/month order cannot be proven from the data, the request fails with AMBIGUOUS_DATE_FORMAT; pass options.ds_format.
• Timestamps with an offset are read with that offset; timestamps without one are read in options.timezone, or in UTC when it is not set (issue naive_timestamps_assumed_utc).
• An unparseable timestamp fails the request with VALIDATION_ERROR (2026-10 reports the first one only).
• Daylight-saving effects are reported, never changed: dst_missing, dst_duplicate, dst_nonexistent_time, mixed_timezone. For hourly or finer data read entirely as UTC, a gap or repeat of exactly one hour on a Sunday between 00:00 and 03:00 UTC in March, April, September, October or November also raises a dst_missing / dst_duplicate hint and a dst_suspected warning.

4.2 Sort

Rows not in time order are stably sorted (decision time.reorder, issue out_of_order).

4.3 Recognize missing values

• Missing-value tokens. Text cells equal to na, n/a, nan, null, none, ?, #n/a, #value! or the empty string (case-insensitive) become null (decision value.tokens).
• A lone dash - or -- may mean 0 or missing: it is read as missing, and reading it as 0 is proposed (decision value.dash_token, proposed).
• Missing-value codes. The candidates are 9999, -9999, 99999, -99999, -999, 999999, 65535 and -32768. A candidate is nulled automatically when it lies outside the range of the other observations by more than the larger of their 1st-to-99th percentile spread and 10 MAD, and makes up at most 5% of the points; otherwise it is only proposed and the value is kept (decision value.sentinel).

The two text rules apply only to text cells (for example an uploaded CSV); in JSON requests y is a number or null, so they normally do not fire.

4.4 Determine the frequency

• A freq you pass is used as is.
• When omitted it is inferred: first as a calendar frequency anchored to months, quarters or years (median spacing of at least about 28 days, and at least 90% of stamps on the same day of month, such as the 1st, the month end or a fixed day); otherwise as the most common spacing, treating spacings within 2% or 1 second of each other as equal.
• A mode share of at least 90% is high confidence; 70% to 90% is low confidence, which raises a freq_low_confidence warning but does not block; below 70% the request fails with FREQ_INFERENCE_FAILED.
• When the second most common spacing makes up at least 25% of all spacings and at least two thirds of the most common one, the frequency is ambiguous and the request fails with FREQ_AMBIGUOUS. Entities inferring different frequencies when freq is omitted also return FREQ_AMBIGUOUS.

4.5 Align to the grid

• Stamps slightly off a slot are snapped to the nearest slot (decision grid.snap, issue timestamp_jitter). Tolerance: 2% of the step for fixed steps (never more than half a step), for example 72 seconds for hourly data; ±12 hours for calendar frequencies.
• Rows beyond the tolerance are dropped (decision grid.off_grid, issue off_grid, warning off_grid_dropped). When more than 10% of an entity's rows are off-grid, the request fails with FREQ_INFERENCE_FAILED.

4.6 De-duplicate

When several rows land on the same slot after alignment, the last one is kept (decision time.duplicates, action keep_last). If the rows carry the same value the class is structural; if the values differ the class is drop and a duplicate_conflict warning is raised. At least 3 conflicting duplicates that also exceed 5% of the entity's rows add an entity_column_suspected issue: several series may be mixed under one unique_id.

4.7 Trim edge nulls

Nulls at the start and end of a series are dropped (decision gap.edge_trim, issue edge_null). An entity with no values at all fails with SERIES_EMPTY. From 2026-12 on, trailing nulls are handled differently; see Fixed forecast origin.

4.8 Model context window

History is never cut to fit the model. An entity longer than the selected model's maximum context is rejected with 422 SERIES_TOO_LONG before cleaning (distinct timestamps) or after it (the regular grid, filled slots included); send only the most recent points. The engine's window.context decision and context_window issue therefore never appear in API responses. The model's limits are in Models & Limits.

4.9 Split at a long gap

• When a gap between observations reaches the threshold, only the latest segment after the last long gap is kept (decision gap.split).
• Threshold: options.gap_split_min_steps when passed (0 disables the split); otherwise the largest of the short-gap limit plus one (4 steps), 3 seasonal cycles, and 5% of the series span.
• Seasonal cycle length in steps: one day for sub-daily data, one week for daily up to weekly steps, one year for calendar frequencies, and 1 for weekly or longer fixed steps.
• The latest segment must cover at least the model's minimum context and, with the automatic threshold, at least 3 seasonal cycles; otherwise nothing is split.
• The split is applied when it drops at most half the observations; beyond that it is only proposed (issue segment_split_skipped) and the gap is filled instead.
• No split runs with clean_series: false.

4.10 Missing-share check

Missing share = (stamps missing from the grid + null values) ÷ grid points, computed before any filling. Below 1% is Healthy, 1% to 10% is Warning, and 10% or more is Critical, which fails with SERIES_DISCONTINUOUS (blocking health_critical). 2026-13 replaces this check; see Missing-data rule.

The health score health.score (0 to 100) = 100 − min(40, 400 × gap share) − min(20, 200 × missing share) − min(15, 5 × conflicting duplicates) − min(15, 100 × outlier share), rounded. A Critical entity scores at most 49.

4.11 Insert empty slots

Stamps missing from the grid are inserted as empty slots (decision grid.regrid, issue gap).

4.12 Detect outliers (flag only, never modified)

• Rolling median / MAD: the window is 1/50 of the points, clamped to 21 to 101 and made odd; where the window holds at least 5 observations and a MAD above 0, points scoring 0.6745 × |x − median| ÷ MAD above 3.5 are flagged (issue outlier, warning outlier_flagged_not_modified).
• Calendar-frequency data spanning at least 3 full yearly cycles first has the median of the same season in the other years subtracted, so a regular yearly peak is not an outlier.
• Outliers reach the model with their original values; metrics.outlier_points_suppressed is always 0.

4.13 Detect flat runs and level shifts (flag only)

• Flat runs flat_run. A run of identical non-zero observations reaching the minimum length is flagged. The minimum is half a seasonal cycle, clamped to 8 to 288 steps for fixed steps, with a floor of 4 steps for calendar frequencies. Series whose observations are all non-negative integers with a 90th percentile of at most 20 (small counts) are not checked.
• Level shifts level_shift. Adjacent windows whose medians differ by a factor of about 1000 (log ratio within 3 ± 0.25) are flagged; this usually means a unit mix-up. The window is 1/4 of the points, clamped to 5 to 48.

4.14 Fill gaps

• Every missing point (inserted slots and nulls) is filled by linear interpolation between its neighbours (decision gap.impute, action linear).
• A run of more than 3 missing steps (the short-gap limit) is still filled, with issue long_gap and warning long_gap_filled.
• The imputed action's gaps[] lists each gap's range, method, tier and reason. In 2026-10 the tier only describes the gap length and does not change the method (P is the seasonal cycle length from section 4.9, n the number of points):

• With clean_series: false nothing is filled; if any repair is needed the request fails with TIMESTAMP_IRREGULAR or SERIES_DISCONTINUOUS (blocking cleaning_required).

4.15 Minimum length

With fewer points than the model's minimum context after cleaning, the request fails with SERIES_TOO_SHORT (blocking too_short_for_model).

5. What default@2026-11 adds

2026-11 runs every step of section 4 in the same order with the same thresholds, and adds rules at these points.

Timestamp parsing (step 4.1)

Every unparseable timestamp is listed in one VALIDATION_ERROR: details.cleaning_report.blocking[0] is unparseable_timestamp, with the request row numbers in rows (at most 50) and up to 5 examples.

Number-format checks (step 4.3)

Text cells that are not plain numbers are reported by kind and stay missing rather than being guessed (read_as: "missing"):

Business-day grid B (step 4.4)

• With business_days: "auto" and freq omitted, daily data meeting all of these conditions uses the business-day grid: at most 1.05 stamps per date on average; no Saturday or Sunday dates; at least 15 weekday dates; a span of at least 21 days; weekday coverage (weekdays with data ÷ weekdays in the span) of at least 90%. Dates are read in the entity's local time (options.timezone, else the most common offset in the data).
• On B, weekends are not gaps and forecast stamps skip them (decision grid.business_days, class structural; issue business_days_pattern, whose count is the number of weekend days skipped). There is no holiday calendar: a missing weekday holiday is still a gap.
• business_days: "on" forces B for daily data (when freq is omitted or daily), and weekend rows are dropped as off-grid; when freq is omitted and the data is not daily, business_days_not_applicable is reported. "off" always keeps calendar days.
• You can also pass freq: "B" directly. Under auto, no business-day detection runs when you pass freq.
• Decline grid.business_days:<unique_id> to keep calendar days; weekends then become gaps.

Real output: 20 consecutive weekdays from Monday 2026-09-07, no freq:

"freq": {"inferred": "B", "used": "B", "source": "inferred",
       "status": "HIGH_CONFIDENCE", "confidence": 1.0, "candidates": []}
{"id": "grid.business_days:fx", "rule": "grid.business_days", "class": "structural",
"status": "applied", "action": "business_day_grid",
"before": {"freq": "D"}, "after": {"freq": "B"}, "reversible": true,
"evidence": {"weekday_dates": 20, "weekend_dates": 0, "span_days": 26,
            "coverage": 1.0, "skipped_weekend_days": 6, "source": "detected"}}
// forecast stamps for h=3: 2026-10-05, 2026-10-06, 2026-10-07 (Mon to Wed)

Under 2026-10 the same data is inferred as calendar days (low confidence), the 6 weekend days become gaps, the missing share exceeds 10%, and the request fails with SERIES_DISCONTINUOUS.

Constant-series check (after step 4.12)

Observations that never change raise a constant_series warning; all zeros raise all_zero_series.

Seasonality test

Each entity's report carries series[].seasonality: {period, source, acf, cycles, accepted, reason}.

• The candidate period comes from the frequency: steps per day for sub-daily data (24 for hourly), steps per week for daily up to weekly steps (7 for daily), 5 for business days, steps per year for weekly or longer fixed steps (52 for weekly), and steps per year for calendar frequencies (12 for monthly). A candidate below 2 means no period.
• A frequency-derived period is accepted when the series spans at least 3 cycles and the lag-one-period autocorrelation (over pairs observed at both ends) is at least 0.3.
• A period given with seasonal_period only needs the series to span 2 cycles.

Per-gap imputation

Each gap gets its own method (impute_method: "auto"):

• pchip: Fritsch–Carlson monotone cubic interpolation through all observations; each segment stays between its two end observations, so fills never overshoot.
• seasonal: the median of the same phase over the observations within 7 cycles on either side, plus the offsets of the gap's two neighbouring observations from their own medians, blended linearly across the gap so the fill joins both ends.
• A forced linear or pchip is used for every gap (reason: method_option). A forced seasonal needs a period spanning at least 2 cycles (autocorrelation is not required) and enough same-phase history; otherwise it falls back to linear with reason seasonal_unavailable (no usable period) or insufficient_history.
• The tier thresholds are those of section 4.14, with P taken from seasonality.period; without a period, every gap longer than 3 steps is T1.
• When gaps use different methods, the gap.impute action is mixed and evidence.methods counts the points filled by each method.

Physical constraints on filled values

Constraints act only on filled values; observed values are never changed. They run in this order:

1. Structural zeros (constraint.structural_zero, action zero). A seasonal phase observed in at least 7 cycles and zero in at least 90% of them (for example solar output at night) keeps its filled values at 0. The series must span at least 7 cycles; if every phase qualifies the series is constant and nothing is done. structural_zero: "off" turns it off.
2. Non-negativity (constraint.nonnegative, action clip_at_zero). See the nonnegative parameter in section 3.
3. Bounds (constraint.bounds, action clip_to_bounds). Clips into the declared bounds.

Each constraint that actually changes a filled value produces a decision (class fabricate, reversible: true) that can be declined by id. The gaps it touched list the rule in gaps[].constraints, and actions[imputed].constraints counts the points each rule changed. Observed values breaking a constraint are only flagged: negative_in_nonnegative (negative observations with nonnegative: "true", or under auto when at least 99% of observations are non-negative and only a few are negative, in which case nothing is clipped) and out_of_bounds.

Real output: hourly solar data over 8 days, day 6 at half output, 14 points missing from 2026-06-06T16:00Z to 2026-06-07T05:00Z (overnight):

"seasonality": {"period": 24, "source": "frequency", "acf": 0.9307, "cycles": 8.0,
              "accepted": true, "reason": "accepted"}
"gaps": [{"range": {"start": "2026-06-06T16:00:00Z", "end": "2026-06-07T05:00:00Z", "points": 14},
        "method": "seasonal", "tier": "T1", "reason": "seasonal_period_accepted",
        "constraints": ["constraint.structural_zero", "constraint.nonnegative"]}]
{"id": "constraint.structural_zero:pv-01", "rule": "constraint.structural_zero",
"class": "fabricate", "status": "applied", "action": "zero",
"before": {"imputed_points": 14}, "after": {"changed": 12}, "reversible": true,
"evidence": {"period": 24, "phases": [0, 1, 2, 3, 4, 5, 6, 18, 19, 20, 21, 22, 23],
            "zero_share": 0.9, "min_cycles": 7}}
{"id": "constraint.nonnegative:pv-01", "rule": "constraint.nonnegative",
"class": "fabricate", "status": "applied", "action": "clip_at_zero",
"before": {"imputed_points": 14}, "after": {"changed": 1}, "reversible": true,
"evidence": {"source": "auto"}}
// fills: 17.1, then 13 points at 0 (night stays at 0)

Under default@2026-10 the same 14 points are filled linearly (tier: T1, reason: no_seasonal_method), and the night-time fills are above 0.

Fixed forecast origin (2026-12)

2026-12 keeps every 2026-11 rule above and changes one thing in step 4.7: the forecast always starts one step after the last ds you send, even when the last y values are null (for example an interval whose value has not arrived yet).

• Trailing nulls after the last observation, up to 3 grid steps (the short-gap limit), stay on the grid instead of being trimmed (decision gap.masked_tail, class fabricate, action mask_for_model; issue and warning masked_tail; action masked with reason: trailing_null). They are not used as data: they do not count in fed_points, are not billed, and the entity's model_input[] entry reports them as masked_tail with a trailing_values_missing entry in the response warnings[].
• A longer trailing run, or declining gap.masked_tail:<unique_id>, fails with SERIES_DISCONTINUOUS (blocking trailing_gap_too_long, with count and max_steps) rather than producing a forecast that starts earlier than you asked. Send the newest observed values or drop the trailing empty rows.
• Under strict, gap.masked_tail is demoted to a proposal like other non-structural decisions; accept it by id.
• Leading nulls are still trimmed as in step 4.7. Under 2026-10 and 2026-11 trailing nulls are trimmed too (edge_null) and the forecast starts after the last non-null y.

Missing-data rule (2026-13)

2026-13 keeps every 2026-12 rule and replaces the missing-share check of step 4.10. Earlier versions refuse any series with 10% or more missing; 2026-13 looks at where values are missing instead, because a long gap just before the forecast start hurts the forecast far more than the same number of scattered missing values. Gap lengths are measured in seasonal periods of the series: seasonal_period when you pass it, otherwise 24 steps for hourly data, 48 for 30-minute, 96 for 15-minute, 7 for daily, 52 for weekly and 12 for monthly. Trailing nulls kept for the model (masked_tail) do not count.

Without a seasonal period (for example an irregular frequency) only the share is checked. A refused entity keeps blocking code health_critical and adds reason, reasons, period, near_gap_steps and far_gap_steps; caused_by works as before. Send the missing recent values, or only the history before the gap.

health.factors lists what the health score deducted, as {code, points, count?, ratio?} with the points summing to 100 − score: gap_near_origin, far_gap, missing_share (together at most 49 points on an accepted entity), conflicting_duplicates, outliers (at least 1 point per flagged entity), flat_runs, level_shifts, and blocked (keeps a refused entity at 49 or below). Tier follows the table: refused is Critical, a warning is Warning.

Change from 2026-12. Some series that 2026-12 refused because 10% or more was missing are now accepted (with the warning missing_pattern when 20% or more is missing), and some series with a long gap just before the forecast start, which 2026-12 accepted while less than 10% was missing, are now refused with reason: gap_near_origin_too_long. To keep the old behaviour for a while, pin options.cleaning.version: "2026-12" (deprecated).

6. The strict preset

strict runs every rule of default for the same version, then keeps only structural decisions: every other applied decision becomes proposed, and the request fails with 422 and the report:

• If a demoted decision is a time.* or grid.* rule the request fails with TIMESTAMP_IRREGULAR, otherwise with SERIES_DISCONTINUOUS (blocking cleaning_required).
• A decision id listed in accept is applied as usual; once every demoted decision is accepted the request goes through.
• Under strict the report's actions still describe what default would have done; the decisions statuses are authoritative.

7. The raw preset

raw@2026-10 does not clean: no sorting, de-duplication, filling or time-zone inference, only a structural check before the data goes to the model as sent. freq is required and must be a fixed step; each ds must be exactly YYYY-MM-DDTHH:MM[:SS[.ffffff]] followed by Z or ±HH:MM; each entity must be strictly increasing at a step equal to freq; every y must be finite. Otherwise the request fails with RAW_INPUT_INVALID. The response has no cleaning_report, and accept / decline and cleaning parameters are not accepted. Details are in the raw preset.

8. Reading the result: effective_policy, decisions, accept and decline

• effective_policy.id is the policy that actually ran; cleaning_report.policy holds every parameter the cleaning service resolved (short-gap limit gap_fill_max_steps: 3, outlier threshold outlier_threshold: 3.5, duplicate handling duplicate_policy: "last" and so on), plus the accept / decline you sent.
• cleaning_report.decisions[] lists every data-changing decision and every proposal that was not applied. The id format is rule:unique_id; value.sentinel adds the value as a selector, for example value.sentinel:meter-001:-9999.
• status: applied changed the data; proposed left it untouched; declined was reverted by you.
• accept applies a proposal: value.sentinel, value.dash_token (read as 0), gap.split, and any decision demoted by strict.
• decline reverts a decision with reversible: true: value.sentinel, gap.split, grid.business_days, constraint.structural_zero, constraint.nonnegative, constraint.bounds. Declining gap.masked_tail does not trim the tail; it fails the request with trailing_gap_too_long.
• Do not list the same id in both. An id that matches nothing in this data adds a decision_not_applicable warning and the request proceeds.
• issues[] are findings (with positions), warnings[] are non-blocking notices, blocking[] are the reasons for a rejection. When a request is rejected, the error code follows the first blocking reason.

Request examples and before/after for each rule are in Cleaning Presets & Report.

9. Glossary

Decision rules

Every decision demoted by strict can be accepted.

Issue codes issues[].code

Warnings warnings[].code

Blocking reasons blocking[].code

Actions actions[].code

Imputation reasons gaps[].reason

Seasonality verdicts seasonality.reason

Gap tiers gaps[].tier