relben API

Guide: RelBench Utilities

Catalog

class relben.EvalTask(database: 'str', table: 'str', target: 'str', task_type: 'str')
Parameters:

  • database (str)

  • table (str)

  • target (str)

  • task_type (str)

relben.select_tasks(selectors)
Parameters:

selectors (list[str] | None)

Return type:

list[EvalTask]

Submissions

class relben.EvalSample(entity_id, anchor, key, params=<factory>)

One test task row and its official submission key.

Parameters:

  • entity_id (Any)

  • anchor (datetime)

  • key (dict[str, Any])

  • params (dict[str, Any])

relben.write_submission(path, target_column, rows)

Write one keyed submission CSV: key columns plus the prediction.

Every row must share the same key columns; a mismatch raises ValueError because the official scorer joins on them. Returns the number of rows written.

Parameters:

  • path (Path)

  • target_column (str)

  • rows (Iterable[tuple[Mapping[str, Any], float]])

Return type:

int

relben.write_manifest(output_dir, *, runner, tasks, extra=None)

Record which tasks a runner emitted, beside its submission CSVs.

Parameters:

  • output_dir (Path)

  • runner (str)

  • tasks (list[str])

  • extra (Mapping[str, Any] | None)

Return type:

Path

relben.atomic_json(path, value)

Write JSON through a temp file so readers never see a partial record.

Parameters:

  • path (Path)

  • value (dict)

Return type:

None

Reports

relben.write_report(output, results, runners, *, baseline=None)

Write results.json and a markdown score matrix to output.

Each result row carries task, runner, metric, and value, plus optional valid. Gains against baseline use runner - baseline for metrics in the higher-is-better set and the reverse for everything else, so positive always means the runner improved on the baseline.

Parameters:

  • output (Path)

  • results (list[dict])

  • runners (list[str])

  • baseline (str | None)

Return type:

None

Hurdle Gates

relben.nmae(predictions, labels, *, train_std)

Mean absolute error normalized by the training-split label std.

Parameters:

train_std (float)

Return type:

float

relben.tune_hurdle_gate(regression_predictions, existence_probabilities, labels, *, grid=None)

Search a probability threshold that zeroes predictions for unlikely rows.

Zero-inflated targets often score better when a low existence probability forces the prediction to zero. The search compares gated MAE against the ungated baseline over a validation split and reports whether gating won.

Returns (best_mae, best_threshold, use_gate).

Return type:

tuple[float, float, bool]