User Guide

ASTRA provides a command-line interface (CLI) for benchmarking and comparing machine learning models using statistical testing.

Benchmarking Models

Use astra benchmark to run performance benchmarks across different models:

astra benchmark <data>

where data is the path to a dataset (CSV, pickle, or parquet) containing input features, the target variable, and a fold column. See here for an example.

Tip

You can specify some or all options via a YAML configuration file:

astra benchmark --config <config.yml>

See here for the format. Arguments in the config file take precedence over CLI flags.

Option	Type	Default	Description
--name	str	data filename	Name of the experiment. Results are saved in results/<name>/. Used to load cached results if they exist.
--features	str	Features	Name of the column containing the input features.
--target	str	Target	Name of the column containing the target variable.
--run_nested_CV	bool	False	Run nested CV with hyperparameter tuning for the best models.
--use_optuna	bool	False	Use Optuna for hyperparameter tuning instead of grid search.
--n_trials	int	100	Number of trials for Optuna hyperparameter search.
--timeout	int	3600	Time limit in seconds for Optuna hyperparameter search.
--fold_col	str | list	Fold	Name(s) of the 0-indexed fold column(s). Providing a list enables nxk-fold CV (n repeats × k folds).
--main_metric	str	R2	Main metric for model selection. Determines whether the task is regression or classification.
--sec_metrics	list[str]	MSE, MAE	Secondary metrics to report alongside the main metric.
--parametric	str	auto	Whether to use parametric statistical tests. auto checks assumptions and selects the appropriate test automatically.
--impute	str | float | None	None	Missing value imputation strategy. Choices: mean, median, knn, or a numeric constant.
--remove_constant	float | None	None	Remove features whose variance falls below this threshold.
--remove_correlated	float | None	None	Remove features whose correlation with another feature exceeds this threshold.
--scaler	str | None	None	Scale features before training. Choices: Standard, MinMax.
--n_jobs	int	1	Number of parallel jobs for hyperparameter tuning.

Note

Metrics are not case-sensitive. See here for all available metrics.

Warning

nxk-fold CV (multiple --fold_col values) does not currently support nested CV or final hyperparameter tuning.

By default, ASTRA benchmarks all implemented classification or regression models over default hyperparameter grids. Custom models, starting hyperparameters, and search spaces can be specified in the configuration file.

The benchmark creates the following files under results/<name>/:

• default_CV.pkl — CV scores for all models with default hyperparameters.

• nested_CV.pkl — CV scores for all models with optimised hyperparameters (nested grid search).

• final_CV.pkl — CV scores for the final model with optimised hyperparameters.

• final_CV_hparam_search.csv — Full hyperparameter search results.

• final_model.pkl — Best model refit on the full dataset.

• final_hyperparameters.pkl — Optimal hyperparameters.

• benchmark.log — Log of the benchmarking process.

Tip

All .pkl files can be loaded with Python's built-in pickle.load() or with pandas.read_pickle().

Comparing Models

Use astra compare to statistically analyse benchmark results:

astra compare <CV_results_path>

where CV_results_path is either a list of paths to directories each containing final_CV.pkl, or a single directory containing multiple *final_CV.pkl files. The model name is inferred from the parent directory name (list mode) or from the filename prefix (single directory mode).

Option	Type	Default	Description
--main_metric	str	—	The main metric to use for comparison.
--sec_metrics	list[str]	—	Secondary metrics to report.
--parametric	str	auto	Whether to use parametric statistical tests. auto checks assumptions and selects the appropriate test automatically.