astra.model_selection

This module contains functions for model selection and evaluation.

Functions

check_assumptions(results_dict[, verbose])	Check homogeneity of variances and normality assumed by parametric statistical tests.
check_best_model(results_dic, ...[, ...])	Check if there is a model that is significantly better than the others, only counting pairwise wins that are statistically significant (p < 0.05) and practically meaningful (Cohen's d >= min_effect_size).
check_pareto_dominant(results_dict, ...[, ...])	Find a model that is Pareto-dominant across all metrics (not significantly worse than any other model on any metric, significantly better than at least one other model on at least one metric, using p < 0.05 and Cohen's d >= min_effect_size).
corrected_ttest(a, b[, n_folds])	Nadeau-Bengio corrected repeated cross-validation t-test.
find_n_best_models(results_dic, metric[, ...])	Find the n best models that don't perform significantly differently with respect to a given metric as determined using repeated measures ANOVA (if parametric=True) or the Friedman test (if parametric=False).
get_best_hparams(model_class, df, ...[, ...])	Get the best hyperparameters for a model using grid search with (non-nested) cross-validation.
get_best_model(results_dict, main_metric, ...)	Get the best model from a dictionary of model results.
get_cv_performance(model_class, df, ...[, ...])	Get the cross-validated performance of a model.
get_optimised_cv_performance(model_class, ...)	Get the cross-validated performance of a model with optimised hyperparameters.
perform_statistical_tests(results_dic, metric)	Perform Tukey's HSD and Nadeau-Bengio corrected pairwise t-tests (if parametric=True) or Conover post-hoc and Wilcoxon signed-rank tests (if parametric=False) tests on the performance of models.
run_CV(name, data_df, features, target, ...)	Run cross-validation for multiple models and save the results.
tukey_hsd(mse, residual_dof, score_means, ...)	Performs Tukey's HSD test using repeated measures ANOVA output.

astra.model_selection._cohens_d(a: ndarray, b: ndarray) → float

Calculate Cohen's d effect size for the difference between two sets of scores.

Parameters:

• a (np.ndarray) -- Fold scores for the first model.

• b (np.ndarray) -- Fold scores for the second model.

Returns:

Cohen's d effect size.

Return type:

float

astra.model_selection._min_detectable_effect(n_total: int, n_folds: int, alpha: float, power: float = 0.8) → float

Minimum Cohen's d detectable by the Nadeau-Bengio corrected t-test.

Derived from the corrected test's non-centrality parameter: lambda = d / sqrt(1/n + rho/(1-rho)), where rho = 1/k for k-fold CV. Setting lambda = z_{alpha/2} + z_{power} and solving for d gives the minimum detectable effect size.

Parameters:

• n_total (int) -- Total number of fold scores (repeats × k for repeated CV, k otherwise).

• n_folds (int) -- Number of folds per repeat k.

• alpha (float) -- Significance level (possibly Bonferroni-corrected).

• power (float, default=0.8) -- Desired statistical power.

Returns:

Minimum detectable Cohen's d at the given alpha and power.

Return type:

float

astra.model_selection.check_assumptions(results_dict: dict[str, dict[str, list[float]]], verbose: bool = True) → bool

Check homogeneity of variances and normality assumed by parametric statistical tests.

Parameters:

• results_dict (dict[str, dict[str, list[float]]]) -- A dictionary mapping model names to dictionaries of metric names and scores.

• verbose (bool, default=True) -- Whether to print warnings if assumptions are violated.

Returns:

True if all assumptions are met, False otherwise.

Return type:

bool

astra.model_selection.check_best_model(results_dic: dict[str, dict[str, list[float]]], test_statistics: DataFrame, metric: str, min_effect_size: float = 0.2, use_mean: bool = True) → str | None

Check if there is a model that is significantly better than the others, only counting pairwise wins that are statistically significant (p < 0.05) and practically meaningful (Cohen's d >= min_effect_size).

Parameters:

• results_dic (dict[str, dict[str, list[float]]]) -- A dictionary mapping model names to dictionaries of metric names and scores.

• test_statistics (pd.DataFrame) -- A dataframe containing the results of the statistical test.

• metric (str) -- The metric to use for model comparison.

• min_effect_size (float, default=0.2) -- Minimum Cohen's d required to count a pairwise difference as meaningful.

• use_mean (bool, default=True) -- If True, use mean fold scores to determine the direction of pairwise comparisons; if False, use median. Should be True for mean-based (parametric) tests and False for rank-based (non-parametric) tests.

Returns:

The name of the best model, or None if no model is significantly better.

Return type:

str or None

astra.model_selection.check_pareto_dominant(results_dict: dict[str, dict[str, list[float]]], main_metric: str, secondary_metrics: list[str], parametric: bool, min_effect_size: float = 0.2) → str | None

Find a model that is Pareto-dominant across all metrics (not significantly worse than any other model on any metric, significantly better than at least one other model on at least one metric, using p < 0.05 and Cohen's d >= min_effect_size).

Parameters:

• results_dict (dict[str, dict[str, list[float]]]) -- A dictionary mapping model names to dictionaries of metric names and scores.

• main_metric (str) -- The primary metric.

• secondary_metrics (list[str]) -- Secondary metrics to consider.

• parametric (bool) -- Whether to use parametric post-hoc tests.

• min_effect_size (float, default=0.2) -- Minimum Cohen's d required to count a pairwise difference as meaningful.

Returns:

The name of the Pareto-dominant model, or None if none exists.

Return type:

str or None

astra.model_selection.corrected_ttest(a: ndarray, b: ndarray, n_folds: int | None = None) → float

Nadeau-Bengio corrected repeated cross-validation t-test.

Adjusts the variance estimate for the correlation between CV fold scores arising from overlapping training sets. The correction factor is (1/n + rho/(1-rho)), where rho = 1/k is the fraction of data used for testing in each fold of k-fold CV.

Parameters:

• a (np.ndarray) -- Fold scores for the first model.

• b (np.ndarray) -- Fold scores for the second model.

• n_folds (int or None, default=None) -- Number of folds per repeat, when using repeated CV. For standard k-fold CV this equals len(a). If None, len(a) is used.

Returns:

Two-tailed p-value.

Return type:

float

astra.model_selection.find_n_best_models(results_dic: dict[str, dict[str, list[float]]], metric: str, parametric: bool = False, bf_corr: bool = True) → list[str]

Find the n best models that don't perform significantly differently with respect to a given metric as determined using repeated measures ANOVA (if parametric=True) or the Friedman test (if parametric=False). The function iteratively removes the model with the worst median score until no statistically significant difference is found or only one model remains.

Parameters:

• results_dic (dict[str, dict[str, list[float]]]) -- A dictionary mapping model names to dictionaries of metric names and scores.

• metric (str) -- The metric to use for model comparison.

• parametric (bool, default=False) -- Whether to use parametric tests instead of non-parametric tests.

• bf_corr (bool, default=True) -- Whether to apply Bonferroni correction to the significance level.

Returns:

A list of the n best models.

Return type:

list[str]

astra.model_selection.get_best_hparams(model_class: BaseEstimator, df: DataFrame, features_col: str, target_col: str, fold_col: str, main_metric: str, sec_metrics: list[str], parameters: dict[str, list] | dict[str, BaseDistribution], n_jobs: int, use_optuna: bool = False, n_trials: int = 100, timeout: int = 3600, impute: str | float | int | None = None, remove_constant: float | None = None, remove_correlated: float | None = None, scaler: str | None = None) → GridSearchCV | OptunaSearchCV

Get the best hyperparameters for a model using grid search with (non-nested) cross-validation.

Parameters:

• model_class (BaseEstimator) -- A scikit-learn model.

• df (pd.DataFrame) -- A dataframe containing the features and target values.

• features_col (str) -- The name of the column containing the features.

• target_col (str) -- The name of the column containing the target values.

• fold_col (str) -- The name of the column containing the fold indices.

• main_metric (str) -- The metric to optimise hyperparameters for.

• sec_metrics (list[str]) -- A list of secondary metrics to track during hyperparameter search.

• parameters (dict[str, list] or dict[str, BaseDistribution]) -- A dictionary of hyperparameters to search over.

• n_jobs (int) -- The number of jobs to run in parallel. For Optuna, this parallelises trials (not CV folds within each trial). With n_jobs > 1, the timeout is a soft limit: new trials stop being submitted once timeout elapses, but all in-flight trials run to completion. The actual runtime can therefore exceed timeout by up to n_jobs * single_trial_duration.

• use_optuna (bool, default=False) -- Whether to use Optuna for hyperparameter optimisation instead of grid search.

• n_trials (int, default=100) -- The number of trials to run during Optuna hyperparameter optimisation.

• timeout (int, default=3600) -- The maximum time in seconds to run Optuna hyperparameter optimisation. Note that this is a soft timeout: when n_jobs > 1, up to n_jobs trials already in flight will be allowed to finish, so the actual runtime may exceed this value.

• impute (str or float or int or None, default=None) -- The imputation strategy to use for missing values. If None, no imputation is performed. Valid choices are 'mean', 'median', 'knn', or a float or int value for constant imputation.

• remove_constant (float or None, default=None) -- If specified, features with variance below this threshold will be removed. If None, no features are removed.

• remove_correlated (float or None, default=None) -- If specified, features with correlation above this threshold will be removed. If None, no features are removed.

• scaler (str or None, default=None) -- The type of scaler to use. Valid choices are 'MinMax' and 'Standard'.

Returns:

A GridSearchCV or OptunaSearchCV object containing the best hyperparameters.

Return type:

GridSearchCV or OptunaSearchCV

astra.model_selection.get_best_model(results_dict: dict[str, dict[str, list[float]]], main_metric: str, secondary_metrics: list[str], parametric: bool = False, bf_corr: bool = True, n_folds: int | None = None) → tuple[str, str]

Get the best model from a dictionary of model results.

Parameters:

• results_dict (dict[str, dict[str, list[float]]]) -- A dictionary mapping model names to dictionaries of metric names and scores.

• main_metric (str) -- The main metric to use for model comparison.

• secondary_metrics (list[str]) -- A list of secondary metrics to use for model comparison.

• parametric (bool, default=False) -- Whether to use parametric tests instead of non-parametric tests.

• bf_corr (bool, default=True) -- Whether to apply Bonferroni correction to the significance level.

Returns:

A tuple containing the name of the best model and the reason for its selection.

Return type:

tuple[str, str]

astra.model_selection.get_cv_performance(model_class: BaseEstimator, df: DataFrame, features_col: str, target_col: str, fold_col: str, metric_list: list[str], impute: str | float | int | None = None, remove_constant: float | None = None, remove_correlated: float | None = None, scaler: str | None = None, custom_params: dict[str, list] | None = None) → dict[str, list[float]]

Get the cross-validated performance of a model.

Parameters:

• model_class (BaseEstimator) -- A scikit-learn model.

• df (pd.DataFrame) -- A dataframe containing the features and target values.

• features_col (str) -- The name of the column containing the features.

• target_col (str) -- The name of the column containing the target values.

• fold_col (str) -- The name of the column containing the fold indices.

• metric_list (list[str]) -- A list of metrics to use for evaluation.

• impute (str or float or int or None, default=None) -- The imputation strategy to use for missing values. If None, no imputation is performed. Valid choices are 'mean', 'median', 'knn', or a float or int value for constant imputation.

• remove_constant (float or None, default=None) -- If specified, features with variance below this threshold will be removed. If None, no features are removed.

• remove_correlated (float or None, default=None) -- If specified, features with correlation above this threshold will be removed. If None, no features are removed.

• scaler (str or None, default=None) -- The type of scaler to use. Valid choices are 'MinMax' and 'Standard'.

• custom_params (dict[str, list] or None, default=None) -- A dictionary of custom parameters for the model. If None, default parameters are used.

Returns:

A dictionary mapping metrics to lists of scores.

Return type:

dict[str, list[float]]

astra.model_selection.get_optimised_cv_performance(model_class: BaseEstimator, df: DataFrame, features_col: str, target_col: str, fold_col: str, metric_list: list[str], main_metric: str, parameters: dict[str, list] | dict[str, BaseDistribution], n_jobs: int, use_optuna: bool = False, n_trials: int = 100, timeout: int = 3600, impute: str | float | int | None = None, remove_constant: float | None = None, remove_correlated: float | None = None, scaler: str | None = None) → dict[str, list[float]]

Get the cross-validated performance of a model with optimised hyperparameters. The hyperparameters are optimised using grid search with nested cross-validation.

Parameters:

• model_class (BaseEstimator) -- A scikit-learn model.

• df (pd.DataFrame) -- A dataframe containing the features and target values.

• features_col (str) -- The name of the column containing the features.

• target_col (str) -- The name of the column containing the target values.

• fold_col (str) -- The name of the column containing the fold indices.

• metric_list (list[str]) -- A list of metrics to use for evaluation.

• main_metric (str) -- The main metric to optimise hyperparameters for.

• parameters (dict[str, list] or dict[str, BaseDistribution]) -- A dictionary of hyperparameters to search over.

• n_jobs (int) -- The number of jobs to run in parallel. For Optuna, this parallelises trials (not CV folds within each trial). With n_jobs > 1, the timeout is a soft limit: new trials stop being submitted once timeout elapses, but all in-flight trials run to completion. The actual runtime can therefore exceed timeout by up to n_jobs * single_trial_duration.

• use_optuna (bool, default=False) -- Whether to use Optuna for hyperparameter optimisation instead of grid search.

• n_trials (int, default=100) -- The number of trials to run during Optuna hyperparameter optimisation.

• timeout (int, default=3600) -- The maximum time in seconds to run Optuna hyperparameter optimisation. Note that this is a soft timeout: when n_jobs > 1, up to n_jobs trials already in flight will be allowed to finish, so the actual runtime may exceed this value.

• impute (str or float or int or None, default=None) -- The imputation strategy to use for missing values. If None, no imputation is performed. Valid choices are 'mean', 'median', 'knn', or a float or int value for constant imputation.

• remove_constant (float or None, default=None) -- If specified, features with variance below this threshold will be removed. If None, no features are removed.

• remove_correlated (float or None, default=None) -- If specified, features with correlation above this threshold will be removed. If None, no features are removed.

• scaler (str or None, default=None) -- The type of scaler to use. Valid choices are 'MinMax' and 'Standard'.

Returns:

A dictionary mapping metrics to lists of scores.

Return type:

dict[str, list[float]]

astra.model_selection.perform_statistical_tests(results_dic: dict[str, dict[str, list[float]]], metric: str, parametric: bool = False, n_folds: int | None = None) → tuple[DataFrame, DataFrame]

Perform Tukey's HSD and Nadeau-Bengio corrected pairwise t-tests (if parametric=True) or Conover post-hoc and Wilcoxon signed-rank tests (if parametric=False) tests on the performance of models. Note that Wilcoxon is anti-conservative under CV fold dependency, but no established non-parametric analogue of the Nadeau-Bengio correction exists.

Parameters:

• results_dic (dict[str, dict[str, list[float]]]) -- A dictionary mapping model names to dictionaries of metric names and scores.

• metric (str) -- The metric to use for model comparison.

• parametric (bool, default=False) -- Whether to use parametric tests instead of non-parametric tests.

• n_folds (int or None, default=None) -- Number of folds per repeat, passed through to corrected_ttest for rho. See corrected_ttest for details. Ignored when parametric=False.

Returns:

A tuple containing the test results for the two statistical tests.

Return type:

tuple[pd.DataFrame, pd.DataFrame]

astra.model_selection.run_CV(name: str, data_df: DataFrame, features: str, target: str, fold_col: str, models: dict[str, BaseEstimator], metric_list: list[str], impute: str | float | int | None = None, remove_constant: float | None = None, remove_correlated: float | None = None, scaler: str | None = None, custom_params: dict[str, dict[str, list]] | None = None, repeated: bool = False)

Run cross-validation for multiple models and save the results.

Parameters:

• name (str) -- Name for the results directory and the experiment.

• data_df (pd.DataFrame) -- DataFrame containing the data for cross-validation.

• features (str) -- Name of the column containing features.

• target (str) -- Name of the column containing the target variable.

• fold_col (str) -- Name of the column containing fold assignments.

• models (dict[str, BaseEstimator]) -- Dictionary of models to evaluate, with model names as keys and scikit-learn-like estimators as values.

• metric_list (list of str) -- List of metrics to evaluate during cross-validation.

• impute (str | float | int or None, default None) -- Imputation strategy to apply before the model. Valid options are 'mean', 'median', 'knn', or a numeric value for constant imputation. If None, no imputation is applied.

• remove_constant (float or None, default None) -- Threshold for variance to remove constant features. If None, no features are removed.

• remove_correlated (float or None, default None) -- Threshold for correlation to remove correlated features. If None, no features are removed.

• scaler (str or None, default None) -- Type of scaler to apply before the model. Valid options are 'MinMax' or 'Standard'. If None, no scaling is applied.

• custom_params (dict[str, dict[str, list]] or None, default None) -- Dictionary of custom hyperparameter grids for each model. If None, default grids will be used.

• repeated (bool, default False) -- Whether the cross-validation is repeated. If True, results are saved with the fold column name.

Returns:

Dictionary containing cross-validation results for each model.

Return type:

dict

astra.model_selection.tukey_hsd(mse: float, residual_dof: int, score_means: Series, n_folds: int) → DataFrame

Performs Tukey's HSD test using repeated measures ANOVA output.

Parameters:

• mse (float) -- Mean squared error from ANOVA.

• residual_dof (int) -- Residual degrees of freedom.

• score_means (pd.Series) -- Mean scores.

• n_folds (int) -- Total number of folds per model.

Returns:

p-values for pairwise comparisons between models.

Return type:

pd.DataFrame