astra.data.processing¶

This module contains the CorrelationFilter class, which is used to filter out features that are highly correlated with each other based on a specified threshold.

Classes

CorrelationFilter([threshold])

A transformer that removes features from a dataset that are highly correlated with each other.

class astra.data.processing.CorrelationFilter(threshold: float = 0.95)[source]¶

Bases: BaseEstimator, TransformerMixin

A transformer that removes features from a dataset that are highly correlated with each other.

Parameters:: threshold (float, default=0.95) -- The correlation threshold above which features will be considered highly correlated and removed.

to_drop¶

A set of indices of features to be dropped from the dataset after fitting.

Type:: set

fit(X: np.ndarray, y: None = None) → CorrelationFilter[source]¶: Fit the transformer to the data by calculating the correlation matrix and identifying features to drop.

transform(X: np.ndarray) → np.ndarray[source]¶: Transform the input data by removing the features identified during fitting.

fit_transform(X: np.ndarray, y: None = None) → np.ndarray[source]¶: Fit the transformer and transform the input data in one step.

get_feature_names_out(input_features: list[str]) → list[str][source]¶: Get the names of the features that are retained after transformation.

Notes

This transformer is compatible with the scikit-learn API, allowing it to be used seamlessly with other transformers and estimators.

Examples

>>> from astra.data.processing import CorrelationFilter
>>> import numpy as np
>>> X = np.array([[1, 2, 3],
...               [4, 5, 6],
...               [7, 8, 9],
...               [1, 2, 3]])
>>> cf = CorrelationFilter(threshold=0.9)
>>> cf.fit_transform(X)
array([[1],
       [4],
       [7],
       [1]])

__annotate_func__ = None¶

__annotations_cache__ = {}¶

__dict__ = mappingproxy({'__module__': 'astra.data.processing', '__firstlineno__': 11, '__doc__': '\nA transformer that removes features from a dataset that are highly correlated with each other.\n\nParameters\n----------\nthreshold : float, default=0.95\n The correlation threshold above which features will be considered highly correlated and removed.\n\nAttributes\n----------\nto_drop : set\n A set of indices of features to be dropped from the dataset after fitting.\n\nMethods\n-------\nfit(X: np.ndarray, y: None = None) -> CorrelationFilter\n Fit the transformer to the data by calculating the correlation matrix and identifying features to drop.\ntransform(X: np.ndarray) -> np.ndarray\n Transform the input data by removing the features identified during fitting.\nfit_transform(X: np.ndarray, y: None = None) -> np.ndarray\n Fit the transformer and transform the input data in one step.\nget_feature_names_out(input_features: list[str]) -> list[str]\n Get the names of the features that are retained after transformation.\n\nNotes\n-----\nThis transformer is compatible with the scikit-learn API, allowing it to be used\nseamlessly with other transformers and estimators.\n\nExamples\n--------\n>>> from astra.data.processing import CorrelationFilter\n>>> import numpy as np\n>>> X = np.array([[1, 2, 3],\n... [4, 5, 6],\n... [7, 8, 9],\n... [1, 2, 3]])\n>>> cf = CorrelationFilter(threshold=0.9)\n>>> cf.fit_transform(X)\narray([[1],\n [4],\n [7],\n [1]])\n', '__init__': <function CorrelationFilter.__init__>, 'fit': <function CorrelationFilter.fit>, 'transform': <function CorrelationFilter.transform>, 'fit_transform': <function CorrelationFilter.fit_transform>, 'get_feature_names_out': <function CorrelationFilter.get_feature_names_out>, '__static_attributes__': ('threshold', 'to_drop'), '_sklearn_auto_wrap_output_keys': {'transform'}, '__annotate_func__': None, '__annotations_cache__': {}})¶

__dir__()¶: Default dir() implementation.

__firstlineno__ = 11¶

__getstate__()¶: Helper for pickle.

__init__(threshold: float = 0.95) → None[source]¶

Initialize the CorrelationFilter with a correlation threshold.

Parameters:: threshold (float, default=0.95) -- The correlation threshold above which features will be considered highly correlated and removed.

classmethod __init_subclass__(**kwargs)¶

Set the set_{method}_request methods.

This uses PEP-487 [1] to set the set_{method}_request methods. It looks for the information available in the set default values which are set using __metadata_request__* class attributes, or inferred from method signatures.

The __metadata_request__* class attributes are used when a method does not explicitly accept a metadata through its arguments or if the developer would like to specify a request value for those metadata which are different from the default None.

References

__module__ = 'astra.data.processing'¶

__repr__(N_CHAR_MAX=700)¶: Return repr(self).

__setstate__(state)¶

__sklearn_clone__()¶

__sklearn_tags__()¶

__static_attributes__ = ('threshold', 'to_drop')¶

__weakref__¶: list of weak references to the object

_doc_link_module = 'sklearn'¶

property _doc_link_template¶

_doc_link_url_param_generator = None¶

classmethod _get_class_level_metadata_request_values(method: str)¶

Get class level metadata request values.

This method first checks the method's signature for passable metadata and then updates these with the metadata request values set at class level via the __metadata_request__{method} class attributes.

This method (being a class-method), does not take request values set at instance level into account.

_get_doc_link()¶

Generates a link to the API documentation for a given estimator.

This method generates the link to the estimator's documentation page by using the template defined by the attribute _doc_link_template.

Returns:: url -- The URL to the API documentation for this estimator. If the estimator does not belong to module _doc_link_module, the empty string (i.e. "") is returned.
Return type:: str

_get_metadata_request()¶

Get requested metadata for the instance.

Please check User Guide on how the routing mechanism works.

Returns:: request -- A MetadataRequest instance.
Return type:: MetadataRequest

classmethod _get_param_names()¶: Get parameter names for the estimator

_get_params_html(deep=True, doc_link='')¶

Get parameters for this estimator with a specific HTML representation.

Parameters:

deep (bool, default=True) -- If True, will return the parameters for this estimator and contained subobjects that are estimators.
doc_link (str) -- URL to the estimator documentation. Used for linking to the estimator's parameters documentation available in HTML displays.

Returns:

params -- Parameter names mapped to their values. We return a ParamsDict dictionary, which renders a specific HTML representation in table form.

Return type:

ParamsDict

_html_repr()¶

Build a HTML representation of an estimator.

Read more in the User Guide.

Parameters:: estimator (estimator object) -- The estimator to visualize.
Returns:: html -- HTML representation of estimator.
Return type:: str

Examples

>>> from sklearn.utils._repr_html.estimator import estimator_html_repr
>>> from sklearn.linear_model import LogisticRegression
>>> estimator_html_repr(LogisticRegression())
'<style>#sk-container-id...'

property _repr_html_¶: HTML representation of estimator. This is redundant with the logic of _repr_mimebundle_. The latter should be favored in the long term, _repr_html_ is only implemented for consumers who do not interpret _repr_mimbundle_.

_repr_html_inner()¶: This function is returned by the @property _repr_html_ to make hasattr(estimator, "_repr_html_") return `True or False depending on get_config()["display"].

_repr_mimebundle_(**kwargs)¶: Mime bundle used by jupyter kernels to display estimator

_sklearn_auto_wrap_output_keys = {'transform'}¶

_validate_params()¶

Validate types and values of constructor parameters

The expected type and values must be defined in the _parameter_constraints class attribute, which is a dictionary param_name: list of constraints. See the docstring of validate_parameter_constraints for a description of the accepted constraints.

fit(X: ndarray, y: None = None) → CorrelationFilter[source]¶

Fit the transformer to the data by calculating the correlation matrix and identifying features to drop.

Parameters:

X (array-like, shape (n_samples, n_features)) -- The input data to fit the transformer on.
y (None, default=None) -- Ignored, exists for compatibility with the scikit-learn API.

Returns:

self -- Returns the instance itself.

Return type:

CorrelationFilter

fit_transform(X: ndarray, y: None = None) → ndarray[source]¶

Fit the transformer and transform the input data in one step.

Parameters:

X (array-like, shape (n_samples, n_features)) -- The input data to fit and transform.
y (None, default=None) -- Ignored, exists for compatibility with the scikit-learn API.

Returns:

The transformed data with highly correlated features removed.

Return type:

np.ndarray

get_feature_names_out(input_features: list[str]) → list[str][source]¶

Get the names of the features that are retained after transformation.

Parameters:: input_features (list of str) -- The names of the input features.
Returns:: The names of the features that are retained after transformation.
Return type:: list of str

get_metadata_routing()¶

Get metadata routing of this object.

Please check User Guide on how the routing mechanism works.

Returns:: routing -- A MetadataRequest encapsulating routing information.
Return type:: MetadataRequest

get_params(deep=True)¶

Get parameters for this estimator.

Parameters:: deep (bool, default=True) -- If True, will return the parameters for this estimator and contained subobjects that are estimators.
Returns:: params -- Parameter names mapped to their values.
Return type:: dict

set_output(*, transform=None)¶

Set output container.

See Introducing the set_output API for an example on how to use the API.

Parameters:

transform ({"default", "pandas", "polars"}, default=None) -- Configure output of transform and fit_transform.

"default": Default output format of a transformer
"pandas": DataFrame output
"polars": Polars output
None: Transform configuration is unchanged

Added in version 1.4: "polars" option was added.

Returns:

self -- Estimator instance.

Return type:

estimator instance

set_params(**params)¶

Set the parameters of this estimator.

The method works on simple estimators as well as on nested objects (such as Pipeline). The latter have parameters of the form <component>__<parameter> so that it's possible to update each component of a nested object.

Parameters:: **params (dict) -- Estimator parameters.
Returns:: self -- Estimator instance.
Return type:: estimator instance

transform(X: ndarray) → ndarray[source]¶

Transform the input data by removing the features identified during fitting.

Parameters:: X (array-like, shape (n_samples, n_features)) -- The input data to transform.
Returns:: The transformed data with highly correlated features removed.
Return type:: np.ndarray