TrainTestFeatureDrift

class TrainTestFeatureDrift[source]

Calculate drift between train dataset and test dataset per feature, using statistical measures.

Check calculates a drift score for each column in test dataset, by comparing its distribution to the train dataset. For numerical columns, we use the Earth Movers Distance. See https://en.wikipedia.org/wiki/Wasserstein_metric For categorical columns, we use the Population Stability Index (PSI). See https://www.lexjansen.com/wuss/2017/47_Final_Paper_PDF.pdf

Parameters
columnsUnion[Hashable, List[Hashable]] , default: None

Columns to check, if none are given checks all columns except ignored ones.

ignore_columnsUnion[Hashable, List[Hashable]] , default: None

Columns to ignore, if none given checks based on columns variable.

n_top_columnsint , optional

amount of columns to show ordered by feature importance (date, index, label are first)

sort_feature_bystr , default: feature importance

Indicates how features will be sorted. Can be either “feature importance” or “drift score”

max_num_categoriesint , default: 10

Only for categorical columns. Max number of allowed categories. If there are more, they are binned into an “Other” category. If max_num_categories=None, there is no limit. This limit applies for both drift calculation and for distribution plots.

n_samplesint , default: 100_000

Number of samples to use for drift computation and plot.

random_stateint , default: 42

Random seed for sampling.

__init__(columns: Optional[Union[Hashable, List[Hashable]]] = None, ignore_columns: Optional[Union[Hashable, List[Hashable]]] = None, n_top_columns: int = 5, sort_feature_by: str = 'feature importance', max_num_categories: int = 10, n_samples: int = 100000, random_state: int = 42)[source]
__new__(*args, **kwargs)

Methods

TrainTestFeatureDrift.add_condition(name, ...)

Add new condition function to the check.

TrainTestFeatureDrift.add_condition_drift_score_not_greater_than([...])

Add condition - require drift score to not be more than a certain threshold.

TrainTestFeatureDrift.clean_conditions()

Remove all conditions from this check instance.

TrainTestFeatureDrift.conditions_decision(result)

Run conditions on given result.

TrainTestFeatureDrift.name()

Name of class in split camel case.

TrainTestFeatureDrift.params([show_defaults])

Return parameters to show when printing the check.

TrainTestFeatureDrift.remove_condition(index)

Remove given condition by index.

TrainTestFeatureDrift.run(train_dataset, ...)

Run check.

TrainTestFeatureDrift.run_logic(context)

Calculate drift for all columns.

Example

thumbnail

TrainTestFeatureDrift Example Notebook

previous

TrustScoreComparison.run_logic

next

TrainTestFeatureDrift.add_condition