TrustScoreComparison

class TrustScoreComparison

Compares the model’s trust score for the train dataset with scores of the test dataset.

The Trust Score algorithm and code was published in the paper: “To Trust or not to trust c classifier”. See the original paper at arxiv 1805.11783, or see the version of the paper presented at NeurIPS in 2018: https://proceedings.neurips.cc/paper/2018/file/7180cffd6a8e829dacfc2a31b3f72ece-Paper.pdf

The process is as follows:

1. Pre-process the train and test data into scaled numerics.

2. Train a TrustScore regressor based on train data + label.

3. Predict on test data using the model.

4. Use TrustScore to score the prediction of the model.

Parameters

k_filterint , default: 10

used in TrustScore (Number of neighbors used during either kNN distance or probability filtering)

alphafloat , default: 0.001

used in TrustScore (Fraction of instances to filter out to reduce impact of outliers)

max_number_categoriesint , default: 10

Indicates the maximum number of unique categories in a single categorical column (rare categories will be changed to a form of “other”)

min_test_samplesint , default: 300

Minimal number of samples in train data to be able to run this check

sample_sizeint , default: 10_000

Number of samples to use for the check for train and test. if dataset contains less than sample_size than all the dataset will be used.

random_stateint , default: 42

The random state to use for sampling.

n_to_showint , default: 5

Number of samples to show of worst and best trust score.

percent_top_scores_to_hidefloat , default: 0.05

__init__(k_filter: int = 10, alpha: float = 0.001, max_number_categories: int = 10, min_test_samples: int = 300, sample_size: int = 10000, random_state: int = 42, n_to_show: int = 5, percent_top_scores_to_hide: float = 0.05)

__new__(*args, **kwargs)

Methods

TrustScoreComparison.add_condition(name, ...)	Add new condition function to the check.
TrustScoreComparison.add_condition_mean_score_percent_decline_not_greater_than([...])	Add condition.
TrustScoreComparison.clean_conditions()	Remove all conditions from this check instance.
TrustScoreComparison.conditions_decision(result)	Run conditions on given result.
TrustScoreComparison.name()	Name of class in split camel case.
TrustScoreComparison.params([show_defaults])	Return parameters to show when printing the check.
TrustScoreComparison.remove_condition(index)	Remove given condition by index.
TrustScoreComparison.run(train_dataset, ...)	Run check.
TrustScoreComparison.run_logic(context)	Run check.

Example

TrustScoreComparison Example Notebook

previous

Distribution

next

TrustScoreComparison.add_condition