Export Outputs to Weights & Biases (wandb)¶

In this guide, we will demonstrate how to export the Check’s and the Suite’s output to wandb.

This enables to view the exported results in wandb and compare the results you receive with diffrent parameters or model/data.

Notebook Structure:

Load Data and Run Check
Export Check to wandb
Export Suite to wandb

Load Data¶

[2]:

from deepchecks.datasets.classification import iris

train_dataset, test_dataset = iris.load_data()
model = iris.load_fitted_model()

Run a Check¶

[3]:

from deepchecks.checks import ModelErrorAnalysis

result = ModelErrorAnalysis().add_condition_segments_performance_relative_difference_not_greater_than(\
                                ).run(train_dataset, test_dataset, model)

Observe CheckResult Display and Value¶

[4]:

result

Model Error Analysis

Find features that best split the data into segments of high and low model error. Read More...

Conditions Summary

Status	Condition	More Info
!	The performance difference of the detected segments must not be greater than 5%	Found change in Accuracy in features above threshold: {'petal length (cm)': '8%', 'petal width (cm)': '8%'}

Additional Outputs

The following graphs show the distribution of error for top features that are most useful for distinguishing high error samples from low error samples. Top features are calculated using `feature_importances_`.

[5]:

# the value can be observed
result.value

[5]:

{'scorer_name': 'Accuracy',
 'feature_segments': {'petal length (cm)': {'segment1': {'score': 0.92,
    'n_samples': 25,
    'frac_samples': 0.6578947368421053},
   'segment2': {'score': 1.0,
    'n_samples': 13,
    'frac_samples': 0.34210526315789475}},
  'petal width (cm)': {'segment1': {'score': 0.92,
    'n_samples': 25,
    'frac_samples': 0.6578947368421053},
   'segment2': {'score': 1.0,
    'n_samples': 13,
    'frac_samples': 0.34210526315789475}}}}

Export a Check’s Output (CheckResult) to wandb¶

Exporting the output to wandb is possible using the to_wandb function. This function exports the check outputs to a wandb project. The output display that will be export will be a bit diffrent from what you usually see. Only the tables and the plots are being exported. On default if you export a single check to wandb without a wandb run active it will create a project with the name deepchecks and the and the check’s metadata in the config and export the results there.

See Check to_wandb options¶

[6]:

from deepchecks import CheckResult

help(CheckResult.to_wandb)

Help on function to_wandb in module deepchecks.base.check:

to_wandb(self, dedicated_run: bool = True, **kwargs: Any)
    Export check result to wandb.
    Parameters
    ----------
    dedicated_run : bool , default: None
        If to initiate and finish a new wandb run.
        If None it will be dedicated if wandb.run is None.
    kwargs: Keyword arguments to pass to wandb.init.
            Default project name is deepchecks.
            Default config is the check metadata (params, train/test/ name etc.).

To use this function you first need to login to your wandb account

[7]:

import wandb
wandb.login()

[7]:

False

Then just run the to_wandb function.

[ ]:

result.to_wandb()

Exporting a Suite’s Output (SuiteResult) to wandb¶

Run Suite and export to wandb¶

Exporting a suite to wandb is the same as exporting a check. The display in wand will have a section per check. On default if you export a suite to wandb without a wandb run active it will create a project with the name deepchecks and the suite’s name in the config and send the results there.

[8]:

from deepchecks.suites import full_suite

suite = full_suite()

[ ]:

suite_result = suite.run(train_dataset=train_dataset, test_dataset=test_dataset, model=model)
suite_result.to_wandb()

You can also set all the kwargs the wandb.init will get:¶

[ ]:

suite_result.to_wandb(project='my-suite-project', config={'suite-name': 'full-suite'})

Export Outputs to JSON

Classifying Malicious URLs