CAVE in jupyter

Short introduction to motivation of automated parameter optimization

CAVE

CAVE stands for Configuration Assessment, Visualization and Evaluation. It is designed to create comprehensive reports about an optimization process. The resulting figures and interactive plots can be used to gain insights in the parameter importance, feature importance, search behaviour and quality. We will walk you through the different analysis-options, that CAVE offers.

Quickstart Guid

We will analyze the results of a SMAC-run (...).

Please make sure you are in the examples-folder to run this example. We will start by initializing the CAVE-object with three arguments:

--folders specifies the path to the directory containing the optimization outputs
--output-dir specifies the directory for the analysis-output
--ta_exec_dir specifies the path to the directory, from where the optimization was executed. This is important, if any files (like instance-files) are specified in relative paths



In [ ]:

    
from cave.cavefacade import CAVE

#cave = CAVE(["workflow-result"], "test_jupyter", ["."], file_format='BOHB')
cave = CAVE(folders=["./smac3/example_output/run_1", "./smac3/example_output/run_2"],
            output_dir="cave_on_jupyter",
            ta_exec_dir=["./smac3"],
            verbose_level="OFF"
           )

We can use the CAVE-object to generate general information in tables:



In [ ]:

    
cave.overview_table()



In [ ]:

    
cave.compare_default_incumbent()

Performance Analysis

To analyze how much better the incumbent is to the default, CAVE provides several methods. Obviously we want to compare the average. If you optimize the runtime of instances, the Penalized Average Runtime (PAR) will be interesting for you, too:



In [ ]:

    
cave.performance_table();

Only available for instances, CAVE can provide scatter and cumulated distribution plots (using an EPM to estimate performance of config-instance pairs that were not actually executed). Individual plots will be generated for train and test instances (if distinction is made). Left -> train, right -> test



In [ ]:

    
cave.plot_scatter()



In [ ]:

    
cave.plot_ecdf()

A special feature of CAVE is the analysis of algorithm footprints. Instances are mapped on a 2d-plot and marked blue if the selected algorithm is better than the other one. Use the tools on the right side to navigate within the plot.



In [ ]:

    
cave.algorithm_footprints()

But what is really interesting: CAVE can use the whole information collected during the optimization to gain insights into your configurator's behaviour. The Configurator Footprint might take a little time to generate for big data, but it is a wonderful way to see, how your configurator explored the search space over time.



In [ ]:

    
cave.configurator_footprint()



In [ ]:

    
cave.feature_importance()



In [ ]:

    
cave.pimp_forward_selection()



In [ ]:

    
cave.cave_ablation()



In [ ]:

    
cave.cave_fanova()



In [ ]:

    
cave.pimp_comparison_table()



In [ ]:

    
cave.algorithm_footprints()



In [ ]:

    
cave.cost_over_time()



In [ ]:

    
cave.parallel_coordinates()