Analyzing results#

Optimas provides a convenient ExplorationDiagnostics class to easily analyze and visualize the output of an exploration without having to manually access each file.

The examples below showcase the functionality of this class by analyzing the output of this basic Bayesian optimization example.

Import and initialize diagnostics#

The diagnostics class only requires the path to the exploration directory as input parameter.

[2]:

from optimas.diagnostics import ExplorationDiagnostics

diags = ExplorationDiagnostics("./exploration")

[INFO 06-24 17:16:43] optimas.generators.base: Completed trial 0 with objective(s) {'f': np.float64(26.326525)}
[INFO 06-24 17:16:43] optimas.generators.base: Completed trial 1 with objective(s) {'f': np.float64(-99.720534)}
[INFO 06-24 17:16:43] optimas.generators.base: Completed trial 2 with objective(s) {'f': np.float64(-128.304592)}
[INFO 06-24 17:16:43] optimas.generators.base: Completed trial 3 with objective(s) {'f': np.float64(-15.410015)}
[INFO 06-24 17:16:43] optimas.generators.base: Completed trial 4 with objective(s) {'f': np.float64(-134.197009)}
[INFO 06-24 17:16:43] optimas.generators.base: Completed trial 5 with objective(s) {'f': np.float64(-85.315574)}
[INFO 06-24 17:16:43] optimas.generators.base: Completed trial 6 with objective(s) {'f': np.float64(-105.575953)}
[INFO 06-24 17:16:43] optimas.generators.base: Completed trial 7 with objective(s) {'f': np.float64(-112.015604)}
[INFO 06-24 17:16:43] optimas.generators.base: Completed trial 8 with objective(s) {'f': np.float64(-112.247164)}
[INFO 06-24 17:16:43] optimas.generators.base: Completed trial 9 with objective(s) {'f': np.float64(-37.015604)}
[INFO 06-24 17:16:43] optimas.generators.base: Completed trial 10 with objective(s) {'f': np.float64(-388.979561)}
[INFO 06-24 17:16:43] optimas.generators.base: Completed trial 11 with objective(s) {'f': np.float64(50.170087)}
[INFO 06-24 17:16:43] optimas.generators.base: Completed trial 12 with objective(s) {'f': np.float64(-308.479872)}
[INFO 06-24 17:16:43] optimas.generators.base: Completed trial 13 with objective(s) {'f': np.float64(-106.164417)}
[INFO 06-24 17:16:43] optimas.generators.base: Completed trial 14 with objective(s) {'f': np.float64(-396.948282)}

Access exploration history#

The diagnostics provide easy access to the exploration history, which is returned as a pandas DataFrame.

[3]:

diags.history

[3]:
trial_index trial_status trial_ignored x0 x1 f cancel_requested gen_ended_time gen_informed gen_informed_time ... given_back kill_sent num_gpus num_procs sim_ended sim_ended_time sim_id sim_started sim_started_time sim_worker
0 0 COMPLETED False 5.945774 3.255866 26.326525 False 1.782321e+09 True 1.782321e+09 ... False False 0 1 True 1.782321e+09 0 True 1.782321e+09 1
1 1 COMPLETED False 13.979908 8.224355 -99.720534 False 1.782321e+09 True 1.782321e+09 ... False False 0 1 True 1.782321e+09 1 True 1.782321e+09 2
2 2 COMPLETED False 15.000000 12.400103 -128.304592 False 1.782321e+09 True 1.782321e+09 ... False False 0 1 True 1.782321e+09 2 True 1.782321e+09 1
3 3 COMPLETED False 15.000000 4.261297 -15.410015 False 1.782321e+09 True 1.782321e+09 ... False False 0 1 True 1.782321e+09 3 True 1.782321e+09 2
4 4 COMPLETED False 0.825663 12.669087 -134.197009 False 1.782321e+09 True 1.782321e+09 ... False False 0 1 True 1.782321e+09 4 True 1.782321e+09 1
5 5 COMPLETED False 15.000000 11.083786 -85.315574 False 1.782321e+09 True 1.782321e+09 ... False False 0 1 True 1.782321e+09 5 True 1.782321e+09 2
6 6 COMPLETED False 15.000000 14.106201 -105.575953 False 1.782321e+09 True 1.782321e+09 ... False False 0 1 True 1.782321e+09 6 True 1.782321e+09 1
7 7 COMPLETED False 0.000000 15.000000 -112.015604 False 1.782321e+09 True 1.782321e+09 ... False False 0 1 True 1.782321e+09 7 True 1.782321e+09 2
8 8 COMPLETED False 0.000000 6.742143 -112.247164 False 1.782321e+09 True 1.782321e+09 ... False False 0 1 True 1.782321e+09 8 True 1.782321e+09 1
9 9 COMPLETED False 15.000000 0.000000 -37.015604 False 1.782321e+09 True 1.782321e+09 ... False False 0 1 True 1.782321e+09 9 True 1.782321e+09 2
10 10 COMPLETED False 13.011567 12.847173 -388.979561 False 1.782321e+09 True 1.782321e+09 ... False False 0 1 True 1.782321e+09 10 True 1.782321e+09 1
11 11 COMPLETED False 2.318150 15.000000 50.170087 False 1.782321e+09 True 1.782321e+09 ... False False 0 1 True 1.782321e+09 11 True 1.782321e+09 2
12 12 COMPLETED False 12.741981 14.257649 -308.479872 False 1.782321e+09 True 1.782321e+09 ... False False 0 1 True 1.782321e+09 12 True 1.782321e+09 1
13 13 COMPLETED False 12.840276 9.625179 -106.164417 False 1.782321e+09 True 1.782321e+09 ... False False 0 1 True 1.782321e+09 13 True 1.782321e+09 2
14 14 COMPLETED False 12.496440 12.800210 -396.948282 False 1.782321e+09 True 1.782321e+09 ... False False 0 1 True 1.782321e+09 14 True 1.782321e+09 1

15 rows × 22 columns

Built-in plotting utilities#

Several basic plotting functions are provided by the diagnostics class. The example below uses plot_objective() to show the value of the objective f for each evaluation, as well as the evolution of the cumulative best.

[4]:

diags.plot_objective(show_trace=True)
../../_images/user_guide_basic_usage_exploration_diagnostics_7_0.png

User plots#

The ExplorationDiagnostics exposes all necessary data to perform any analysis or plot of the exploration. As an example, the code below generates a plot of the phase-space of the optimization, including the value of each evaluation and the boundaries of the varying parameters.

[5]:

import matplotlib.pyplot as plt

fig, ax = plt.subplots()
vocs = diags._exploration.generator.vocs
df = diags.history
vps = list(vocs.variables.keys())
f1 = list(vocs.objectives.keys())[0]
ax.axvline(vocs.variables[vps[0]].domain[0])
ax.axvline(vocs.variables[vps[0]].domain[1])
ax.set_xlabel(vps[0])
ax.axhline(vocs.variables[vps[1]].domain[0])
ax.axhline(vocs.variables[vps[1]].domain[1])
ax.set_ylabel(vps[1])
ax.scatter(df[vps[0]], df[vps[1]], c=df[f1])

[5]:

<matplotlib.collections.PathCollection at 0x730bbf53cb50>
../../_images/user_guide_basic_usage_exploration_diagnostics_9_1.png