Skip to content

Metrics

running_time_closed_gap(schedules, performance, budget, par, feature_time)

Calculates the closed gap metric for a given selector.

Parameters:

Name Type Description Default
schedules Dict[str, List[Tuple[str, float]]]

The schedules to evaluate.

 required
performance DataFrame

The performance data for the algorithms.

 required
budget float

The budget for the scenario.

 required
par float

The penalization factor for unsolved instances.

 required
feature_time DataFrame

The feature time data for each instance.

 required

Returns:

Name Type Description
float float

The closed gap value, representing the improvement of the selector over the single best solver
float

relative to the virtual best solver.
Source code in asf/metrics/baselines.py 
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
def running_time_closed_gap(
    schedules: Dict[str, List[Tuple[str, float]]],
    performance: pd.DataFrame,
    budget: float,
    par: float,
    feature_time: pd.DataFrame,
) -> float:
    """
    Calculates the closed gap metric for a given selector.

    Args:
        schedules (Dict[str, List[Tuple[str, float]]]): The schedules to evaluate.
        performance (pd.DataFrame): The performance data for the algorithms.
        budget (float): The budget for the scenario.
        par (float): The penalization factor for unsolved instances.
        feature_time (pd.DataFrame): The feature time data for each instance.

    Returns:
        float: The closed gap value, representing the improvement of the selector over the single best solver
        relative to the virtual best solver.
    """
    sbs_val = single_best_solver(performance, False)
    vbs_val = virtual_best_solver(performance, False)
    s_val = running_time_selector_performance(
        schedules, performance, budget, par, feature_time
    )

    return (sbs_val - s_val) / (sbs_val - vbs_val)

running_time_selector_performance(schedules, performance, budget=5000, par=10, feature_time=None)

Calculates the total running time for a selector based on the given schedules and performance data.

Parameters:

Name Type Description Default
schedules Dict[str, List[Tuple[str, float]]]

The schedules to evaluate, where each key is an instance and the value is a list of tuples (algorithm, allocated budget).

 required
performance DataFrame

The performance data for the algorithms.

 required
budget float

The budget for the scenario.

 5000
par float

The penalization factor for unsolved instances.

 10
feature_time Optional[DataFrame]

The feature time data for each instance. Defaults to zero if not provided.

 None

Returns:

Type Description
Dict[str, Union[float, int]]

Dict[str, Union[float, int]]: A dictionary mapping each instance to its total running time.
Source code in asf/metrics/baselines.py 
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
def running_time_selector_performance(
    schedules: Dict[str, List[Tuple[str, float]]],
    performance: pd.DataFrame,
    budget: float = 5000,
    par: float = 10,
    feature_time: Optional[pd.DataFrame] = None,
) -> Dict[str, Union[float, int]]:
    """
    Calculates the total running time for a selector based on the given schedules and performance data.

    Args:
        schedules (Dict[str, List[Tuple[str, float]]]): The schedules to evaluate, where each key is an instance
            and the value is a list of tuples (algorithm, allocated budget).
        performance (pd.DataFrame): The performance data for the algorithms.
        budget (float): The budget for the scenario.
        par (float): The penalization factor for unsolved instances.
        feature_time (Optional[pd.DataFrame]): The feature time data for each instance. Defaults to zero if not provided.

    Returns:
        Dict[str, Union[float, int]]: A dictionary mapping each instance to its total running time.
    """
    if feature_time is None:
        feature_time = pd.DataFrame(
            0, index=performance.index, columns=["feature_time"]
        )
    total_time = {}
    for instance, schedule in schedules.items():
        allocated_times = {algorithm: 0 for algorithm in performance.columns}
        solved = False
        for algorithm, algo_budget in schedule:
            remaining_budget = (
                budget
                - sum(allocated_times.values())
                - feature_time.loc[instance].item()
            )
            remaining_time_to_solve = performance.loc[instance, algorithm] - (
                algo_budget + allocated_times[algorithm]
            )
            if remaining_time_to_solve < 0:
                allocated_times[algorithm] = performance.loc[instance, algorithm]
                solved = True
                break
            elif remaining_time_to_solve <= remaining_budget:
                allocated_times[algorithm] += remaining_time_to_solve
            else:
                allocated_times[algorithm] += remaining_budget
                break
        if solved:
            total_time[instance] = (
                sum(allocated_times.values()) + feature_time.loc[instance].item()
            )
        else:
            total_time[instance] = budget * par

    total_time = sum(list(total_time.values()))
    return total_time

single_best_solver(performance, maximize=False)

Selects the single best solver across all instances based on the aggregated performance.

Parameters:

Name Type Description Default
schedules DataFrame

The schedules to evaluate (not used in this function).

 required
performance DataFrame

The performance data for the algorithms.

 required
maximize bool

Whether to maximize or minimize the performance.

 False

Returns:

Name Type Description
float float

The best aggregated performance value across all instances.
Source code in asf/metrics/baselines.py 
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
def single_best_solver(performance: pd.DataFrame, maximize: bool = False) -> float:
    """
    Selects the single best solver across all instances based on the aggregated performance.

    Args:
        schedules (pd.DataFrame): The schedules to evaluate (not used in this function).
        performance (pd.DataFrame): The performance data for the algorithms.
        maximize (bool): Whether to maximize or minimize the performance.

    Returns:
        float: The best aggregated performance value across all instances.
    """
    perf_sum = performance.sum(axis=0)
    if maximize:
        return perf_sum.max()
    else:
        return perf_sum.min()

virtual_best_solver(performance, maximize=False)

Selects the virtual best solver for each instance by choosing the best performance per instance.

Parameters:

Name Type Description Default
schedules DataFrame

The schedules to evaluate (not used in this function).

 required
performance DataFrame

The performance data for the algorithms.

 required
maximize bool

Whether to maximize or minimize the performance.

 False

Returns:

Name Type Description
float float

The sum of the best performance values for each instance.
Source code in asf/metrics/baselines.py 
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
def virtual_best_solver(performance: pd.DataFrame, maximize: bool = False) -> float:
    """
    Selects the virtual best solver for each instance by choosing the best performance per instance.

    Args:
        schedules (pd.DataFrame): The schedules to evaluate (not used in this function).
        performance (pd.DataFrame): The performance data for the algorithms.
        maximize (bool): Whether to maximize or minimize the performance.

    Returns:
        float: The sum of the best performance values for each instance.
    """
    if maximize:
        return performance.max(axis=1).sum()
    else:
        return performance.min(axis=1).sum()