timesead.evaluation.ts_precision_recall

Functions

constant_bias_fn(→ float)	Compute the overlap size for a constant bias function that assigns the same weight to all positions.
back_bias_fn(→ float)	Compute the overlap size for a bias function that assigns the more weight to predictions towards the back of a
front_bias_fn(→ float)	Compute the overlap size for a bias function that assigns the more weight to predictions towards the front of a
middle_bias_fn(→ float)	Compute the overlap size for a bias function that assigns the more weight to predictions in the middle of a
inverse_proportional_cardinality_fn(→ float)	Cardinality function that assigns an inversely proportional weight to predictions within a single ground-truth
improved_cardinality_fn(cardinality, gt_length)	Recall-consistent cardinality function introduced by [Wagner2023] that assigns lower weight to ground-truth windows
compute_window_indices(→ List[Tuple[int, int]])	Compute a list of indices where anomaly windows begin and end.
ts_precision_and_recall(→ Tuple[float, float])	Computes precision and recall for time series as defined in [Tatbul2018].

Module Contents

timesead.evaluation.ts_precision_recall.constant_bias_fn(inputs: torch.Tensor) → float

Compute the overlap size for a constant bias function that assigns the same weight to all positions.

This functions computes

\[\omega(\text{inputs}) = \frac{1}{n} \sum_{i = 1}^{n} \text{inputs}_i,\]

where \(n = \lvert \text{inputs} \rvert\).

Note

To improve the runtime of our algorithm, we calculate the overlap \(\omega\) directly as part of the bias function.

Parameters:

inputs (torch.Tensor) – A 1-D Tensor containing the predictions inside a ground-truth window.

Returns:

The overlap \(\omega\).

Return type:

float

timesead.evaluation.ts_precision_recall.back_bias_fn(inputs: torch.Tensor) → float

Compute the overlap size for a bias function that assigns the more weight to predictions towards the back of a ground-truth anomaly window.

This functions computes

\[\omega(\text{inputs}) = \frac{2}{n * (n + 1)} \sum_{i = 1}^{n} \text{inputs}_i \cdot i,\]

where \(n = \lvert \text{inputs} \rvert\).

Note

To improve the runtime of our algorithm, we calculate the overlap \(\omega\) directly as part of the bias function.

Parameters:

inputs (torch.Tensor) – A 1-D Tensor containing the predictions inside a ground-truth window.

Returns:

The overlap \(\omega\).

Return type:

float

timesead.evaluation.ts_precision_recall.front_bias_fn(inputs: torch.Tensor) → float

Compute the overlap size for a bias function that assigns the more weight to predictions towards the front of a ground-truth anomaly window.

This functions computes

\[\omega(\text{inputs}) = \frac{2}{n * (n + 1)} \sum_{i = 1}^{n} \text{inputs}_i \cdot (n + 1 - i),\]

where \(n = \lvert \text{inputs} \rvert\).

Note

To improve the runtime of our algorithm, we calculate the overlap \(\omega\) directly as part of the bias function.

Parameters:

inputs (torch.Tensor) – A 1-D Tensor containing the predictions inside a ground-truth window.

Returns:

The overlap \(\omega\).

Return type:

float

timesead.evaluation.ts_precision_recall.middle_bias_fn(inputs: torch.Tensor) → float

Compute the overlap size for a bias function that assigns the more weight to predictions in the middle of a ground-truth anomaly window.

This functions computes

\[\begin{split}\omega(\text{inputs}) = \frac{2}{m * (m + 1) + (n - m) * (n - m + 1)} \sum_{i = 1}^{n} \text{inputs}_i \cdot \begin{cases} i & \text{if } i \leq m\\ (n + 1 - i) & \text{otherwise} \end{cases},\end{split}\]

where \(n = \lvert \text{inputs} \rvert\) and \(m = \lceil \frac{n}{2} \rceil\).

Note

To improve the runtime of our algorithm, we calculate the overlap \(\omega\) directly as part of the bias function.

Parameters:

inputs (torch.Tensor) – A 1-D Tensor containing the predictions inside a ground-truth window.

Returns:

The overlap \(\omega\).

Return type:

float

timesead.evaluation.ts_precision_recall.inverse_proportional_cardinality_fn(cardinality: int, gt_length: int) → float

Cardinality function that assigns an inversely proportional weight to predictions within a single ground-truth window.

This is the default cardinality function recommended in [Tatbul2018].

Note

This function leads to a metric that is not recall-consistent! Please see [Wagner2023] for more details.

Parameters:

• cardinality (int) – Number of predicted windows that overlap the ground-truth window in question.

• gt_length (int) – Length of the ground-truth window (unused).

Returns:

The cardinality factor \(\frac{1}{\text{cardinality}}\).

Return type:

float

[Tatbul2018] (1,2,3,4)

N. Tatbul, T.J. Lee, S. Zdonik, M. Alam, J. Gottschlich. Precision and recall for time series. Advances in neural information processing systems. 2018;31.

[Wagner2023] (1,2,3,4)

D. Wagner, T. Michels, F.C.F. Schulz, A. Nair, M. Rudolph, and M. Kloft. TimeSeAD: Benchmarking Deep Multivariate Time-Series Anomaly Detection. Transactions on Machine Learning Research (TMLR), (to appear) 2023.

timesead.evaluation.ts_precision_recall.improved_cardinality_fn(cardinality: int, gt_length: int)

Recall-consistent cardinality function introduced by [Wagner2023] that assigns lower weight to ground-truth windows that overlap with many predicted windows.

This function computes

\[\left(\frac{\text{gt_length} - 1}{\text{gt_length}}\right)^{\text{cardinality} - 1}.\]

Parameters:

• cardinality (int) – Number of predicted windows that overlap the ground-truth window in question.

• gt_length (int) – Length of the ground-truth window.

Returns:

The cardinality factor.

timesead.evaluation.ts_precision_recall.compute_window_indices(binary_labels: torch.Tensor) → List[Tuple[int, int]]

Compute a list of indices where anomaly windows begin and end.

Parameters:

binary_labels (torch.Tensor) – A 1-D Tensor containing 1 for an anomalous time step or 0 otherwise.

Returns:

A list of tuples (start, end) for each anomaly window in binary_labels, where start is the index at which the window starts and end is the first index after the end of the window.

Return type:

List[Tuple[int, int]]

timesead.evaluation.ts_precision_recall.ts_precision_and_recall(anomalies: torch.Tensor, predictions: torch.Tensor, alpha: float = 0, recall_bias_fn: Callable[[torch.Tensor], float] = constant_bias_fn, recall_cardinality_fn: Callable[[int], float] = inverse_proportional_cardinality_fn, precision_bias_fn: Callable | None = None, precision_cardinality_fn: Callable | None = None, anomaly_ranges: List[Tuple[int, int]] | None = None, prediction_ranges: List[Tuple[int, int]] | None = None, weighted_precision: bool = False) → Tuple[float, float]

Computes precision and recall for time series as defined in [Tatbul2018].

Note

The default parameters for this function correspond to the defaults recommended in [Tatbul2018]. However, those might not be desirable in most cases, please see [Wagner2023] for a detailed discussion.

Parameters:

• anomalies (torch.Tensor) – Binary 1-D Tensor of shape (length,) containing the true labels.

• predictions (torch.Tensor) – Binary 1-D Tensor of shape (length,) containing the predicted labels.

• alpha (float) – Weight for existence term in recall.

• recall_bias_fn (Callable[[torch.Tensor], float]) – Function that computes the bias term for a given ground-truth window.

• recall_cardinality_fn (Callable[[int], float]) – Function that compute the cardinality factor for a given ground-truth window.

• precision_bias_fn (Optional[Callable]) – Function that computes the bias term for a given predicted window. If None, this will be the same as recall_bias_function.

• precision_cardinality_fn (Optional[Callable]) – Function that computes the cardinality factor for a given predicted window. If None, this will be the same as recall_cardinality_function.

• weighted_precision (bool) – If True, the precision score of a predicted window will be weighted with the length of the window in the final score. Otherwise, each window will have the same weight.

• anomaly_ranges (Optional[List[Tuple[int, int]]]) – A list of tuples (start, end) for each anomaly window in anomalies, where start is the index at which the window starts and end is the first index after the end of the window. This can be None, in which case the list is computed automatically from anomalies.

• prediction_ranges (Optional[List[Tuple[int, int]]]) – A list of tuples (start, end) for each anomaly window in predictions, where start is the index at which the window starts and end is the first index after the end of the window. This can be None, in which case the list is computed automatically from predictions.

Returns:

A tuple consisting of the time-series precision and recall for the given labels.

Return type:

Tuple[float, float]