timesead.evaluation.ts_precision_recall ======================================= .. py:module:: timesead.evaluation.ts_precision_recall Functions --------- .. autoapisummary:: timesead.evaluation.ts_precision_recall.constant_bias_fn timesead.evaluation.ts_precision_recall.back_bias_fn timesead.evaluation.ts_precision_recall.front_bias_fn timesead.evaluation.ts_precision_recall.middle_bias_fn timesead.evaluation.ts_precision_recall.inverse_proportional_cardinality_fn timesead.evaluation.ts_precision_recall.improved_cardinality_fn timesead.evaluation.ts_precision_recall.compute_window_indices timesead.evaluation.ts_precision_recall.ts_precision_and_recall Module Contents --------------- .. py:function:: constant_bias_fn(inputs: torch.Tensor) -> float Compute the overlap size for a constant bias function that assigns the same weight to all positions. This functions computes .. math:: \omega(\text{inputs}) = \frac{1}{n} \sum_{i = 1}^{n} \text{inputs}_i, where :math:`n = \lvert \text{inputs} \rvert`. .. note:: To improve the runtime of our algorithm, we calculate the overlap :math:`\omega` directly as part of the bias function. :param inputs: A 1-D :class:`~torch.Tensor` containing the predictions inside a ground-truth window. :return: The overlap :math:`\omega`. .. py:function:: back_bias_fn(inputs: torch.Tensor) -> float Compute the overlap size for a bias function that assigns the more weight to predictions towards the back of a ground-truth anomaly window. This functions computes .. math:: \omega(\text{inputs}) = \frac{2}{n * (n + 1)} \sum_{i = 1}^{n} \text{inputs}_i \cdot i, where :math:`n = \lvert \text{inputs} \rvert`. .. note:: To improve the runtime of our algorithm, we calculate the overlap :math:`\omega` directly as part of the bias function. :param inputs: A 1-D :class:`~torch.Tensor` containing the predictions inside a ground-truth window. :return: The overlap :math:`\omega`. .. py:function:: front_bias_fn(inputs: torch.Tensor) -> float Compute the overlap size for a bias function that assigns the more weight to predictions towards the front of a ground-truth anomaly window. This functions computes .. math:: \omega(\text{inputs}) = \frac{2}{n * (n + 1)} \sum_{i = 1}^{n} \text{inputs}_i \cdot (n + 1 - i), where :math:`n = \lvert \text{inputs} \rvert`. .. note:: To improve the runtime of our algorithm, we calculate the overlap :math:`\omega` directly as part of the bias function. :param inputs: A 1-D :class:`~torch.Tensor` containing the predictions inside a ground-truth window. :return: The overlap :math:`\omega`. .. py:function:: middle_bias_fn(inputs: torch.Tensor) -> float Compute the overlap size for a bias function that assigns the more weight to predictions in the middle of a ground-truth anomaly window. This functions computes .. math:: \omega(\text{inputs}) = \frac{2}{m * (m + 1) + (n - m) * (n - m + 1)} \sum_{i = 1}^{n} \text{inputs}_i \cdot \begin{cases} i & \text{if } i \leq m\\ (n + 1 - i) & \text{otherwise} \end{cases}, where :math:`n = \lvert \text{inputs} \rvert` and :math:`m = \lceil \frac{n}{2} \rceil`. .. note:: To improve the runtime of our algorithm, we calculate the overlap :math:`\omega` directly as part of the bias function. :param inputs: A 1-D :class:`~torch.Tensor` containing the predictions inside a ground-truth window. :return: The overlap :math:`\omega`. .. py:function:: inverse_proportional_cardinality_fn(cardinality: int, gt_length: int) -> float Cardinality function that assigns an inversely proportional weight to predictions within a single ground-truth window. This is the default cardinality function recommended in [Tatbul2018]_. .. note:: This function leads to a metric that is not recall-consistent! Please see [Wagner2023]_ for more details. :param cardinality: Number of predicted windows that overlap the ground-truth window in question. :param gt_length: Length of the ground-truth window (unused). :return: The cardinality factor :math:`\frac{1}{\text{cardinality}}`. .. [Tatbul2018] N. Tatbul, T.J. Lee, S. Zdonik, M. Alam, J. Gottschlich. Precision and recall for time series. Advances in neural information processing systems. 2018;31. .. [Wagner2023] D. Wagner, T. Michels, F.C.F. Schulz, A. Nair, M. Rudolph, and M. Kloft. TimeSeAD: Benchmarking Deep Multivariate Time-Series Anomaly Detection. Transactions on Machine Learning Research (TMLR), (to appear) 2023. .. py:function:: improved_cardinality_fn(cardinality: int, gt_length: int) Recall-consistent cardinality function introduced by [Wagner2023]_ that assigns lower weight to ground-truth windows that overlap with many predicted windows. This function computes .. math:: \left(\frac{\text{gt_length} - 1}{\text{gt_length}}\right)^{\text{cardinality} - 1}. :param cardinality: Number of predicted windows that overlap the ground-truth window in question. :param gt_length: Length of the ground-truth window. :return: The cardinality factor. .. py:function:: compute_window_indices(binary_labels: torch.Tensor) -> List[Tuple[int, int]] Compute a list of indices where anomaly windows begin and end. :param binary_labels: A 1-D :class:`~torch.Tensor` containing ``1`` for an anomalous time step or ``0`` otherwise. :return: A list of tuples ``(start, end)`` for each anomaly window in ``binary_labels``, where ``start`` is the index at which the window starts and ``end`` is the first index after the end of the window. .. py:function:: ts_precision_and_recall(anomalies: torch.Tensor, predictions: torch.Tensor, alpha: float = 0, recall_bias_fn: Callable[[torch.Tensor], float] = constant_bias_fn, recall_cardinality_fn: Callable[[int], float] = inverse_proportional_cardinality_fn, precision_bias_fn: Optional[Callable] = None, precision_cardinality_fn: Optional[Callable] = None, anomaly_ranges: Optional[List[Tuple[int, int]]] = None, prediction_ranges: Optional[List[Tuple[int, int]]] = None, weighted_precision: bool = False) -> Tuple[float, float] Computes precision and recall for time series as defined in [Tatbul2018]_. .. note:: The default parameters for this function correspond to the defaults recommended in [Tatbul2018]_. However, those might not be desirable in most cases, please see [Wagner2023]_ for a detailed discussion. :param anomalies: Binary 1-D :class:`~torch.Tensor` of shape ``(length,)`` containing the true labels. :param predictions: Binary 1-D :class:`~torch.Tensor` of shape ``(length,)`` containing the predicted labels. :param alpha: Weight for existence term in recall. :param recall_bias_fn: Function that computes the bias term for a given ground-truth window. :param recall_cardinality_fn: Function that compute the cardinality factor for a given ground-truth window. :param precision_bias_fn: Function that computes the bias term for a given predicted window. If ``None``, this will be the same as ``recall_bias_function``. :param precision_cardinality_fn: Function that computes the cardinality factor for a given predicted window. If ``None``, this will be the same as ``recall_cardinality_function``. :param weighted_precision: If True, the precision score of a predicted window will be weighted with the length of the window in the final score. Otherwise, each window will have the same weight. :param anomaly_ranges: A list of tuples ``(start, end)`` for each anomaly window in ``anomalies``, where ``start`` is the index at which the window starts and ``end`` is the first index after the end of the window. This can be ``None``, in which case the list is computed automatically from ``anomalies``. :param prediction_ranges: A list of tuples ``(start, end)`` for each anomaly window in ``predictions``, where ``start`` is the index at which the window starts and ``end`` is the first index after the end of the window. This can be ``None``, in which case the list is computed automatically from ``predictions``. :return: A tuple consisting of the time-series precision and recall for the given labels.