timesead.evaluation.evaluator ============================= .. py:module:: timesead.evaluation.evaluator Classes ------- .. autoapisummary:: timesead.evaluation.evaluator.Evaluator Module Contents --------------- .. py:class:: Evaluator A class that can compute several evaluation metrics for a dataset. Each method returns the score as a single float, but it can also return additional information in a dict. .. py:method:: auc(labels: torch.Tensor, scores: torch.Tensor) -> Tuple[float, Dict[str, Any]] Compute the classic point-wise area under the receiver operating characteristic curve. This will return a value between 0 and 1 where 1 indicates a perfect classifier. .. seealso:: Scikit-learns's :func:`~sklearn.metrics.roc_auc_score` function. :param labels: A 1-D :class:`~torch.Tensor` containing the ground-truth labels. 1 corresponds to an anomaly, 0 means that the point is normal. :param scores: A 1-D :class:`~torch.Tensor` containing the scores returned by an :class:`~timesead.models.common.AnomalyDetector`. :return: A tuple consisting of the AUC score and an empty dict. .. py:method:: f1_score(labels: torch.Tensor, scores: torch.Tensor, pos_label: int = 1) -> Tuple[float, Dict[str, Any]] Compute the classic point-wise F1 score. This will return a value between 0 and 1 where 1 indicates a perfect classifier. .. seealso:: Scikit-learn's :func:`~sklearn.metrics.f1_score` function. :param labels: A 1-D :class:`~torch.Tensor` containing the ground-truth labels. 1 corresponds to an anomaly, 0 means that the point is normal. :param scores: A 1-D :class:`~torch.Tensor` containing binary predictions of whether a point is an anomaly or not. :param pos_label: Class to report. :return: A tuple consisting of the F1 score and an empty dict. .. py:method:: best_fbeta_score(labels: torch.Tensor, scores: torch.Tensor, beta: float) -> Tuple[float, Dict[str, Any]] Compute the classic point-wise :math:`F_{\beta}` score. This method will apply all possible thresholds to the values in ``scores`` and compute the :math:`F_{\beta}` score for the resulting binary predictions. It then returns the highest score. .. seealso:: Scikit-learn's :func:`~sklearn.metrics.fbeta_score` function. :param labels: A 1-D :class:`~torch.Tensor` containing the ground-truth labels. 1 corresponds to an anomaly, 0 means that the point is normal. :param scores: A 1-D :class:`~torch.Tensor` containing the scores returned by an :class:`~timesead.models.common.AnomalyDetector`. :param beta: Positive number that determines the trade-off between precision and recall when computing the F-score. :math:`\beta = 1` assigns equal weight to both while :math:`\beta < 1` emphasizes precision and vice versa. :return: A tuple consisting of the best :math:`F_{\beta}` score and a dict containing the threshold that produced the maximal score. .. py:method:: best_f1_score(labels: torch.Tensor, scores: torch.Tensor) -> Tuple[float, Dict[str, Any]] Compute the classic point-wise :math:`F_{1}` score. This method will apply all possible thresholds to the values in ``scores`` and compute the :math:`F_{1}` score for the resulting binary predictions. It then returns the highest score. .. seealso:: Scikit-learn's :func:`~sklearn.metrics.f1_score` function. :param labels: A 1-D :class:`~torch.Tensor` containing the ground-truth labels. 1 corresponds to an anomaly, 0 means that the point is normal. :param scores: A 1-D :class:`~torch.Tensor` containing the scores returned by an :class:`~timesead.models.common.AnomalyDetector`. :return: A tuple consisting of the best :math:`F_{1}` score and a dict containing the threshold that produced the maximal score. .. py:method:: auprc(labels: torch.Tensor, scores: torch.Tensor, integration: str = 'trapezoid') -> Tuple[float, Dict[str, Any]] Compute the classic point-wise area under the precision-recall curve. This will return a value between 0 and 1 where 1 indicates a perfect classifier. .. seealso:: Scikit-learn's :func:`~sklearn.metrics.average_precision` function. Scikit-learn's :func:`~sklearn.metrics.precision_recall_curve` function. :param labels: A 1-D :class:`~torch.Tensor` containing the ground-truth labels. 1 corresponds to an anomaly, 0 means that the point is normal. :param scores: A 1-D :class:`~torch.Tensor` containing the scores returned by an :class:`~timesead.models.common.AnomalyDetector`. :param integration: Method to use for computing the area under the curve. ``'riemann'`` corresponds to a simple Riemann sum, whereas ``'trapezoid'`` uses the trapezoidal rule. :return: A tuple consisting of the AuPRC score and an empty dict. .. py:method:: average_precision(labels: torch.Tensor, scores: torch.Tensor) -> Tuple[float, Dict[str, Any]] Compute the classic point-wise average precision score. .. note:: This is just a shorthand for :meth:`auprc` with ``integration='riemann'``. .. seealso:: Scikit-learn's :func:`~sklearn.metrics.average_precision` function. :param labels: A 1-D :class:`~torch.Tensor` containing the ground-truth labels. 1 corresponds to an anomaly, 0 means that the point is normal. :param scores: A 1-D :class:`~torch.Tensor` containing the scores returned by an :class:`~timesead.models.common.AnomalyDetector` :return: A tuple consisting of the average precision score and an empty dict. .. py:method:: ts_auprc(labels: torch.Tensor, scores: torch.Tensor, integration='trapezoid', weighted_precision: bool = True) -> Tuple[float, Dict[str, Any]] Compute the area under the precision-recall curve using precision and recall for time series [Tatbul2018]_. .. note:: This function uses the improved cardinality function described in [Wagner2023]_. :param labels: A 1-D :class:`~torch.Tensor` containing the ground-truth labels. 1 corresponds to an anomaly, 0 means that the point is normal. :param scores: A 1-D :class:`~torch.Tensor` containing the scores returned by an :class:`~timesead.models.common.AnomalyDetector` :param integration: Method to use for computing the area under the curve. ``'riemann'`` corresponds to a simple Riemann sum, whereas ``'trapezoid'`` uses the trapezoidal rule. :param weighted_precision: If ``True``, the precision score of a predicted window will be weighted with the length of the window in the final score. Otherwise, each window will have the same weight. :return: A tuple consisting of the AuPRC score and an empty dict. .. [Tatbul2018] N. Tatbul, T.J. Lee, S. Zdonik, M. Alam, J. Gottschlich. Precision and recall for time series. Advances in neural information processing systems. 2018;31. .. [Wagner2023] D. Wagner, T. Michels, F.C.F. Schulz, A. Nair, M. Rudolph, and M. Kloft. TimeSeAD: Benchmarking Deep Multivariate Time-Series Anomaly Detection. Transactions on Machine Learning Research (TMLR), (to appear) 2023. .. py:method:: ts_average_precision(labels: torch.Tensor, scores: torch.Tensor, weighted_precision: bool = True) -> Tuple[float, Dict[str, Any]] Compute the average precision score using precision and recall for time series [Tatbul2018]_. .. note:: This is just a shorthand for :meth:`ts_auprc` with ``integration='riemann'``. :param labels: A 1-D :class:`~torch.Tensor` containing the ground-truth labels. 1 corresponds to an anomaly, 0 means that the point is normal. :param scores: A 1-D :class:`~torch.Tensor` containing the scores returned by an :class:`~timesead.models.common.AnomalyDetector` :param weighted_precision: If ``True``, the precision score of a predicted window will be weighted with the length of the window in the final score. Otherwise, each window will have the same weight. :return: A tuple consisting of the average precision score and an empty dict. .. py:method:: ts_auprc_unweighted(labels: torch.Tensor, scores: torch.Tensor) -> Tuple[float, Dict[str, Any]] Compute the area under the precision-recall curve using precision and recall for time series [Tatbul2018]_. .. note:: This is just a shorthand for :meth:`ts_auprc` with ``integration='riemann'`` and ``weighted_precision=False``. :param labels: A 1-D :class:`~torch.Tensor` containing the ground-truth labels. 1 corresponds to an anomaly, 0 means that the point is normal. :param scores: A 1-D :class:`~torch.Tensor` containing the scores returned by an :class:`~timesead.models.common.AnomalyDetector` :return: A tuple consisting of the AuPRC score and an empty dict. .. py:method:: best_ts_fbeta_score(labels: torch.Tensor, scores: torch.Tensor, beta: float) -> Tuple[float, Dict[str, Any]] Compute the :math:`F_{\beta}` score using precision and recall for time series [Tatbul2018]_. This method will apply all possible thresholds to the values in ``scores`` and compute the :math:`F_{\beta}` score for the resulting binary predictions. It then returns the highest score. .. note:: This function uses the improved cardinality function and weighted precision as described in [Wagner2023]_. :param labels: A 1-D :class:`~torch.Tensor` containing the ground-truth labels. 1 corresponds to an anomaly, 0 means that the point is normal. :param scores: A 1-D :class:`~torch.Tensor` containing the scores returned by an :class:`~timesead.models.common.AnomalyDetector` :param beta: Positive number that determines the trade-off between precision and recall when computing the F-score. :math:`\beta = 1` assigns equal weight to both while :math:`\beta < 1` emphasizes precision and vice versa. :return: A tuple consisting of the best :math:`F_{\beta}` score and a dict containing the threshold, recall and precision that produced the maximal score. .. py:method:: best_ts_fbeta_score_classic(labels: torch.Tensor, scores: torch.Tensor, beta: float) -> Tuple[float, Dict[str, Any]] Compute the :math:`F_{\beta}` score using precision and recall for time series [Tatbul2018]_. This method will apply all possible thresholds to the values in ``scores`` and compute the :math:`F_{\beta}` score for the resulting binary predictions. It then returns the highest score. .. note:: This function uses the default cardinality function (:math:`\frac[1}{x}`) and unweighted precision, i.e., the default parameters described in [Tatbul2018]_. :param labels: A 1-D :class:`~torch.Tensor` containing the ground-truth labels. 1 corresponds to an anomaly, 0 means that the point is normal. :param scores: A 1-D :class:`~torch.Tensor` containing the scores returned by an :class:`~timesead.models.common.AnomalyDetector` :param beta: Positive number that determines the trade-off between precision and recall when computing the F-score. :math:`\beta = 1` assigns equal weight to both while :math:`\beta < 1` emphasizes precision and vice versa. :return: A tuple consisting of the best :math:`F_{\beta}` score and a dict containing the threshold, recall and precision that produced the maximal score. .. py:method:: best_ts_f1_score(labels: torch.Tensor, scores: torch.Tensor) -> Tuple[float, Dict[str, Any]] Compute the :math:`F_{1}` score using precision and recall for time series [Tatbul2018]_. This method will apply all possible thresholds to the values in ``scores`` and compute the :math:`F_{1}` score for the resulting binary predictions. It then returns the highest score. .. note:: This function uses the improved cardinality function and weighted precision as described in [Wagner2023]_. :param labels: A 1-D :class:`~torch.Tensor` containing the ground-truth labels. 1 corresponds to an anomaly, 0 means that the point is normal. :param scores: A 1-D :class:`~torch.Tensor` containing the scores returned by an :class:`~timesead.models.common.AnomalyDetector` :return: A tuple consisting of the best :math:`F_{1}` score and a dict containing the threshold, recall and precision that produced the maximal score. .. py:method:: best_ts_f1_score_classic(labels: torch.Tensor, scores: torch.Tensor) -> Tuple[float, Dict[str, Any]] Compute the :math:`F_{1}` score using precision and recall for time series [Tatbul2018]_. This method will apply all possible thresholds to the values in ``scores`` and compute the :math:`F_{1}` score for the resulting binary predictions. It then returns the highest score. .. note:: This function uses the default cardinality function (:math:`\frac[1}{x}`) and unweighted precision, i.e., the default parameters described in [Tatbul2018]_. :param labels: A 1-D :class:`~torch.Tensor` containing the ground-truth labels. 1 corresponds to an anomaly, 0 means that the point is normal. :param scores: A 1-D :class:`~torch.Tensor` containing the scores returned by an :class:`~timesead.models.common.AnomalyDetector` :return: A tuple consisting of the best :math:`F_{1}` score and a dict containing the threshold, recall and precision that produced the maximal score.