Skip to content

Metrics

MeanAveragePrecisionMetric

Bases: BaseMetric

A class used to compute the Mean Average Precision (mAP) metric.

mAP is a popular metric for object detection tasks, measuring the average precision across all classes and IoU thresholds.

Source code in maestro/trainer/common/utils/metrics.py 
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
class MeanAveragePrecisionMetric(BaseMetric):
    """A class used to compute the Mean Average Precision (mAP) metric.

    mAP is a popular metric for object detection tasks, measuring the average precision
    across all classes and IoU thresholds.
    """

    name = "mean_average_precision"

    def describe(self) -> list[str]:
        """Returns a list of metric names that this class will compute.

        Returns:
            List[str]: A list of metric names.
        """
        return ["map50:95", "map50", "map75"]

    def compute(self, targets: list[sv.Detections], predictions: list[sv.Detections]) -> dict[str, float]:
        """Computes the mAP metrics based on the targets and predictions.

        Args:
            targets (List[sv.Detections]): The ground truth detections.
            predictions (List[sv.Detections]): The predicted detections.

        Returns:
            Dict[str, float]: A dictionary of computed mAP metrics with metric names as
                keys and their values.
        """
        result = MeanAveragePrecision().update(targets=targets, predictions=predictions).compute()
        return {"map50:95": result.map50_95, "map50": result.map50, "map75": result.map75}

Functions

compute(targets, predictions)

Computes the mAP metrics based on the targets and predictions.

Parameters:

Name Type Description Default

targets

List[Detections]

The ground truth detections.

 required

predictions

List[Detections]

The predicted detections.

 required

Returns:

Type Description
dict[str, float]

Dict[str, float]: A dictionary of computed mAP metrics with metric names as keys and their values.
Source code in maestro/trainer/common/utils/metrics.py 
66
67
68
69
70
71
72
73
74
75
76
77
78
def compute(self, targets: list[sv.Detections], predictions: list[sv.Detections]) -> dict[str, float]:
    """Computes the mAP metrics based on the targets and predictions.

    Args:
        targets (List[sv.Detections]): The ground truth detections.
        predictions (List[sv.Detections]): The predicted detections.

    Returns:
        Dict[str, float]: A dictionary of computed mAP metrics with metric names as
            keys and their values.
    """
    result = MeanAveragePrecision().update(targets=targets, predictions=predictions).compute()
    return {"map50:95": result.map50_95, "map50": result.map50, "map75": result.map75}

describe()

Returns a list of metric names that this class will compute.

Returns:

Type Description
list[str]

List[str]: A list of metric names.
Source code in maestro/trainer/common/utils/metrics.py 
58
59
60
61
62
63
64
def describe(self) -> list[str]:
    """Returns a list of metric names that this class will compute.

    Returns:
        List[str]: A list of metric names.
    """
    return ["map50:95", "map50", "map75"]

WordErrorRateMetric

Bases: BaseMetric

A class used to compute the Word Error Rate (WER) metric.

WER measures the edit distance between predicted and reference transcriptions at the word level, commonly used in speech recognition and machine translation.

Source code in maestro/trainer/common/utils/metrics.py 
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
class WordErrorRateMetric(BaseMetric):
    """A class used to compute the Word Error Rate (WER) metric.

    WER measures the edit distance between predicted and reference transcriptions
    at the word level, commonly used in speech recognition and machine translation.
    """

    name = "word_error_rate"

    def describe(self) -> list[str]:
        """Returns a list of metric names that this class will compute.

        Returns:
            List[str]: A list of metric names.
        """
        return ["wer"]

    def compute(self, targets: list[str], predictions: list[str]) -> dict[str, float]:
        """Computes the WER metric based on the targets and predictions.

        Args:
            targets (List[str]): The ground truth texts (references), where each element
                represents the reference text for the corresponding prediction.
            predictions (List[str]): The predicted texts (hypotheses) to be evaluated.

        Returns:
            Dict[str, float]: A dictionary containing the computed WER score, with the
                metric name ("wer") as the key and its value as the score.
        """
        if len(targets) != len(predictions):
            raise ValueError("The number of targets and predictions must be the same.")

        total_wer = 0.0
        count = len(targets)

        for target, prediction in zip(targets, predictions):
            total_wer += wer(target, prediction)

        average_wer = total_wer / count if count > 0 else 0.0
        return {"wer": average_wer}

Functions

compute(targets, predictions)

Computes the WER metric based on the targets and predictions.

Parameters:

Name Type Description Default

targets

List[str]

The ground truth texts (references), where each element represents the reference text for the corresponding prediction.

 required

predictions

List[str]

The predicted texts (hypotheses) to be evaluated.

 required

Returns:

Type Description
dict[str, float]

Dict[str, float]: A dictionary containing the computed WER score, with the metric name ("wer") as the key and its value as the score.
Source code in maestro/trainer/common/utils/metrics.py 
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
def compute(self, targets: list[str], predictions: list[str]) -> dict[str, float]:
    """Computes the WER metric based on the targets and predictions.

    Args:
        targets (List[str]): The ground truth texts (references), where each element
            represents the reference text for the corresponding prediction.
        predictions (List[str]): The predicted texts (hypotheses) to be evaluated.

    Returns:
        Dict[str, float]: A dictionary containing the computed WER score, with the
            metric name ("wer") as the key and its value as the score.
    """
    if len(targets) != len(predictions):
        raise ValueError("The number of targets and predictions must be the same.")

    total_wer = 0.0
    count = len(targets)

    for target, prediction in zip(targets, predictions):
        total_wer += wer(target, prediction)

    average_wer = total_wer / count if count > 0 else 0.0
    return {"wer": average_wer}

describe()

Returns a list of metric names that this class will compute.

Returns:

Type Description
list[str]

List[str]: A list of metric names.
Source code in maestro/trainer/common/utils/metrics.py 
90
91
92
93
94
95
96
def describe(self) -> list[str]:
    """Returns a list of metric names that this class will compute.

    Returns:
        List[str]: A list of metric names.
    """
    return ["wer"]

CharacterErrorRateMetric

Bases: BaseMetric

A class used to compute the Character Error Rate (CER) metric.

CER is similar to WER but operates at the character level, making it useful for tasks like optical character recognition (OCR) and handwriting recognition.

Source code in maestro/trainer/common/utils/metrics.py 
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
class CharacterErrorRateMetric(BaseMetric):
    """A class used to compute the Character Error Rate (CER) metric.

    CER is similar to WER but operates at the character level, making it useful for
    tasks like optical character recognition (OCR) and handwriting recognition.
    """

    name = "character_error_rate"

    def describe(self) -> list[str]:
        """Returns a list of metric names that this class will compute.

        Returns:
            List[str]: A list of metric names.
        """
        return ["cer"]

    def compute(self, targets: list[str], predictions: list[str]) -> dict[str, float]:
        """Computes the CER metric based on the targets and predictions.

        Args:
            targets (List[str]): The ground truth texts (references), where each element
                represents the reference text for the corresponding prediction.
            predictions (List[str]): The predicted texts (hypotheses) to be evaluated.

        Returns:
            Dict[str, float]: A dictionary containing the computed CER score, with the
                metric name ("cer") as the key and its value as the score.
        """
        if len(targets) != len(predictions):
            raise ValueError("The number of targets and predictions must be the same.")

        total_cer = 0.0
        count = len(targets)

        for target, prediction in zip(targets, predictions):
            total_cer += cer(target, prediction)

        average_cer = total_cer / count if count > 0 else 0.0
        return {"cer": average_cer}

Functions

compute(targets, predictions)

Computes the CER metric based on the targets and predictions.

Parameters:

Name Type Description Default

targets

List[str]

The ground truth texts (references), where each element represents the reference text for the corresponding prediction.

 required

predictions

List[str]

The predicted texts (hypotheses) to be evaluated.

 required

Returns:

Type Description
dict[str, float]

Dict[str, float]: A dictionary containing the computed CER score, with the metric name ("cer") as the key and its value as the score.
Source code in maestro/trainer/common/utils/metrics.py 
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
def compute(self, targets: list[str], predictions: list[str]) -> dict[str, float]:
    """Computes the CER metric based on the targets and predictions.

    Args:
        targets (List[str]): The ground truth texts (references), where each element
            represents the reference text for the corresponding prediction.
        predictions (List[str]): The predicted texts (hypotheses) to be evaluated.

    Returns:
        Dict[str, float]: A dictionary containing the computed CER score, with the
            metric name ("cer") as the key and its value as the score.
    """
    if len(targets) != len(predictions):
        raise ValueError("The number of targets and predictions must be the same.")

    total_cer = 0.0
    count = len(targets)

    for target, prediction in zip(targets, predictions):
        total_cer += cer(target, prediction)

    average_cer = total_cer / count if count > 0 else 0.0
    return {"cer": average_cer}

describe()

Returns a list of metric names that this class will compute.

Returns:

Type Description
list[str]

List[str]: A list of metric names.
Source code in maestro/trainer/common/utils/metrics.py 
132
133
134
135
136
137
138
def describe(self) -> list[str]:
    """Returns a list of metric names that this class will compute.

    Returns:
        List[str]: A list of metric names.
    """
    return ["cer"]