pytext.metric_reporters package¶
Submodules¶
pytext.metric_reporters.channel module¶
-
class
pytext.metric_reporters.channel.
Channel
(stages: Tuple[pytext.common.constants.Stage, ...] = (<Stage.TRAIN: 'Training'>, <Stage.EVAL: 'Evaluation'>, <Stage.TEST: 'Test'>))[source]¶ Bases:
object
Channel defines how to format and report the result of a PyText job to an output stream.
-
stages
¶ in which stages the report will be triggered, default is all stages, which includes train, eval, test
-
report
(stage, epoch, metrics, model_select_metric, loss, preds, targets, scores, context, meta, *args)[source]¶ Defines how to format and report data to the output channel.
Parameters: - stage (Stage) – train, eval or test
- epoch (int) – current epoch
- metrics (Any) – all metrics
- model_select_metric (double) – a single numeric metric to pick best model
- loss (double) – average loss
- preds (List[Any]) – list of predictions
- targets (List[Any]) – list of targets
- scores (List[Any]) – list of scores
- context (Dict[str, List[Any]]) – dict of any additional context data, each context is a list of data that maps to each example
- meta (Dict[str, Any]) – global metadata, such as target names
-
-
class
pytext.metric_reporters.channel.
ConsoleChannel
(stages: Tuple[pytext.common.constants.Stage, ...] = (<Stage.TRAIN: 'Training'>, <Stage.EVAL: 'Evaluation'>, <Stage.TEST: 'Test'>))[source]¶ Bases:
pytext.metric_reporters.channel.Channel
Simple Channel that prints results to console.
-
report
(stage, epoch, metrics, model_select_metric, loss, preds, targets, scores, context, meta, *args)[source]¶ Defines how to format and report data to the output channel.
Parameters: - stage (Stage) – train, eval or test
- epoch (int) – current epoch
- metrics (Any) – all metrics
- model_select_metric (double) – a single numeric metric to pick best model
- loss (double) – average loss
- preds (List[Any]) – list of predictions
- targets (List[Any]) – list of targets
- scores (List[Any]) – list of scores
- context (Dict[str, List[Any]]) – dict of any additional context data, each context is a list of data that maps to each example
- meta (Dict[str, Any]) – global metadata, such as target names
-
-
class
pytext.metric_reporters.channel.
FileChannel
(stages, file_path)[source]¶ Bases:
pytext.metric_reporters.channel.Channel
Simple Channel that writes results to a TSV file.
-
report
(stage, epoch, metrics, model_select_metric, loss, preds, targets, scores, context, meta, *args)[source]¶ Defines how to format and report data to the output channel.
Parameters: - stage (Stage) – train, eval or test
- epoch (int) – current epoch
- metrics (Any) – all metrics
- model_select_metric (double) – a single numeric metric to pick best model
- loss (double) – average loss
- preds (List[Any]) – list of predictions
- targets (List[Any]) – list of targets
- scores (List[Any]) – list of scores
- context (Dict[str, List[Any]]) – dict of any additional context data, each context is a list of data that maps to each example
- meta (Dict[str, Any]) – global metadata, such as target names
-
-
class
pytext.metric_reporters.channel.
TensorBoardChannel
(summary_writer, metric_name='accuracy')[source]¶ Bases:
pytext.metric_reporters.channel.Channel
TensorBoardChannel defines how to format and report the result of a PyText job to TensorBoard.
-
summary_writer
¶ An instance of the TensorBoardX SummaryWriter class, or an object that implements the same interface. https://tensorboardx.readthedocs.io/en/latest/tensorboard.html
-
metric_name
¶ The name of the default metric to display on the TensorBoard dashboard, defaults to “accuracy”
-
add_scalars
(prefix, metrics, epoch)[source]¶ Recursively flattens the metrics object and adds each field name and value as a scalar for the corresponding epoch using the summary writer.
Parameters: - prefix (str) – The tag prefix for the metric. Each field name in the metrics object will be prepended with the prefix.
- metrics (Any) – The metrics object.
-
add_texts
(tag, metrics)[source]¶ Recursively flattens the metrics object and adds each field name and value as a text using the summary writer. For example, if tag = “test”, and metrics = { accuracy: 0.7, scores: { precision: 0.8, recall: 0.6 } }, then under “tag=test” we will display “accuracy=0.7”, and under “tag=test/scores” we will display “precision=0.8” and “recall=0.6” in TensorBoard.
Parameters: - tag (str) – The tag name for the metric. If a field needs to be flattened further, it will be prepended as a prefix to the field name.
- metrics (Any) – The metrics object.
-
report
(stage, epoch, metrics, model_select_metric, loss, preds, targets, scores, context, meta, *args)[source]¶ Defines how to format and report data to TensorBoard using the summary writer. In the current implementation, during the train/eval phase we recursively report each metric field as scalars, and during the test phase we report the final metrics to be displayed as texts.
Parameters: - stage (Stage) – train, eval or test
- epoch (int) – current epoch
- metrics (Any) – all metrics
- model_select_metric (double) – a single numeric metric to pick best model
- loss (double) – average loss
- preds (List[Any]) – list of predictions
- targets (List[Any]) – list of targets
- scores (List[Any]) – list of scores
- context (Dict[str, List[Any]]) – dict of any additional context data, each context is a list of data that maps to each example
- meta (Dict[str, Any]) – global metadata, such as target names
-
pytext.metric_reporters.classification_metric_reporter module¶
-
class
pytext.metric_reporters.classification_metric_reporter.
ClassificationMetricReporter
(label_names: List[str], channels: List[pytext.metric_reporters.channel.Channel])[source]¶ Bases:
pytext.metric_reporters.metric_reporter.MetricReporter
-
Config
¶ alias of
pytext.config.component.ComponentMeta.__new__.<locals>.Config
-
pytext.metric_reporters.compositional_metric_reporter module¶
-
class
pytext.metric_reporters.compositional_metric_reporter.
CompositionalFileChannel
(stages, file_path)[source]¶
-
class
pytext.metric_reporters.compositional_metric_reporter.
CompositionalMetricReporter
(actions_vocab, channels: List[pytext.metric_reporters.channel.Channel])[source]¶ Bases:
pytext.metric_reporters.metric_reporter.MetricReporter
-
Config
¶ alias of
pytext.config.component.ComponentMeta.__new__.<locals>.Config
-
static
get_model_select_metric
(metrics)[source]¶ Return a single numeric metric value that is used for model selection, returns the metric itself by default, but usually metrics will be more complicated data structures
-
static
node_to_metrics_node
(node: Union[pytext.data.data_structures.annotation.Intent, pytext.data.data_structures.annotation.Slot], start: int = 0) → pytext.metrics.intent_slot_metrics.Node[source]¶ The input start is the absolute start position in utterance
-
static
tree_from_tokens_and_indx_actions
(token_str_list: List[str], actions_vocab: List[str], actions_indices: List[int])[source]¶
-
static
tree_to_metric_node
(tree: pytext.data.data_structures.annotation.Tree) → pytext.metrics.intent_slot_metrics.Node[source]¶ Creates a Node from tree assuming the utterance is a concatenation of the tokens by whitespaces. The function does not necessarily reproduce the original utterance as extra whitespaces can be introduced.
-
pytext.metric_reporters.disjoint_multitask_metric_reporter module¶
-
class
pytext.metric_reporters.disjoint_multitask_metric_reporter.
DisjointMultitaskMetricReporter
(reporters: Dict[str, pytext.metric_reporters.metric_reporter.MetricReporter], target_task_name: Optional[str])[source]¶ Bases:
pytext.metric_reporters.metric_reporter.MetricReporter
-
add_batch_stats
(n_batches, preds, targets, scores, loss, m_input, **context)[source]¶ Aggregates a batch of output data (predictions, scores, targets/true labels and loss).
Parameters: - n_batches (int) – number of current batch
- preds (torch.Tensor) – predictions of current batch
- targets (torch.Tensor) – targets of current batch
- scores (torch.Tensor) – scores of current batch
- loss (double) – average loss of current batch
- m_input (Tuple[torch.Tensor, ..]) – model inputs of current batch
- context (Dict[str, Any]) – any additional context data, it could be either a list of data which maps to each example, or a single value for the batch
-
compare_metric
(new_metric, old_metric)[source]¶ Check if new metric indicates better model performance
Returns: bool, true if model with new_metric performs better
-
report_metric
(stage, epoch, reset=True, print_to_channels=True)[source]¶ Calculate metrics and average loss, report all statistic data to channels
Parameters: - stage (Stage) – training, evaluation or test
- epoch (int) – current epoch
- reset (bool) – if all data should be reset after report, default is True
- print_to_channels (bool) – if report data to channels, default is True
-
pytext.metric_reporters.intent_slot_detection_metric_reporter module¶
-
class
pytext.metric_reporters.intent_slot_detection_metric_reporter.
IntentSlotChannel
(stages, file_path)[source]¶
-
class
pytext.metric_reporters.intent_slot_detection_metric_reporter.
IntentSlotMetricReporter
(doc_label_names: List[str], word_label_names: List[str], use_bio_labels: bool, channels: List[pytext.metric_reporters.channel.Channel])[source]¶ Bases:
pytext.metric_reporters.metric_reporter.MetricReporter
-
Config
¶ alias of
pytext.config.component.ComponentMeta.__new__.<locals>.Config
-
pytext.metric_reporters.language_model_metric_reporter module¶
-
class
pytext.metric_reporters.language_model_metric_reporter.
LanguageModelChannel
(stages, file_path)[source]¶
-
class
pytext.metric_reporters.language_model_metric_reporter.
LanguageModelMetricReporter
(channels)[source]¶ Bases:
pytext.metric_reporters.metric_reporter.MetricReporter
-
Config
¶ alias of
pytext.config.component.ComponentMeta.__new__.<locals>.Config
-
calculate_metric
() → pytext.metrics.language_model_metrics.LanguageModelMetric[source]¶ Calculate metrics, each sub class should implement it
-
static
get_model_select_metric
(metrics) → float[source]¶ Return a single numeric metric value that is used for model selection, returns the metric itself by default, but usually metrics will be more complicated data structures
-
lower_is_better
= True¶
-
pytext.metric_reporters.metric_reporter module¶
-
class
pytext.metric_reporters.metric_reporter.
MetricReporter
(channels)[source]¶ Bases:
pytext.config.component.Component
MetricReporter is responsible of three things:
- Aggregate output from trainer, which includes model inputs, predictions, targets, scores, and loss.
- Calculate metrics using the aggregated output, and define how the metric is used to find best model
- Optionally report the metrics and aggregated output to various channels
-
lower_is_better
¶ bool – Whether a lower metric indicates better performance. Set to True for e.g. perplexity, and False for e.g. accuracy. Default is False
-
channels
¶ List[Channel] – A list of Channel that will receive metrics and the aggregated trainer output then format and report them in any customized way.
-
Config
[source]¶ alias of
MetricReporter.Config
-
add_batch_stats
(n_batches, preds, targets, scores, loss, m_input, **context)[source]¶ Aggregates a batch of output data (predictions, scores, targets/true labels and loss).
Parameters: - n_batches (int) – number of current batch
- preds (torch.Tensor) – predictions of current batch
- targets (torch.Tensor) – targets of current batch
- scores (torch.Tensor) – scores of current batch
- loss (double) – average loss of current batch
- m_input (Tuple[torch.Tensor, ..]) – model inputs of current batch
- context (Dict[str, Any]) – any additional context data, it could be either a list of data which maps to each example, or a single value for the batch
-
classmethod
aggregate_data
(all_data, new_batch)[source]¶ Aggregate a batch of data, basically just convert tensors to list of native python data
-
classmethod
compare_metric
(new_metric, old_metric)[source]¶ Check if new metric indicates better model performance
Returns: bool, true if model with new_metric performs better
-
get_meta
()[source]¶ Get global meta data that is not specific to any batch, the data will be pass along to channels
-
static
get_model_select_metric
(metrics)[source]¶ Return a single numeric metric value that is used for model selection, returns the metric itself by default, but usually metrics will be more complicated data structures
-
lower_is_better
= False
-
report_metric
(stage, epoch, reset=True, print_to_channels=True)[source]¶ Calculate metrics and average loss, report all statistic data to channels
Parameters: - stage (Stage) – training, evaluation or test
- epoch (int) – current epoch
- reset (bool) – if all data should be reset after report, default is True
- print_to_channels (bool) – if report data to channels, default is True
pytext.metric_reporters.word_tagging_metric_reporter module¶
-
class
pytext.metric_reporters.word_tagging_metric_reporter.
WordTaggingMetricReporter
(label_names: List[str], use_bio_labels: bool, channels: List[pytext.metric_reporters.channel.Channel])[source]¶ Bases:
pytext.metric_reporters.metric_reporter.MetricReporter
-
Config
¶ alias of
pytext.config.component.ComponentMeta.__new__.<locals>.Config
-
Module contents¶
-
class
pytext.metric_reporters.
Channel
(stages: Tuple[pytext.common.constants.Stage, ...] = (<Stage.TRAIN: 'Training'>, <Stage.EVAL: 'Evaluation'>, <Stage.TEST: 'Test'>))[source]¶ Bases:
object
Channel defines how to format and report the result of a PyText job to an output stream.
-
stages
¶ in which stages the report will be triggered, default is all stages, which includes train, eval, test
-
report
(stage, epoch, metrics, model_select_metric, loss, preds, targets, scores, context, meta, *args)[source]¶ Defines how to format and report data to the output channel.
Parameters: - stage (Stage) – train, eval or test
- epoch (int) – current epoch
- metrics (Any) – all metrics
- model_select_metric (double) – a single numeric metric to pick best model
- loss (double) – average loss
- preds (List[Any]) – list of predictions
- targets (List[Any]) – list of targets
- scores (List[Any]) – list of scores
- context (Dict[str, List[Any]]) – dict of any additional context data, each context is a list of data that maps to each example
- meta (Dict[str, Any]) – global metadata, such as target names
-
-
class
pytext.metric_reporters.
MetricReporter
(channels)[source]¶ Bases:
pytext.config.component.Component
MetricReporter is responsible of three things:
- Aggregate output from trainer, which includes model inputs, predictions, targets, scores, and loss.
- Calculate metrics using the aggregated output, and define how the metric is used to find best model
- Optionally report the metrics and aggregated output to various channels
-
lower_is_better
¶ bool – Whether a lower metric indicates better performance. Set to True for e.g. perplexity, and False for e.g. accuracy. Default is False
-
channels
¶ List[Channel] – A list of Channel that will receive metrics and the aggregated trainer output then format and report them in any customized way.
-
Config
[source]¶ alias of
MetricReporter.Config
-
add_batch_stats
(n_batches, preds, targets, scores, loss, m_input, **context)[source]¶ Aggregates a batch of output data (predictions, scores, targets/true labels and loss).
Parameters: - n_batches (int) – number of current batch
- preds (torch.Tensor) – predictions of current batch
- targets (torch.Tensor) – targets of current batch
- scores (torch.Tensor) – scores of current batch
- loss (double) – average loss of current batch
- m_input (Tuple[torch.Tensor, ..]) – model inputs of current batch
- context (Dict[str, Any]) – any additional context data, it could be either a list of data which maps to each example, or a single value for the batch
-
classmethod
aggregate_data
(all_data, new_batch)[source]¶ Aggregate a batch of data, basically just convert tensors to list of native python data
-
classmethod
compare_metric
(new_metric, old_metric)[source]¶ Check if new metric indicates better model performance
Returns: bool, true if model with new_metric performs better
-
get_meta
()[source]¶ Get global meta data that is not specific to any batch, the data will be pass along to channels
-
static
get_model_select_metric
(metrics)[source]¶ Return a single numeric metric value that is used for model selection, returns the metric itself by default, but usually metrics will be more complicated data structures
-
lower_is_better
= False
-
report_metric
(stage, epoch, reset=True, print_to_channels=True)[source]¶ Calculate metrics and average loss, report all statistic data to channels
Parameters: - stage (Stage) – training, evaluation or test
- epoch (int) – current epoch
- reset (bool) – if all data should be reset after report, default is True
- print_to_channels (bool) – if report data to channels, default is True
-
class
pytext.metric_reporters.
ClassificationMetricReporter
(label_names: List[str], channels: List[pytext.metric_reporters.channel.Channel])[source]¶ Bases:
pytext.metric_reporters.metric_reporter.MetricReporter
-
Config
¶ alias of
pytext.config.component.ComponentMeta.__new__.<locals>.Config
-
-
class
pytext.metric_reporters.
IntentSlotMetricReporter
(doc_label_names: List[str], word_label_names: List[str], use_bio_labels: bool, channels: List[pytext.metric_reporters.channel.Channel])[source]¶ Bases:
pytext.metric_reporters.metric_reporter.MetricReporter
-
Config
¶ alias of
pytext.config.component.ComponentMeta.__new__.<locals>.Config
-
-
class
pytext.metric_reporters.
LanguageModelMetricReporter
(channels)[source]¶ Bases:
pytext.metric_reporters.metric_reporter.MetricReporter
-
Config
¶ alias of
pytext.config.component.ComponentMeta.__new__.<locals>.Config
-
calculate_metric
() → pytext.metrics.language_model_metrics.LanguageModelMetric[source]¶ Calculate metrics, each sub class should implement it
-
static
get_model_select_metric
(metrics) → float[source]¶ Return a single numeric metric value that is used for model selection, returns the metric itself by default, but usually metrics will be more complicated data structures
-
lower_is_better
= True¶
-
-
class
pytext.metric_reporters.
WordTaggingMetricReporter
(label_names: List[str], use_bio_labels: bool, channels: List[pytext.metric_reporters.channel.Channel])[source]¶ Bases:
pytext.metric_reporters.metric_reporter.MetricReporter
-
Config
¶ alias of
pytext.config.component.ComponentMeta.__new__.<locals>.Config
-
-
class
pytext.metric_reporters.
CompositionalMetricReporter
(actions_vocab, channels: List[pytext.metric_reporters.channel.Channel])[source]¶ Bases:
pytext.metric_reporters.metric_reporter.MetricReporter
-
Config
¶ alias of
pytext.config.component.ComponentMeta.__new__.<locals>.Config
-
static
get_model_select_metric
(metrics)[source]¶ Return a single numeric metric value that is used for model selection, returns the metric itself by default, but usually metrics will be more complicated data structures
-
static
node_to_metrics_node
(node: Union[pytext.data.data_structures.annotation.Intent, pytext.data.data_structures.annotation.Slot], start: int = 0) → pytext.metrics.intent_slot_metrics.Node[source]¶ The input start is the absolute start position in utterance
-
static
tree_from_tokens_and_indx_actions
(token_str_list: List[str], actions_vocab: List[str], actions_indices: List[int])[source]¶
-
static
tree_to_metric_node
(tree: pytext.data.data_structures.annotation.Tree) → pytext.metrics.intent_slot_metrics.Node[source]¶ Creates a Node from tree assuming the utterance is a concatenation of the tokens by whitespaces. The function does not necessarily reproduce the original utterance as extra whitespaces can be introduced.
-