pytext.metric_reporters package

Submodules

pytext.metric_reporters.channel module

class pytext.metric_reporters.channel.Channel(stages: Tuple[pytext.common.constants.Stage, ...] = (<Stage.TRAIN: 'Training'>, <Stage.EVAL: 'Evaluation'>, <Stage.TEST: 'Test'>))

Bases: object

Channel defines how to format and report the result of a PyText job to an output stream.

stages

in which stages the report will be triggered, default is all stages, which includes train, eval, test

report(stage, epoch, metrics, model_select_metric, loss, preds, targets, scores, context, meta, *args)

Defines how to format and report data to the output channel.

Parameters:

• stage (Stage) – train, eval or test
• epoch (int) – current epoch
• metrics (Any) – all metrics
• model_select_metric (double) – a single numeric metric to pick best model
• loss (double) – average loss
• preds (List[Any]) – list of predictions
• targets (List[Any]) – list of targets
• scores (List[Any]) – list of scores
• context (Dict[str, List[Any]]) – dict of any additional context data, each context is a list of data that maps to each example
• meta (Dict[str, Any]) – global metadata, such as target names

class pytext.metric_reporters.channel.ConsoleChannel(stages: Tuple[pytext.common.constants.Stage, ...] = (<Stage.TRAIN: 'Training'>, <Stage.EVAL: 'Evaluation'>, <Stage.TEST: 'Test'>))

Bases: pytext.metric_reporters.channel.Channel

Simple Channel that prints results to console.

report(stage, epoch, metrics, model_select_metric, loss, preds, targets, scores, context, meta, *args)

Defines how to format and report data to the output channel.

Parameters:

• stage (Stage) – train, eval or test
• epoch (int) – current epoch
• metrics (Any) – all metrics
• model_select_metric (double) – a single numeric metric to pick best model
• loss (double) – average loss
• preds (List[Any]) – list of predictions
• targets (List[Any]) – list of targets
• scores (List[Any]) – list of scores
• context (Dict[str, List[Any]]) – dict of any additional context data, each context is a list of data that maps to each example
• meta (Dict[str, Any]) – global metadata, such as target names

class pytext.metric_reporters.channel.FileChannel(stages, file_path)

Bases: pytext.metric_reporters.channel.Channel

Simple Channel that writes results to a TSV file.

gen_content(metrics, loss, preds, targets, scores, contexts)

get_title()

report(stage, epoch, metrics, model_select_metric, loss, preds, targets, scores, context, meta, *args)

Defines how to format and report data to the output channel.

Parameters:

• stage (Stage) – train, eval or test
• epoch (int) – current epoch
• metrics (Any) – all metrics
• model_select_metric (double) – a single numeric metric to pick best model
• loss (double) – average loss
• preds (List[Any]) – list of predictions
• targets (List[Any]) – list of targets
• scores (List[Any]) – list of scores
• context (Dict[str, List[Any]]) – dict of any additional context data, each context is a list of data that maps to each example
• meta (Dict[str, Any]) – global metadata, such as target names

class pytext.metric_reporters.channel.TensorBoardChannel(summary_writer, metric_name='accuracy')

Bases: pytext.metric_reporters.channel.Channel

TensorBoardChannel defines how to format and report the result of a PyText job to TensorBoard.

summary_writer

An instance of the TensorBoardX SummaryWriter class, or an object that implements the same interface. https://tensorboardx.readthedocs.io/en/latest/tensorboard.html

metric_name

The name of the default metric to display on the TensorBoard dashboard, defaults to “accuracy”

add_scalars(prefix, metrics, epoch)

Recursively flattens the metrics object and adds each field name and value as a scalar for the corresponding epoch using the summary writer.

Parameters:

• prefix (str) – The tag prefix for the metric. Each field name in the metrics object will be prepended with the prefix.
• metrics (Any) – The metrics object.

add_texts(tag, metrics)

Recursively flattens the metrics object and adds each field name and value as a text using the summary writer. For example, if tag = “test”, and metrics = { accuracy: 0.7, scores: { precision: 0.8, recall: 0.6 } }, then under “tag=test” we will display “accuracy=0.7”, and under “tag=test/scores” we will display “precision=0.8” and “recall=0.6” in TensorBoard.

Parameters:

• tag (str) – The tag name for the metric. If a field needs to be flattened further, it will be prepended as a prefix to the field name.
• metrics (Any) – The metrics object.

report(stage, epoch, metrics, model_select_metric, loss, preds, targets, scores, context, meta, *args)

Defines how to format and report data to TensorBoard using the summary writer. In the current implementation, during the train/eval phase we recursively report each metric field as scalars, and during the test phase we report the final metrics to be displayed as texts.

Parameters:

• stage (Stage) – train, eval or test
• epoch (int) – current epoch
• metrics (Any) – all metrics
• model_select_metric (double) – a single numeric metric to pick best model
• loss (double) – average loss
• preds (List[Any]) – list of predictions
• targets (List[Any]) – list of targets
• scores (List[Any]) – list of scores
• context (Dict[str, List[Any]]) – dict of any additional context data, each context is a list of data that maps to each example
• meta (Dict[str, Any]) – global metadata, such as target names

pytext.metric_reporters.classification_metric_reporter module

class pytext.metric_reporters.classification_metric_reporter.ClassificationMetricReporter(label_names: List[str], channels: List[pytext.metric_reporters.channel.Channel])

Bases: pytext.metric_reporters.metric_reporter.MetricReporter

Config

alias of pytext.config.component.ComponentMeta.__new__.<locals>.Config

calculate_metric()

Calculate metrics, each sub class should implement it

classmethod from_config(config, meta: pytext.data.data_handler.CommonMetadata)

get_meta()

Get global meta data that is not specific to any batch, the data will be pass along to channels

static get_model_select_metric(metrics)

Return a single numeric metric value that is used for model selection, returns the metric itself by default, but usually metrics will be more complicated data structures

class pytext.metric_reporters.classification_metric_reporter.IntentModelChannel(stages, file_path)

Bases: pytext.metric_reporters.channel.FileChannel

gen_content(metrics, loss, preds, targets, scores, contexts)

get_title()

pytext.metric_reporters.compositional_metric_reporter module

class pytext.metric_reporters.compositional_metric_reporter.CompositionalFileChannel(stages, file_path)

Bases: pytext.metric_reporters.channel.FileChannel

gen_content(metrics, loss, preds, targets, scores, context)

get_title()

class pytext.metric_reporters.compositional_metric_reporter.CompositionalMetricReporter(actions_vocab, channels: List[pytext.metric_reporters.channel.Channel])

Bases: pytext.metric_reporters.metric_reporter.MetricReporter

Config

alias of pytext.config.component.ComponentMeta.__new__.<locals>.Config

calculate_metric()

Calculate metrics, each sub class should implement it

classmethod from_config(config, metadata: pytext.data.data_handler.CommonMetadata)

gen_extra_context()

Generate any extra intermediate context data for metric calculation

static get_model_select_metric(metrics)

Return a single numeric metric value that is used for model selection, returns the metric itself by default, but usually metrics will be more complicated data structures

static node_to_metrics_node(node: Union[pytext.data.data_structures.annotation.Intent, pytext.data.data_structures.annotation.Slot], start: int = 0) → pytext.metrics.intent_slot_metrics.Node

The input start is the absolute start position in utterance

static tree_from_tokens_and_indx_actions(token_str_list: List[str], actions_vocab: List[str], actions_indices: List[int])

static tree_to_metric_node(tree: pytext.data.data_structures.annotation.Tree) → pytext.metrics.intent_slot_metrics.Node

Creates a Node from tree assuming the utterance is a concatenation of the tokens by whitespaces. The function does not necessarily reproduce the original utterance as extra whitespaces can be introduced.

pytext.metric_reporters.disjoint_multitask_metric_reporter module

class pytext.metric_reporters.disjoint_multitask_metric_reporter.DisjointMultitaskMetricReporter(reporters: Dict[str, pytext.metric_reporters.metric_reporter.MetricReporter], target_task_name: Optional[str])

Bases: pytext.metric_reporters.metric_reporter.MetricReporter

Config

alias of DisjointMultitaskMetricReporter.Config

add_batch_stats(n_batches, preds, targets, scores, loss, m_input, **context)

Aggregates a batch of output data (predictions, scores, targets/true labels and loss).

Parameters:

• n_batches (int) – number of current batch
• preds (torch.Tensor) – predictions of current batch
• targets (torch.Tensor) – targets of current batch
• scores (torch.Tensor) – scores of current batch
• loss (double) – average loss of current batch
• m_input (Tuple[torch.Tensor, ..]) – model inputs of current batch
• context (Dict[str, Any]) – any additional context data, it could be either a list of data which maps to each example, or a single value for the batch

add_channel(channel)

compare_metric(new_metric, old_metric)

Check if new metric indicates better model performance

Returns:bool, true if model with new_metric performs better

report_metric(stage, epoch, reset=True, print_to_channels=True)

Calculate metrics and average loss, report all statistic data to channels

Parameters:

• stage (Stage) – training, evaluation or test
• epoch (int) – current epoch
• reset (bool) – if all data should be reset after report, default is True
• print_to_channels (bool) – if report data to channels, default is True

pytext.metric_reporters.intent_slot_detection_metric_reporter module

class pytext.metric_reporters.intent_slot_detection_metric_reporter.IntentSlotChannel(stages, file_path)

Bases: pytext.metric_reporters.channel.FileChannel

static create_annotation(utterance: str, intent_label: str, slots_label: str) → str

gen_content(metrics, loss, preds, targets, scores, context)

get_title()

class pytext.metric_reporters.intent_slot_detection_metric_reporter.IntentSlotMetricReporter(doc_label_names: List[str], word_label_names: List[str], use_bio_labels: bool, channels: List[pytext.metric_reporters.channel.Channel])

Bases: pytext.metric_reporters.metric_reporter.MetricReporter

Config

alias of pytext.config.component.ComponentMeta.__new__.<locals>.Config

aggregate_preds(new_batch)

aggregate_scores(new_batch)

aggregate_targets(new_batch)

calculate_metric()

Calculate metrics, each sub class should implement it

classmethod from_config(config, meta: pytext.data.data_handler.CommonMetadata)

gen_extra_context()

Generate any extra intermediate context data for metric calculation

static get_model_select_metric(metrics)

Return a single numeric metric value that is used for model selection, returns the metric itself by default, but usually metrics will be more complicated data structures

process_pred(pred: List[int]) → List[str]

pred is a list of token label index

pytext.metric_reporters.intent_slot_detection_metric_reporter.create_frame(intent_label, slot_names_str, utterance)

pytext.metric_reporters.language_model_metric_reporter module

class pytext.metric_reporters.language_model_metric_reporter.LanguageModelChannel(stages, file_path)

Bases: pytext.metric_reporters.channel.FileChannel

gen_content(metrics, loss, preds, targets, scores, contexts)

get_title()

class pytext.metric_reporters.language_model_metric_reporter.LanguageModelMetricReporter(channels)

Bases: pytext.metric_reporters.metric_reporter.MetricReporter

Config

alias of pytext.config.component.ComponentMeta.__new__.<locals>.Config

calculate_loss() → float

Calculate the average loss for all aggregated batch

calculate_metric() → pytext.metrics.language_model_metrics.LanguageModelMetric

Calculate metrics, each sub class should implement it

classmethod from_config(config, meta: pytext.data.data_handler.CommonMetadata)

static get_model_select_metric(metrics) → float

Return a single numeric metric value that is used for model selection, returns the metric itself by default, but usually metrics will be more complicated data structures

lower_is_better = True

pytext.metric_reporters.metric_reporter module

class pytext.metric_reporters.metric_reporter.MetricReporter(channels)

Bases: pytext.config.component.Component

MetricReporter is responsible of three things:

1. Aggregate output from trainer, which includes model inputs, predictions, targets, scores, and loss.
2. Calculate metrics using the aggregated output, and define how the metric is used to find best model
3. Optionally report the metrics and aggregated output to various channels

lower_is_better

bool – Whether a lower metric indicates better performance. Set to True for e.g. perplexity, and False for e.g. accuracy. Default is False

channels

List[Channel] – A list of Channel that will receive metrics and the aggregated trainer output then format and report them in any customized way.

Config

alias of MetricReporter.Config

add_batch_stats(n_batches, preds, targets, scores, loss, m_input, **context)

Aggregates a batch of output data (predictions, scores, targets/true labels and loss).

Parameters:

• n_batches (int) – number of current batch
• preds (torch.Tensor) – predictions of current batch
• targets (torch.Tensor) – targets of current batch
• scores (torch.Tensor) – scores of current batch
• loss (double) – average loss of current batch
• m_input (Tuple[torch.Tensor, ..]) – model inputs of current batch
• context (Dict[str, Any]) – any additional context data, it could be either a list of data which maps to each example, or a single value for the batch

add_channel(channel)

classmethod aggregate_data(all_data, new_batch)

Aggregate a batch of data, basically just convert tensors to list of native python data

aggregate_preds(new_batch)

aggregate_scores(new_batch)

aggregate_targets(new_batch)

calculate_loss()

Calculate the average loss for all aggregated batch

calculate_metric()

Calculate metrics, each sub class should implement it

classmethod compare_metric(new_metric, old_metric)

Check if new metric indicates better model performance

Returns:bool, true if model with new_metric performs better

gen_extra_context()

Generate any extra intermediate context data for metric calculation

get_meta()

Get global meta data that is not specific to any batch, the data will be pass along to channels

static get_model_select_metric(metrics)

Return a single numeric metric value that is used for model selection, returns the metric itself by default, but usually metrics will be more complicated data structures

lower_is_better = False

report_metric(stage, epoch, reset=True, print_to_channels=True)

Calculate metrics and average loss, report all statistic data to channels

Parameters:

• stage (Stage) – training, evaluation or test
• epoch (int) – current epoch
• reset (bool) – if all data should be reset after report, default is True
• print_to_channels (bool) – if report data to channels, default is True

pytext.metric_reporters.word_tagging_metric_reporter module

class pytext.metric_reporters.word_tagging_metric_reporter.WordTaggingMetricReporter(label_names: List[str], use_bio_labels: bool, channels: List[pytext.metric_reporters.channel.Channel])

Bases: pytext.metric_reporters.metric_reporter.MetricReporter

Config

alias of pytext.config.component.ComponentMeta.__new__.<locals>.Config

calculate_loss()

Calculate the average loss for all aggregated batch

calculate_metric()

Calculate metrics, each sub class should implement it

classmethod from_config(config, meta: pytext.data.data_handler.CommonMetadata)

static get_model_select_metric(metrics)

Return a single numeric metric value that is used for model selection, returns the metric itself by default, but usually metrics will be more complicated data structures

process_pred(pred: List[int]) → List[str]

pred is a list of token label index

pytext.metric_reporters.word_tagging_metric_reporter.get_slots(word_names)

Module contents

class pytext.metric_reporters.Channel(stages: Tuple[pytext.common.constants.Stage, ...] = (<Stage.TRAIN: 'Training'>, <Stage.EVAL: 'Evaluation'>, <Stage.TEST: 'Test'>))

Bases: object

Channel defines how to format and report the result of a PyText job to an output stream.

stages

in which stages the report will be triggered, default is all stages, which includes train, eval, test

report(stage, epoch, metrics, model_select_metric, loss, preds, targets, scores, context, meta, *args)

Defines how to format and report data to the output channel.

Parameters:

• stage (Stage) – train, eval or test
• epoch (int) – current epoch
• metrics (Any) – all metrics
• model_select_metric (double) – a single numeric metric to pick best model
• loss (double) – average loss
• preds (List[Any]) – list of predictions
• targets (List[Any]) – list of targets
• scores (List[Any]) – list of scores
• context (Dict[str, List[Any]]) – dict of any additional context data, each context is a list of data that maps to each example
• meta (Dict[str, Any]) – global metadata, such as target names

class pytext.metric_reporters.MetricReporter(channels)

Bases: pytext.config.component.Component

MetricReporter is responsible of three things:

1. Aggregate output from trainer, which includes model inputs, predictions, targets, scores, and loss.
2. Calculate metrics using the aggregated output, and define how the metric is used to find best model
3. Optionally report the metrics and aggregated output to various channels

lower_is_better

bool – Whether a lower metric indicates better performance. Set to True for e.g. perplexity, and False for e.g. accuracy. Default is False

channels

List[Channel] – A list of Channel that will receive metrics and the aggregated trainer output then format and report them in any customized way.

Config

alias of MetricReporter.Config

add_batch_stats(n_batches, preds, targets, scores, loss, m_input, **context)

Aggregates a batch of output data (predictions, scores, targets/true labels and loss).

Parameters:

• n_batches (int) – number of current batch
• preds (torch.Tensor) – predictions of current batch
• targets (torch.Tensor) – targets of current batch
• scores (torch.Tensor) – scores of current batch
• loss (double) – average loss of current batch
• m_input (Tuple[torch.Tensor, ..]) – model inputs of current batch
• context (Dict[str, Any]) – any additional context data, it could be either a list of data which maps to each example, or a single value for the batch

add_channel(channel)

classmethod aggregate_data(all_data, new_batch)

Aggregate a batch of data, basically just convert tensors to list of native python data

aggregate_preds(new_batch)

aggregate_scores(new_batch)

aggregate_targets(new_batch)

calculate_loss()

Calculate the average loss for all aggregated batch

calculate_metric()

Calculate metrics, each sub class should implement it

classmethod compare_metric(new_metric, old_metric)

Check if new metric indicates better model performance

Returns:bool, true if model with new_metric performs better

gen_extra_context()

Generate any extra intermediate context data for metric calculation

get_meta()

Get global meta data that is not specific to any batch, the data will be pass along to channels

static get_model_select_metric(metrics)

Return a single numeric metric value that is used for model selection, returns the metric itself by default, but usually metrics will be more complicated data structures

lower_is_better = False

report_metric(stage, epoch, reset=True, print_to_channels=True)

Calculate metrics and average loss, report all statistic data to channels

Parameters:

• stage (Stage) – training, evaluation or test
• epoch (int) – current epoch
• reset (bool) – if all data should be reset after report, default is True
• print_to_channels (bool) – if report data to channels, default is True

class pytext.metric_reporters.ClassificationMetricReporter(label_names: List[str], channels: List[pytext.metric_reporters.channel.Channel])

Bases: pytext.metric_reporters.metric_reporter.MetricReporter

Config

alias of pytext.config.component.ComponentMeta.__new__.<locals>.Config

calculate_metric()

Calculate metrics, each sub class should implement it

classmethod from_config(config, meta: pytext.data.data_handler.CommonMetadata)

get_meta()

Get global meta data that is not specific to any batch, the data will be pass along to channels

static get_model_select_metric(metrics)

Return a single numeric metric value that is used for model selection, returns the metric itself by default, but usually metrics will be more complicated data structures

class pytext.metric_reporters.IntentSlotMetricReporter(doc_label_names: List[str], word_label_names: List[str], use_bio_labels: bool, channels: List[pytext.metric_reporters.channel.Channel])

Bases: pytext.metric_reporters.metric_reporter.MetricReporter

Config

alias of pytext.config.component.ComponentMeta.__new__.<locals>.Config

aggregate_preds(new_batch)

aggregate_scores(new_batch)

aggregate_targets(new_batch)

calculate_metric()

Calculate metrics, each sub class should implement it

classmethod from_config(config, meta: pytext.data.data_handler.CommonMetadata)

gen_extra_context()

Generate any extra intermediate context data for metric calculation

static get_model_select_metric(metrics)

Return a single numeric metric value that is used for model selection, returns the metric itself by default, but usually metrics will be more complicated data structures

process_pred(pred: List[int]) → List[str]

pred is a list of token label index

class pytext.metric_reporters.LanguageModelMetricReporter(channels)

Bases: pytext.metric_reporters.metric_reporter.MetricReporter

Config

alias of pytext.config.component.ComponentMeta.__new__.<locals>.Config

calculate_loss() → float

Calculate the average loss for all aggregated batch

calculate_metric() → pytext.metrics.language_model_metrics.LanguageModelMetric

Calculate metrics, each sub class should implement it

classmethod from_config(config, meta: pytext.data.data_handler.CommonMetadata)

static get_model_select_metric(metrics) → float

Return a single numeric metric value that is used for model selection, returns the metric itself by default, but usually metrics will be more complicated data structures

lower_is_better = True

class pytext.metric_reporters.WordTaggingMetricReporter(label_names: List[str], use_bio_labels: bool, channels: List[pytext.metric_reporters.channel.Channel])

Bases: pytext.metric_reporters.metric_reporter.MetricReporter

Config

alias of pytext.config.component.ComponentMeta.__new__.<locals>.Config

calculate_loss()

Calculate the average loss for all aggregated batch

calculate_metric()

Calculate metrics, each sub class should implement it

classmethod from_config(config, meta: pytext.data.data_handler.CommonMetadata)

static get_model_select_metric(metrics)

Return a single numeric metric value that is used for model selection, returns the metric itself by default, but usually metrics will be more complicated data structures

process_pred(pred: List[int]) → List[str]

pred is a list of token label index

class pytext.metric_reporters.CompositionalMetricReporter(actions_vocab, channels: List[pytext.metric_reporters.channel.Channel])

Bases: pytext.metric_reporters.metric_reporter.MetricReporter

Config

alias of pytext.config.component.ComponentMeta.__new__.<locals>.Config

calculate_metric()

Calculate metrics, each sub class should implement it

classmethod from_config(config, metadata: pytext.data.data_handler.CommonMetadata)

gen_extra_context()

Generate any extra intermediate context data for metric calculation

static get_model_select_metric(metrics)

Return a single numeric metric value that is used for model selection, returns the metric itself by default, but usually metrics will be more complicated data structures

static node_to_metrics_node(node: Union[pytext.data.data_structures.annotation.Intent, pytext.data.data_structures.annotation.Slot], start: int = 0) → pytext.metrics.intent_slot_metrics.Node

The input start is the absolute start position in utterance

static tree_from_tokens_and_indx_actions(token_str_list: List[str], actions_vocab: List[str], actions_indices: List[int])

static tree_to_metric_node(tree: pytext.data.data_structures.annotation.Tree) → pytext.metrics.intent_slot_metrics.Node

Creates a Node from tree assuming the utterance is a concatenation of the tokens by whitespaces. The function does not necessarily reproduce the original utterance as extra whitespaces can be introduced.