LanguageModelDataHandler.Config

Component: LanguageModelDataHandler

class LanguageModelDataHandler.Config[source]

Bases: DataHandler.Config

Configuration class for LanguageModelDataHandler.

columns_to_read

List[str] – List containing the names of the columns to read from the data files.

append_bos

bool – If True appends beginning of sentence marker to sentences. Defaults to True.

append_eos

bool – If True appends end of sentence marker to sentences. Defaults to True.

All Attributes (including base classes)

columns_to_read: List[str] = ['text']
shuffle: bool = True
sort_within_batch: bool = True
train_path: str = 'train.tsv'
eval_path: str = 'eval.tsv'
test_path: str = 'test.tsv'
train_batch_size: int = 128
eval_batch_size: int = 128
test_batch_size: int = 128
append_bos: bool = True
append_eos: bool = True

Default JSON

{
    "columns_to_read": [
        "text"
    ],
    "shuffle": true,
    "sort_within_batch": true,
    "train_path": "train.tsv",
    "eval_path": "eval.tsv",
    "test_path": "test.tsv",
    "train_batch_size": 128,
    "eval_batch_size": 128,
    "test_batch_size": 128,
    "append_bos": true,
    "append_eos": true
}