pandora_llm.data.base.Dataset

Module Contents

class pandora_llm.data.base.Dataset.DatasetDictWithMetadata(datasets={}, metadata={})

Bases: datasets.DatasetDict

A HuggingFace DatasetDict with metadata support. Metadata is stored in the .metadata dict and is saved/pushed to Hub via ‘metadata.json’.

Parameters:

• datasets (dict[str, Union[datasets.Dataset, datasets.table.Table]])

• metadata (dict)

classmethod load_from_disk(dataset_path, **kwargs)

Load the DatasetDict and metadata from disk. :param dataset_path: Path to load the dataset.

Returns:

The loaded dataset dictionary with metadata.

Return type:

DatasetDictWithMetadata

Parameters:

dataset_path (datasets.utils.typing.PathLike)

push_to_hub(repo_id, **kwargs)

Push the DatasetDict and metadata to the Hugging Face Hub. :param repo_id: The Hugging Face Hub repository ID.

Return type:

None

flatten(*args, **kwargs)

cast(*args, **kwargs)

cast_column(*args, **kwargs)

remove_columns(*args, **kwargs)

rename_column(*args, **kwargs)

rename_columns(*args, **kwargs)

select_columns(*args, **kwargs)

class_encode_column(*args, **kwargs)

map(*args, **kwargs)

filter(*args, **kwargs)

flatten_indices(*args, **kwargs)

sort(*args, **kwargs)

shuffle(*args, **kwargs)

align_labels_with_mapping(*args, **kwargs)

static from_csv(*args, **kwargs)

static from_json(*args, **kwargs)

static from_parquet(*args, **kwargs)

static from_text(*args, **kwargs)

class pandora_llm.data.base.Dataset.DatasetWithMetadata(data, **kwargs)

Bases: datasets.Dataset

A HuggingFace dataset with metadata support. Metadata is stored in the .metadata dict and is saved/pushed to Hub via ‘metadata.json’.

Parameters:

data (Union[datasets.Dataset])

metadata

save_to_disk(dataset_path, **kwargs)

Save the dataset and metadata to disk. :param dataset_path: Path to save the dataset.

classmethod load_from_disk(dataset_path, **kwargs)

Load the dataset and metadata from disk. :param dataset_path: Path to load the dataset.

Returns:

The loaded dataset with metadata.

Return type:

DatasetWithMetadata

push_to_hub(repo_id, **kwargs)

Push the dataset and metadata to Hugging Face Hub. :param repo_id: The Hugging Face Hub repository ID.

_select_contiguous(*args, **kwargs)

_new_dataset_with_indices(*args, **kwargs)

map(*args, **kwargs)

train_test_split(*args, **kwargs)

add_column(*args, **kwargs)

add_item(*args, **kwargs)

static from_csv(*args, **kwargs)

static from_generator(*args, **kwargs)

static from_json(*args, **kwargs)

static from_parquet(*args, **kwargs)

static from_text(*args, **kwargs)

static from_spark(*args, **kwargs)

static from_sql(*args, **kwargs)

pandora_llm.data.base.Dataset.load_dataset_with_metadata(path, **kwargs)

Load a dataset from the Hugging Face Hub or local path and load metadata.json if it exists.

Parameters:

• path (str) – The dataset repository ID or local path.

• **kwargs – Additional arguments for datasets.load_dataset.

Returns:

A dataset with metadata loaded from metadata.json, if available.

Return type:

DatasetWithMetadata

pandora_llm.data.base.Dataset.concatenate_datasets_with_metadata(*args, **kwargs)

Return type:

DatasetWithMetadata