pandora_llm.features.BoW¶

Module Contents¶

class pandora_llm.features.BoW.BoW[source]¶

Bases: pandora_llm.features.base.FeatureComputer

Bag of words features

compute_features(dataloader, mode)[source]¶

Computes bow features, with mode specifying the type of data the dataloader is holding.

Parameters:

dataloader (torch.utils.data.DataLoader) – dataloader to compute bow features over
mode (str) – whether to compute bow on ‘tokens’ or ‘text’

Returns:

BoW features for each sample

Raises:

ValueError – if mode is not one of ‘tokens’ or ‘text’

Return type:

jaxtyping.Num[torch.Tensor, n vocab]

train_bow_text(dataloader)[source]¶

Trains BoW’s vocabulary on the given dataloader

Parameters:: dataloader (torch.utils.data.DataLoader) – text dataloader to compute bow features over
Return type:: None

compute_bow_text(dataloader)[source]¶

Computes the bow features on the dataloader

Parameters:: dataloader (torch.utils.data.DataLoader) – input dataloader to compute bow features over
Returns:: Bow features for each sample (N x vocab)
Raises:: Exception – if did not call train beforehand
Return type:: jaxtyping.Integer[torch.Tensor, n vocab]

train_bow_tokens(dataloader)[source]¶

Trains TFIDF’s vocabulary on the given dataloader

Parameters:: dataloader (torch.utils.data.DataLoader) – token dataloader to compute bow features over
Return type:: None

compute_bow_tokens(dataloader)[source]¶

Computes the bow features on the dataloader

Parameters:: dataloader (torch.utils.data.DataLoader) – input dataloader to compute bow features over
Returns:: Bow features for each sample (N x vocab)
Raises:: Exception – if did not call train beforehand
Return type:: jaxtyping.Integer[torch.Tensor, n vocab]