pandora_llm.features.TFIDF

Module Contents

class pandora_llm.features.TFIDF.TFIDF

Bases: pandora_llm.features.base.FeatureComputer

TFIDF features

vectorizer

CountVectorizer object

idf

inverse document frequencies for each vocab item

Type:

Float[torch.Tensor, “vocab”]

vectorizer = None

idf = None

compute_features(dataloader, mode)

Computes tfodf features, with mode specifying the type of data the dataloader is holding.

Parameters:

• dataloader (torch.utils.data.DataLoader) – dataloader to compute TFIDF features over

• mode (str) – whether to compute bow on ‘tokens’ or ‘text’

Returns:

TFIDF features for each sample

Raises:

ValueError – if mode is not one of ‘tokens’ or ‘text’

Return type:

jaxtyping.Num[torch.Tensor, n vocab]

train_tfidf_text(dataloader)

Trains TFIDF’s vocabulary on the given dataloader

Parameters:

dataloader (torch.utils.data.DataLoader) – text dataloader to compute bow features over

Return type:

None

compute_tfidf_text(dataloader)

Computes the tfidf features on the dataset

Parameters:

dataloader (torch.utils.data.DataLoader) – input dataloader to compute tfidf features over

Returns:

Bow features for each sample (N x vocab)

Raises:

Exception – if did not call train beforehand

Return type:

jaxtyping.Num[torch.Tensor, n vocab]

train_tfidf_tokens(dataloader)

Trains TFIDF’s vocabulary on the given dataloader

Parameters:

dataloader – token dataloader to compute bow features over

Return type:

None

compute_tfidf_tokens(dataloader)

Computes the tfidf features on the dataset

Parameters:

dataloader (torch.utils.data.DataLoader) – input dataloader to compute tfidf features over

Returns:

Bow features for each sample (N x vocab)

Raises:

Exception – if did not call train beforehand

Return type:

jaxtyping.Num[torch.Tensor, n vocab]