pandora_llm.features.DCPDD

Module Contents

class pandora_llm.features.DCPDD.DCPDD(*args, **kwargs)[source]

Bases: pandora_llm.features.base.FeatureComputer, pandora_llm.features.base.LLMHandler

Computes the divergence between vocab probability distribution and random internet text. Introduced by Zhang et al. 2024 (https://arxiv.org/pdf/2409.14781).

compute_features(dataloader, accelerator=None, tokenizer=None, mode='primary', smoothing='laplace')[source]

Compute either token-level log probs (mode=”primary”) or reference frequency-based probability (mode=”ref”)

Parameters:
Returns:

Vocab-level log probs

Raises:
Return type:

jaxtyping.Float[torch.Tensor, n seq-1]

static reduce(dataloader, target_log_probs, ref_probs, score_bound=0.01)[source]

Computes divergence between target log probs and reference frequency, applying a score_bound and averaging over the input_ids that are the first occurence in the sequence.

Parameters:

  • dataloader (torch.utils.data.DataLoader) – dataloader that produced the target_log_probs

  • target_log_probs (jaxtyping.Float[torch.Tensor, n seq-1]) – tensor of next token probabilities

  • ref_probs (jaxtyping.Integer[torch.Tensor, vocab]) – reference probability of vocab

  • score_bound (float) – upper bound on divergence score

Returns:

Tensor of divergence (cross-entropy) scores

Return type:

jaxtyping.Float[torch.Tensor, n]

pandora_llm.features.DCPDD.compute_ref_probs(dataloader, tokenizer, smoothing='laplace')[source]

Computes reference probabilities of vocab in given token dataloader (Laplace Smoothed)

Parameters:

  • dataloader (torch.utils.data.DataLoader) – input data to compute vocab frequency over

  • tokenizer (transformers.AutoTokenizer) – tokenizer which contains vocabulary

  • smoothing (str) – smoothing to apply (‘laplace’ or None), defaults to laplace

Returns:

Tensor of vocab counts

Return type:

jaxtyping.Integer[torch.Tensor, vocab]