pandora_llm.utils.extraction_utils

Module Contents

pandora_llm.utils.extraction_utils.compute_extraction_metrics(ground_truth, generations, ground_truth_statistics, generations_statistics, prefix_length, suffix_length, tokenizer, title=None, statistic_name=None, ground_truth_probabilities=None)[source]

Computes all extraction metrics

Parameters:

  • ground_truth (list[list[int]]) – list of input ids for each ground truth (N x L)

  • generations (list[list[list[int]]]) – list of input ids for each generation (N x G x L)

  • ground_truth_statistics (list[float]) – list of statistics for each ground truth (N)

  • generations_statistics (list[list[float]]) – list of statistics for each generation (N x G)

  • prefix_length (int) – number of prefix input ids

  • suffix_length (int) – number of suffix input_ids

  • tokenizer (AutoTokenizer) – tokenizer used to create the input ids

  • title (str) – title of the experiment

  • statistic_name (str) – name of the statistic

  • ground_truth_probabilities (list[float]) – list of probabilities of generating each ground truth suffix from prefix