langcheck.metrics.metric_inputs module

langcheck.metrics.metric_inputs module#

class langcheck.metrics.metric_inputs.MetricInputs(individual_inputs: dict[str, IndividualInputType], pairwise_inputs: dict[str, tuple[IndividualInputType, IndividualInputType]] | None = None, required_params: list[str] | None = None, optional_params: list[str] | None = None, input_name_to_prompt_var_mapping: dict[str, str] | None = None)[source]#

Bases: object

A helper class to handle the inputs for the metric in a consistent way.

get_input_list(key: str) → tuple[list[str] | None, list[str] | None] | list[str] | None[source]#

Get the input list for the key.

get_inputs_for_prompt_template(swap_pairwise: bool = False) → list[dict[str, str | None]][source]#

Get the inputs that can be used as arguments for the prompt template. Each item is a dictionary where the keys are the prompt variables specified in the input_name_to_prompt_var_mapping and the values are the input values, which are corresponding elements from the input lists. For pairwise inputs, the values for the first list and the second list are stored in the attributes with the suffixes “_a” and “_b”.

Parameters:

swap_pairwise – If True, swap the pairwise inputs.

get_required_individual_input(key: str) → list[str][source]#

Get the list of a required parameter in individual_inputs. Mainly used for metrics without eval clients.

to_df() → DataFrame[source]#

Convert the inputs to a DataFrame.

validate_template(template_src: str)[source]#

Validate that the given prompt template string is compatible with the input parameters.

Parameters:

template_src – The prompt template string.

langcheck.metrics.metric_inputs.get_metric_inputs(*, generated_outputs: IndividualInputType | tuple[IndividualInputType, IndividualInputType] = None, prompts: IndividualInputType | tuple[IndividualInputType, IndividualInputType] = None, sources: IndividualInputType | tuple[IndividualInputType, IndividualInputType] = None, reference_outputs: IndividualInputType | tuple[IndividualInputType, IndividualInputType] = None, additional_inputs: dict[str, IndividualInputType] | None = None, additional_input_name_to_prompt_var_mapping: dict[str, str] | None = None, required_params: list[str]) → MetricInputs[source]#

Create a metric inputs object with the standard parameters (i.e. generated_outputs, prompts, sources, reference_outputs) and the specified additional parameters.

Parameters:

  • generated_outputs – The generated outputs.

  • prompts – The prompts.

  • sources – The sources.

  • reference_outputs – The reference outputs.

  • additional_inputs – Additional inputs other than the standard ones.

  • additional_input_name_to_prompt_var_mapping – A dictionary that maps the additional input names to the variable names in the prompt template.

  • required_params – A list of required parameters.

Returns:

A MetricInputs object.

langcheck.metrics.metric_inputs.get_metric_inputs_with_required_lists(*, generated_outputs: IndividualInputType | tuple[IndividualInputType, IndividualInputType] = None, prompts: IndividualInputType | tuple[IndividualInputType, IndividualInputType] = None, sources: IndividualInputType | tuple[IndividualInputType, IndividualInputType] = None, reference_outputs: IndividualInputType | tuple[IndividualInputType, IndividualInputType] = None, additional_inputs: dict[str, IndividualInputType] | None = None, additional_input_name_to_prompt_var_mapping: dict[str, str] | None = None, required_params: list[str]) → tuple[MetricInputs, list[list[str]]][source]#

Create a metric inputs object with the standard parameters (i.e. generated_outputs, prompts, sources, reference_outputs) and the specified additional parameters. This function also returns the list of required parameters as raw lists, which is useful for metrics without eval clients.

Parameters:

  • generated_outputs – The generated outputs.

  • prompts – The prompts.

  • sources – The sources.

  • reference_outputs – The reference outputs.

  • additional_inputs – Additional inputs other than the standard ones.

  • additional_input_name_to_prompt_var_mapping – A dictionary that maps the additional input names to the variable names in the prompt template.

  • required_params – A list of required parameters.

Returns:

A MetricInputs object and the required lists.