langcheck.metrics.eval_clients

langcheck.metrics.eval_clients#

class langcheck.metrics.eval_clients.AnthropicEvalClient(anthropic_client: Anthropic | None = None, anthropic_args: dict[str, Any] | None = None, *, use_async: bool = False, system_prompt: str | None = None)[source]#

Bases: EvalClient

EvalClient defined for Anthropic API.

get_float_score(metric_name: str, language: str, unstructured_assessment_result: list[str | None], score_map: dict[str, float], *, tqdm_description: str | None = None) → list[float | None][source]#

The function that transforms the unstructured assessments (i.e. long texts that describe the evaluation results) into scores.

Parameters:

metric_name – The name of the metric to be used. (e.g. “toxicity”)
language – The language of the prompts. (e.g. “en”)
unstructured_assessment_result – The unstructured assessment results for the given assessment prompts.
score_map – The mapping from the short assessment results (e.g. “Good”) to the scores.
tqdm_description – The description to be shown in the tqdm bar.

Returns:

A list of scores for the given prompts. The scores can be None if the evaluation fails.

get_text_responses(prompts: Iterable[str], *, tqdm_description: str | None = None) → list[str | None][source]#

The function that gets responses to the given prompt texts. We use Anthropic’s ‘claude-3-haiku-20240307’ model by default, but you can configure it by passing the ‘model’ parameter in the anthropic_args.

Parameters:: prompts – The prompts you want to get the responses for.
Returns:: A list of responses to the prompts. The responses can be None if the evaluation fails.

similarity_scorer()[source]#: Get the BaseSimilarityScorer object that corresponds to the EvalClient so that the similarity-related metrics can be computed. TODO: Intergrate scorer/ with eval_clients/

class langcheck.metrics.eval_clients.AzureOpenAIEvalClient(text_model_name: str | None = None, embedding_model_name: str | None = None, azure_openai_client: AzureOpenAI | None = None, openai_args: dict[str, str] | None = None, *, use_async: bool = False, system_prompt: str | None = None)[source]#

Bases: OpenAIEvalClient

similarity_scorer() → OpenAISimilarityScorer[source]#: This method does the sanity check for the embedding_model_name and then calls the parent class’s similarity_scorer method with the additional “model” parameter. See the parent class for the detailed documentation.

class langcheck.metrics.eval_clients.EvalClient[source]#

Bases: object

An abstract class that defines the interface for the evaluation clients. Most metrics that use external APIs such as OpenAI API call the methods defined in this class to compute the metric values.

compute_metric_values_from_template(metric_inputs: MetricInputs, template: Template, metric_name: str, language: str, score_map: dict[str, float]) → MetricValue[float | None][source]#

Compute the metric values from the given Jinja template with the metric inputs. This function assumes that the template parameters are already validated and the template is ready to be rendered.

Parameters:

metric_inputs – The metric inputs that contain the prompts, generated outputs, reference outputs… etc.
template – The Jinja template that is ready to be rendered.
enforce_pairwise_consistency – Whether to enforce pairwise consistency when computing the metric values.
metric_name – The name of the metric to be used. (e.g. “toxicity”)
language – The language of the prompts. (e.g. “en”)
score_map – The mapping from the short assessment results (e.g. “Good”) to the scores.

Returns:

The metric values computed from the template.

Return type:

MetricValue

The function that transforms the unstructured assessments (i.e. long texts that describe the evaluation results) into scores. A typical workflow can be:

1. Extract a short assessment result strings from the unstructured assessment results.

2. Map the short assessment result strings to the scores using the score_map.

Each concrete subclass needs to define the concrete implementation of this function to enable text scoring.

Parameters:

metric_name – The name of the metric to be used. (e.g. “toxicity”)
language – The language of the prompts. (e.g. “en”)
unstructured_assessment_result – The unstructured assessment results for the given assessment prompts.
score_map – The mapping from the short assessment results (e.g. “Good”) to the scores.
tqdm_description – The description to be shown in the tqdm bar.

Returns:

A list of scores for the given prompts. The scores can be None if the evaluation fails.

get_score(metric_name: str, language: str, prompts: str | Iterable[str], score_map: dict[str, float], *, intermediate_tqdm_description: str | None = None, score_tqdm_description: str | None = None) → tuple[list[float | None], list[str | None]][source]#

Give scores to texts embedded in the given prompts. The function itself calls get_text_responses and get_float_score to get the scores. The function returns the scores and the unstructured explanation strings.

Parameters:

metric_name – The name of the metric to be used. (e.g. “toxicity”)
language – The language of the prompts. (e.g. “en”)
prompts – The prompts that contain the original text to be scored, the evaluation criteria… etc. Typically it is based on the Jinja prompt templates and instantiated withing each metric function.
score_map – The mapping from the short assessment results (e.g. “Good”) to the scores.
intermediate_tqdm_description – The description to be shown in the tqdm bar for the unstructured assessment.
score_tqdm_description – The description to be shown in the tqdm bar for the score calculation.

Returns:

A tuple of two lists. The first list contains the scores for each prompt and the second list contains the unstructured assessment results for each prompt. Both can be None if the evaluation fails.

get_text_responses(prompts: Iterable[str], *, tqdm_description: str | None = None) → list[str | None][source]#

The function that gets responses to the given prompt texts. Each concrete subclass needs to define the concrete implementation of this function to enable text scoring.

Parameters:: prompts – The prompts you want to get the responses for.
Returns:: A list of responses to the prompts. The responses can be None if the evaluation fails.

get_text_responses_with_log_likelihood(prompts: Iterable[str], top_logprobs: int | None = None, *, tqdm_description: str | None = None) → list[TextResponseWithLogProbs | None][source]#

The function that gets responses with log likelihood to the given prompt texts. Each concrete subclass needs to define the concrete implementation of this function to enable text scoring.

Parameters:

prompts – The prompts you want to get the responses for.
top_logprobs – The number of logprobs to return for each token.

Returns:

A list of responses to the prompts. Each response is a tuple of the output text and the list of tuples of the output tokens and the log probabilities. The responses can be None if the evaluation fails.

load_prompt_template(language: str, metric_name: str, eval_prompt_version: str | None = None) → Template[source]#

Gets a Jinja template from the specified language, eval client, metric name, and (optionally) eval prompt version.

Parameters:

language (str) – The language of the template.
metric_name (str) – The name of the metric.
eval_prompt_version (str | None) – The version of the eval prompt. If None, the default version is used.

Returns:

The Jinja template.

Return type:

Template

repeat_requests_from_template(prompt_template_inputs: list[dict[str, str]], template: Template, num_perturbations: int = 1) → list[str | None][source]#

Repeats the request using the given Jinja template for num_perturbations times. Note that every EvalClient subclass is expected to implement get_text_responses method to get different responses for the same input.

Parameters:

instances – A single string or a list of strings to be augmented.
template – The Jinja template ready to be rendered.
num_perturbations – The number of perturbed instances to generate for each string in instances.

Returns:

A list of responses for each input. If num_pertuations is > 1, the multiple responses for the same input are included consecutively.

similarity_scorer() → BaseSimilarityScorer[source]#: Get the BaseSimilarityScorer object that corresponds to the EvalClient so that the similarity-related metrics can be computed. TODO: Intergrate scorer/ with eval_clients/

class langcheck.metrics.eval_clients.GeminiEvalClient(model: GenerativeModel | None = None, model_args: dict[str, Any] | None = None, generate_content_args: dict[str, Any] | None = None, embed_model_name: str | None = None, *, system_prompt: str | None = None)[source]#

Bases: EvalClient

EvalClient defined for the Gemini model.

The function that transforms the unstructured assessments (i.e. long texts that describe the evaluation results) into scores. We leverage the function calling API to extract the short assessment results from the unstructured assessments, so please make sure that the model you use supports function calling (https://ai.google.dev/gemini-api/docs/function-calling#supported-models).

Ref:: https://ai.google.dev/gemini-api/docs/function-calling

Parameters:

metric_name – The name of the metric to be used. (e.g. “toxicity”)
language – The language of the prompts. (e.g. “en”)
unstructured_assessment_result – The unstructured assessment results for the given assessment prompts.
score_map – The mapping from the short assessment results (e.g. “Good”) to the scores.
tqdm_description – The description to be shown in the tqdm bar.

Returns:

A list of scores for the given prompts. The scores can be None if the evaluation fails.

get_text_responses(prompts: Iterable[str], *, tqdm_description: str | None = None) → list[str | None][source]#

The function that gets responses to the given prompt texts.

Parameters:: prompts – The prompts you want to get the responses for.
Returns:: A list of responses to the prompts. The responses can be None if the evaluation fails.

similarity_scorer() → GeminiSimilarityScorer[source]#: Get the BaseSimilarityScorer object that corresponds to the EvalClient so that the similarity-related metrics can be computed. TODO: Intergrate scorer/ with eval_clients/

class langcheck.metrics.eval_clients.LlamaEvalClient(model_name: str = 'tokyotech-llm/Llama-3-Swallow-8B-Instruct-v0.1', torch_dtype: str = 'bfloat16', tensor_parallel_size: int = 1, device: str = 'cuda', *, system_prompt: str | None = None)[source]#

Bases: EvalClient

EvalClient defined for the Llama-based models. It currently only supports English and Japanese. The default model is set to “tokyotech-llm/Llama-3-Swallow-8B-Instruct-v0.1”. The following models are also available: - tokyotech-llm/Llama-3-Swallow-70B-Instruct-v0.1 - elyza/Llama-3-ELYZA-JP-8B - rinna/llama-3-youko-8b-instruct - rinna/llama-3-youko-70b-instruct - meta-llama/Meta-Llama-3.1-8B-Instruct - meta-llama/Meta-Llama-3.1-70B-Instruct To use the 70B models, set tensor_parallel_size to 8 or more. To use the Llama 3.1 models, you need to agree to the terms of service and login with your huggingface account.

get_float_score(metric_name: str, language: str, unstructured_assessment_result: list[str | None], score_map: dict[str, float]) → list[float | None][source]#

The function that transforms the unstructured assessments (i.e. long texts that describe the evaluation results) into scores.

Parameters:

metric_name – The name of the metric to be used. (e.g. “toxicity”)
language – The language of the prompts. (e.g. “en”)
unstructured_assessment_result – The unstructured assessment results for the given assessment prompts.
score_map – The mapping from the short assessment results (e.g. “Good”) to the scores.

Returns:

A list of scores for the given prompts. The scores can be None if the evaluation fails.

get_score(metric_name: str, language: str, prompts: str | Iterable[str], score_map: dict[str, float]) → tuple[list[float | None], list[str | None]][source]#

Parameters:

metric_name – The name of the metric to be used. (e.g. “toxicity”)
language – The language of the prompts. (e.g. “en”)
prompts – The prompts that contain the original text to be scored, the evaluation criteria… etc. Typically it is based on the Jinja prompt templates and instantiated withing each metric function.
score_map – The mapping from the short assessment results (e.g. “Good”) to the scores.

Returns:

A tuple of two lists. The first list contains the scores for each prompt and the second list contains the unstructured assessment results for each prompt. Both can be None if the evaluation fails.

get_text_responses(prompts: Iterable[str], language: str) → list[str | None][source]#

The function that generates responses to the given prompt texts.

Parameters:

prompts – The prompts you want to get the responses for.
language – The language of the prompts. (e.g. “en”)

Returns:

A list of responses to the prompts. The responses can be None if the evaluation fails.

similarity_scorer()[source]#: Get the BaseSimilarityScorer object that corresponds to the EvalClient so that the similarity-related metrics can be computed. TODO: Intergrate scorer/ with eval_clients/

class langcheck.metrics.eval_clients.OpenAIEvalClient(openai_client: OpenAI | None = None, openai_args: dict[str, str] | None = None, *, use_async: bool = False, system_prompt: str | None = None)[source]#

Bases: EvalClient

EvalClient defined for OpenAI API.

The function that transforms the unstructured assessments (i.e. long texts that describe the evaluation results) into scores. We leverage the function calling API to extract the short assessment results from the unstructured assessments, so please make sure that the model you use supports function calling (https://platform.openai.com/docs/guides/gpt/function-calling).

Ref:: https://platform.openai.com/docs/guides/gpt/function-calling

Parameters:

metric_name – The name of the metric to be used. (e.g. “toxicity”)
language – The language of the prompts. (e.g. “en”)
unstructured_assessment_result – The unstructured assessment results for the given assessment prompts.
score_map – The mapping from the short assessment results (e.g. “Good”) to the scores.
tqdm_description – The description to be shown in the tqdm bar.

Returns:

A list of scores for the given prompts. The scores can be None if the evaluation fails.

get_text_responses(prompts: Iterable[str], *, tqdm_description: str | None = None) → list[str | None][source]#

The function that gets responses to the given prompt texts. We use OpenAI’s ‘gpt-turbo-3.5’ model by default, but you can configure it by passing the ‘model’ parameter in the openai_args.

Parameters:: prompts – The prompts you want to get the responses for.
Returns:: A list of responses to the prompts. The responses can be None if the evaluation fails.

get_text_responses_with_log_likelihood(prompts: Iterable[str], top_logprobs: int | None = None, *, tqdm_description: str | None = None) → list[TextResponseWithLogProbs | None][source]#

The function that gets responses with log likelihood to the given prompt texts. Each concrete subclass needs to define the concrete implementation of this function to enable text scoring.

NOTE: Please make sure that the model you use supports logprobs. In Azure OpenAI, the API version 2024-06-01 is the earliest GA version that supports logprobs (https://learn.microsoft.com/en-us/azure/ai-services/openai/whats-new#new-ga-api-release).

Parameters:

prompts – The prompts you want to get the responses for.
top_logprobs – The number of logprobs to return for each token.

Returns:

similarity_scorer() → OpenAISimilarityScorer[source]#: https://openai.com/blog/new-embedding-models-and-api-updates

class langcheck.metrics.eval_clients.OpenRouterEvalClient(openrouter_args: dict[str, str] | None = None, *, system_prompt: str | None = None)[source]#

Bases: EvalClient

EvalClient defined for the OpenRouter API.

The function that transforms the unstructured assessments (i.e. long texts that describe the evaluation results) into scores.

Parameters:

metric_name – The name of the metric to be used. (e.g. “toxicity”)
language – The language of the prompts. (e.g. “en”)
unstructured_assessment_result – The unstructured assessment results for the given assessment prompts.
score_map – The mapping from the short assessment results (e.g. “Good”) to the scores.
tqdm_description – The description to be shown in the tqdm bar.

Returns:

A list of scores for the given prompts. The scores can be None if the evaluation fails.

get_score(metric_name: str, language: str, prompts: str | Iterable[str], score_map: dict[str, float]) → tuple[list[float | None], list[str | None]][source]#

Parameters:

metric_name – The name of the metric to be used. (e.g. “toxicity”)
language – The language of the prompts. (e.g. “en”)
prompts – The prompts that contain the original text to be scored, the evaluation criteria… etc. Typically it is based on the Jinja prompt templates and instantiated withing each metric function.
score_map – The mapping from the short assessment results (e.g. “Good”) to the scores.

Returns:

A tuple of two lists. The first list contains the scores for each prompt and the second list contains the unstructured assessment results for each prompt. Both can be None if the evaluation fails.

get_text_responses(prompts: Iterable[str], *, tqdm_description: str | None = None) → list[str | None][source]#

The function that gets responses to the given prompt texts. The user’s default OpenRouter model is used by default, but you can configure it by passing the ‘model’ parameter in the openrouter_args.

Parameters:: prompts – The prompts you want to get the responses for.
Returns:: A list of responses to the prompts. The responses can be None if the evaluation fails.

similarity_scorer()[source]#: Get the BaseSimilarityScorer object that corresponds to the EvalClient so that the similarity-related metrics can be computed. TODO: Intergrate scorer/ with eval_clients/

class langcheck.metrics.eval_clients.PrometheusEvalClient(model_name: str = 'prometheus-eval/prometheus-7b-v2.0', torch_dtype: str = 'bfloat16', tensor_parallel_size: int = 1, device: str = 'cuda', *, system_prompt: str | None = None)[source]#

Bases: EvalClient

EvalClient defined for the Prometheus 2 model. This eval client currently supports only English. Presented in “Prometheus 2: An Open Source Language Model Specialized in Evaluating Other Language Models” <https://arxiv.org/abs/2405.01535>. We adapted the prompts in <prometheus-eval/prometheus- eval/blob/main/libs/prometheus-eval/prometheus_eval/prompts.py>.

get_float_score(metric_name: str, language: str, unstructured_assessment_result: list[str | None], score_map: dict[str, float]) → list[float | None][source]#

The function that transforms the unstructured assessments (i.e. long texts that describe the evaluation results) into scores. We simple find the assessment result which appeared latest in the unstructured text. :param metric_name: The name of the metric to be used. (e.g. “toxicity”) :param language: The language of the prompts. (e.g. “en”) :param unstructured_assessment_result: The unstructured assessment results

for the given assessment prompts.

Parameters:: score_map – The mapping from the short assessment results (e.g. “Good”) to the scores.
Returns:: A list of scores for the given prompts. The scores can be None if the evaluation fails.

get_score(metric_name: str, language: str, prompts: str | Iterable[str], score_map: dict[str, float]) → tuple[list[float | None], list[str | None]][source]#

Parameters:

metric_name – The name of the metric to be used. (e.g. “toxicity”)
language – The language of the prompts. (e.g. “en”)
prompts – The prompts that contain the original text to be scored, the evaluation criteria… etc. Typically it is based on the Jinja prompt templates and instantiated withing each metric function.
score_map – The mapping from the short assessment results (e.g. “Good”) to the scores.

Returns:

A tuple of two lists. The first list contains the scores for each prompt and the second list contains the unstructured assessment results for each prompt. Both can be None if the evaluation fails.

get_text_responses(prompts: Iterable[str]) → list[str | None][source]#

The function that generates responses to the given prompt texts.

Parameters:: prompts – The prompts you want to get the responses for.
Returns:: A list of responses to the prompts. The responses can be None if the evaluation fails.

load_prompt_template(language: str, metric_name: str, eval_prompt_version: str | None = None) → Template[source]#

Gets a Jinja template from the specified language, eval client, metric name, and (optionally) eval prompt version.

Parameters:

language (str) – The language of the template.
metric_name (str) – The name of the metric.
eval_prompt_version (str | None) – The version of the eval prompt. If None, the default version is used.

Returns:

The Jinja template.

Return type:

Template

similarity_scorer()[source]#: Get the BaseSimilarityScorer object that corresponds to the EvalClient so that the similarity-related metrics can be computed. TODO: Intergrate scorer/ with eval_clients/

langcheck.metrics.eval_clients

Contents

langcheck.metrics.eval_clients#