langcheck.augment

langcheck.augment#

langcheck.augment contains all of LangCheck’s text augmentations.

These augmentations can help you automatically generate test cases to evaluate model robustness to different prompts, typos, gender changes, and more.

Currently, only English text augmentations are available. Japanese text augmentations are in development.

Tip

As a shortcut, all English text augmentations are directly accessible from langcheck.augment. For example, you can directly run langcheck.augment.keyboard_typo() instead of langcheck.augment.en.keyboard_typo().

LangCheck’s augmentation functions can take either a single string or a list of strings as input. Optionally, you can set the num_perturbations parameter for most augmentations (except the deterministic ones), which specifies how many perturbed instances to return for each string.

To see more details about each augmentation, refer to the API reference below.

langcheck.augment.change_case(instances: list[str] | str, *, to_case: str = 'uppercase', aug_char_p: float = 1.0, num_perturbations: int = 1, seed: int | None = None) → list[str][source]#

Applies a text perturbation to each string in instances (usually a list of prompts) where some characters are changed to uppercase or lowercase.

Parameters:

  • instances – A single string or a list of strings to be augmented.

  • to_case – Either ‘uppercase’ or ‘lowercase’.

  • aug_char_p – Percentage of all characters that will be augmented.

  • num_perturbations – The number of perturbed instances to generate for each string in instances.

  • seed – The seed for the random number generator. You can fix the seed to deterministically choose which characters to change.

Returns:

A list of perturbed instances.

langcheck.augment.gender(texts: Iterable[str] | str, *, to_gender: str = 'plural') → list[str][source]#

Replace pronouns with that of specified gender.

Parameters:

  • texts – Iterable of texts to be augmented.

  • to_gender – Replacing pronoun type string (‘male’, ‘female’,

  • 'neutral'

  • plural. (or 'plural'). Default to)

Returns:

List of sentences with replaced pronouns.

Note

Replacing neopronouns with other neopronouns is not supported yet because NLTK does not recognize them.

langcheck.augment.jailbreak_template(instances: list[str] | str, templates: list[str] | None = None, *, num_perturbations: int = 1, randomize_order: bool = True, seed: int | None = None) → list[str][source]#

Applies jailbreak templates to each string in instances.

Parameters:

  • instances – A single string or a list of strings to be augmented.

  • templates – A list templates to apply. If None, some templates are randomly selected and used. Available templates are: - basic - chatgpt_dan - chatgpt_good_vs_evil - john - universal_adversarial_suffix

  • num_perturbations – The number of perturbed instances to generate for each string in instances. Should be equal to or less than the number of templates.

  • randomize_order – If True, the order of the templates is randomized. When turned off, num_perturbations needs to be equal to the number of templates.

  • seed – The seed for the random number generator. You can fix the seed to deterministically select the same templates.

Returns:

A list of perturbed instances.

langcheck.augment.keyboard_typo(instances: list[str] | str, *, num_perturbations: int = 1, **kwargs) → list[str][source]#

Applies a keyboard typo text perturbation to each string in instances (usually a list of prompts).

Parameters:

  • instances – A single string or a list of strings to be augmented.

  • num_perturbations – The number of perturbed instances to generate for each string in instances

  • aug_char_p – Percentage of characters (per token) that will be augmented. Defaults to 0.1.

  • aug_char_max – Maximum number of characters which will be augmented. Defaults to None.

  • aug_word_max – Maximum number of words which will be augmented. Defaults to None.

  • include_special_char – Allow special characters to be augmented. Defaults to False.

  • include_numeric – Allow numeric characters to be augmented. Defaults to False.

Note

Any argument that can be passed to nlpaug.augmenter.char.keyboard.KeyboardAug is acceptable. Some of the more useful ones from nlpaug document are listed below:

  • aug_char_p (float): Percentage of character (per token) will be augmented.

  • aug_char_min (int): Minimum number of character will be augmented.

  • aug_char_max (int): Maximum number of character will be augmented.

  • aug_word_p (float): Percentage of word will be augmented.

  • aug_word_min (int): Minimum number of word will be augmented.

  • aug_word_max (int): Maximum number of word will be augmented.

Note that the default values for these arguments may be different from the nlpaug defaults.

Returns:

A list of perturbed instances.

langcheck.augment.ocr_typo(instances: list[str] | str, *, num_perturbations: int = 1, **kwargs) → list[str][source]#

Applies an OCR typo text perturbation to each string in instances (usually a list of prompts).

Parameters:

  • instances – A single string or a list of strings to be augmented.

  • num_perturbations – The number of perturbed instances to generate for each string in instances

  • aug_char_p – Percentage of characters (per token) that will be augmented. Defaults to 0.1.

  • aug_char_max – Maximum number of characters which will be augmented. Defaults to None.

  • aug_word_max – Maximum number of words which will be augmented. Defaults to None.

Note

Any argument that can be passed to nlpaug.augmenter.char.ocr.OcrAug is acceptable. Some of the more useful ones from the nlpaug documentation are listed below:

  • aug_char_p (float): Percentage of characters (per token) that will be augmented.

  • aug_char_min (int): Minimum number of characters that will be augmented.

  • aug_char_max (int): Maximum number of characters that will be augmented.

  • aug_word_p (float): Percentage of words that will be augmented.

  • aug_word_min (int): Minimum number of words that will be augmented.

  • aug_word_max (int): Maximum number of words that will be augmented.

Note that the default values for these arguments may be different from the nlpaug defaults.

Returns:

A list of perturbed instances.

langcheck.augment.payload_splitting(instances: list[str] | str, *, num_perturbations: int = 1, seed: int | None = None) → list[str][source]#

Applies payload splitting augmentation to each string in instances.

Ref: https://arxiv.org/pdf/2302.05733

Parameters:

  • instances – A single string or a list of strings to be augmented.

  • num_perturbations – The number of perturbed instances to generate for each string in instances. Should be equal to or less than the number of templates.

  • seed – The seed for the random number generator. You can fix the seed to deterministically choose the indices to split the instances.

Returns:

A list of perturbed instances.

langcheck.augment.remove_punctuation(instances: list[str] | str, *, aug_char_p: float = 1.0, num_perturbations: int = 1, seed: int | None = None) → list[str][source]#

Applies a text perturbation to each string in instances (usually a list of prompts) where some punctuation is removed.

Parameters:

  • instances – A single string or a list of strings to be augmented.

  • aug_char_p – Percentage of punctuation characters that will be removed.

  • num_perturbations – The number of perturbed instances to generate for each string in instances.

  • seed – The seed for the random number generator. You can fix the seed to deterministically choose which characters to remove.

Returns:

A list of perturbed instances.

langcheck.augment.rephrase(instances: list[str] | str, *, num_perturbations: int = 1, eval_client: EvalClient, eval_prompt_version: str = 'v2') → list[str | None][source]#

Rephrases each string in instances (usually a list of prompts) without changing their meaning. We use a modified version of the prompt presented in “Rethinking Benchmark and Contamination for Language Models with Rephrased Samples” to make an LLM rephrase the given text.

Parameters:

  • instances – A single string or a list of strings to be augmented.

  • num_perturbations – The number of perturbed instances to generate for each string in instances

  • eval_model – The type of model to use.

  • eval_prompt_version – The version of the eval prompt to use when the EvalClient is used. The default version is ‘v2’ (latest).

Returns:

A list of rephrased instances.

langcheck.augment.rephrase_with_system_role_context(instances: list[str] | str, system_role: str, *, num_perturbations: int = 1, eval_client: EvalClient) → list[str | None][source]#

Rephrases each prompt in instances (usually a list of prompts) by adding the specified system role as context to each prompt. This adds context about what role the AI should assume when responding.

For example, if the prompt is “What is the capital of France?” and the role is “teacher”, the augmented prompt might be “You are a teacher and you need to teach geography to your students. Now answer the query: What is the capital of France?”

Parameters:

  • instances – A single prompt or a list of prompts to be augmented.

  • system_role – The role of the system in the augmented prompt.

  • num_perturbations – The number of perturbed instances to generate for each string in instances

  • eval_client – The type of model to use.

Returns:

A list of rephrased instances.

langcheck.augment.rephrase_with_user_role_context(instances: list[str] | str, user_role: str, *, num_perturbations: int = 1, eval_client: EvalClient) → list[str | None][source]#

Rephrases each prompt in instances (usually a list of prompts) by adding the specified user role as context to each prompt. This adds context about the role of the user that is making the request.

For example, if the prompt is “What is the capital of France?” and the role is “student”, the augmented prompt might be “I’m a student working on my geography homework. What is the capital of France?”

Parameters:

  • instances – A single prompt or a list of prompts to be augmented.

  • user_role – The role of the user in the prompt.

  • num_perturbations – The number of perturbed instances to generate for each string in instances

  • eval_client – The type of model to use.

Returns:

A list of rephrased instances.

langcheck.augment.synonym(instances: list[str] | str, *, num_perturbations: int = 1, **kwargs) → list[str][source]#

Applies a text perturbation to each string in instances (usually a list of prompts) where some words are replaced with synonyms.

Parameters:

  • instances – A single string or a list of strings to be augmented.

  • num_perturbations – The number of perturbed instances to generate for each string in instances

  • aug_p – Percentage of words which will be augmented. Defaults to 0.1.

  • aug_max – Maximum number of words which will be augmented. Defaults to None.

Note

Any argument that can be passed to nlpaug.augmenter.word.SynonymAug is acceptable. Some of the more useful ones from the nlpaug documentation are listed below:

  • aug_p (float): Percentage of words which will be augmented.

  • aug_min (int): Minimum number of words that will be augmented.

  • aug_max (int): Maximum number of words that will be augmented.

Note that the default values for these arguments may be different from the nlpaug defaults.

Returns:

A list of perturbed instances.

langcheck.augment.to_full_width(instances: list[str] | str, *, aug_char_p: float = 1.0, num_perturbations: int = 1, seed: int | None = None) → list[str][source]#

Applies a text perturbation to each string in instances (usually a list of prompts) where some ascii characters are converted into full-width characters defined in UTF-8.

Parameters:

  • instances – A single string or a list of strings to be augmented.

  • aug_char_p – Percentage of all characters that will be augmented.

  • num_perturbations – The number of perturbed instances to generate for each string in instances.

  • seed – The seed for the random number generator. You can fix the seed to deterministically choose which characters to change.

Returns:

A list of perturbed instances.