Contributing#
This page contains instructions for contributing to LangCheck.
Installing LangCheck from Source#
To install and run the LangCheck package from your local git repo:
# Install the langcheck package in editable mode with dev dependencies
> python -m pip install -e .[all,dev]
# If you are using zsh, make sure to escape the brackets
> python -m pip install -e .\[all,dev\]
# Try using langcheck
# (If you edit the package, just restart the Python REPL to reflect your changes)
> python
>>> from langcheck.metrics import is_float
>>> is_float(['1', '-2', 'a'])
Metric: is_float
prompt generated_output reference_output metric_value
0 None 1 None 1
1 None -2 None 1
2 None a None 0
Running Static Checks#
We have static checks with ruff and pyright. Please make sure that your code passes those two commands before you submit a pull request.
ruff check src/ tests/
pyright .
Running Tests#
To run tests:
# Run all tests
> python -m pytest -s -vv
# Run non-optional tests only
> python -m pytest -s -vv -m "not optional"
# Run optional tests only (this requires optional Japanese tokenizers like Mecab)
> pip install .[optional]
> python -m pytest -s -vv -m "optional"
Documentation#
To make documentation:
Optional: Re-generate all
docs/langcheck*.rst
files.sphinx-apidoc -f --no-toc --separate --module-first -t docs/_templates/ -o docs src/langcheck/ src/langcheck/stats.py src/langcheck/metrics/model_manager
Warning: This will overwrite all of our custom text in the
.rst
files, so you must check the code diffs forUPDATE_AFTER_SPHINX_APIDOC
comments and manually re-apply them.This is only necessary when you add or remove entire packages/modules. If you only edit existing packages/modules, you can skip this step.
This only modifies the auto-generated
docs/langcheck*.rst
files (the “API Reference” section in the docs). It doesn’t touch theindex.md
and other.md
or.rst
files.This uses autodoc to generate
.rst
files at the package/module-level.
Re-generate all
docs/_build/*.html
files from the raw.rst
and.md
files.make -C docs clean html
This uses autodoc to populate .html files at the function-level.
Note: you’ll see warnings like “more than one target found for cross-reference ‘MetricValue’”. Sphinx seems to get confused when we import a module’s classes into its parent package’s
__init__.py
. This seems to be harmless and there doesn’t seem to be a way to suppress it.
View documentation locally
python -m http.server -d docs/_build/html
Publishing#
To publish a new version of LangCheck:
Increment the version in
pyproject.toml
andsrc/langcheck/__init__.py
Cut a new release on GitHub with release notes
Build the package
> python -m pip install build twine
> python -m build
> twine check dist/*
Publish to PyPi
# Follow auth token instructions at https://pypi.org/manage/account/token/
# TestPyPi
> twine upload -r testpypi dist/*
# PyPi
> twine upload dist/*
Log into ReadTheDocs and activate the new version on the “Versions” tab