Poetry and CI builds -- best practices, knowledge share (travisci), and discussion

[nackjicholson]

• I have searched the issues of this repo and believe that this is not a duplicate.
• I have searched the documentation and believe that my question is not covered.

Question

Hi! I've been working with poetry recently and am happy that I can publish packages to pypi or my own hosted package repository with it. I would like to share my current setup with TravisCI in hopes that it could help others. I would also like to learn from those who have more experience with poetry to get tips on how I might make this setup better, or see how it works with other continuous integration platforms, or other more in depth CI/CD requirements.

My requirements for travis-ci are fairly common.

1. Use poetry to install my project dependencies and dev dependencies.
2. Run lint and unit tests for my project.
3. When the project is git tagged, to build and publish my project to my companies internal pypi registry.

.travis.yml

language: python
python:
- 3.6
env:
  global:
  - secure: "<encrypted MYPYPI_USER=username>"
  - secure: "<encrypted MYPYPI_PASS=p@ssword>"
before_install:
- pip install poetry
install:
- poetry install
script:
- poetry run flake8 my_package test
- poetry run coverage run --source=my_package -m unittest discover -b
before_deploy:
- poetry config repositories.mypypi http://mypypi.example.com/simple
- poetry config http-basic.mypypi $MYPYPI_USER $MYPYPI_PASS
- poetry build -f sdist
deploy:
  provider: script
  script: poetry publish -r mypypi
  skip_cleanup: true
  on:
    tags: true

I have in the past used the built in travis pypi deployment, but it requires a setup.py ( which I don't have anymore! 🙌). So instead I'm running my poetry publish as a script deployment when I tag my repo.

So when master is at a spot where I want to deploy a new version of the package. I do something like.

poetry version minor
git commit -am 'bumped the version'
git tag <version>
# SIDE NOTE: it would be nice to be able to do `git tag $(poetry version --show)`
# or possibly have the bump command output the NEW_VERSION=poetry version minor --output
git push --tags

In order to configure poetry with the credentials to push to our repository I have set $MYPYPI_USER and $MYPYPI_PASS encrypted environment variables in travis.

That's what I have. Cheers 🍺

[epage]

If someone is checking the lock file in, it'd also be nice to include best practices for updating it. Ideally we'd get pyup support for it.

[madig]

Your .travis.yml inspired my own:

language: python
python: "3.6"
dist: xenial

stages:
  - lint
  - test
  - name: deploy
    if: tag IS present

cache:
  pip: true
  directories:
    - "$HOME/.cache/pre-commit"

jobs:
  include:
    - stage: lint
      install:
        - pip install pre-commit
        - pre-commit install-hooks
      script:
        - pre-commit run --all-files
    - stage: test
      install:
        - pip install --upgrade pip
        - pip install poetry
        - poetry install -v
      script:
        - pytest --cov
    - stage: deploy
      script:
        - echo Deploying to PyPI...

before_deploy:
  # User and password environment variables are set as hidden variables through
  # the web interface in the project settings.
  - pip install --upgrade pip
  - pip install poetry
  - poetry config http-basic.pypi $PYPI_USER $PYPI_PASS
  - poetry build

deploy:
  provider: script
  script: poetry publish
  skip_cleanup: true
  on:
    all_branches: true  # Travis recognizes tag names as "branches"
    condition: $TRAVIS_BUILD_STAGE_NAME = Deploy
    repo: daltonmaag/statmake
    tags: true

Notes:

1. I use pre-commit to lint my sources, which I put into its own stage. The lint stage is run before test and deploy is run.
2. Pip packages and the pre-commit installation are cached to speed up stuff.
3. The deploy stage is only enabled if Travis is looking at a tagged commit. It's more of a hack, so the deploy: section at the end can test if it's run from the deploy stage (see the condition). Reading the output of the stages makes me think it would otherwise deploy after every stage, but I'm not sure. That would be a gross misfeature.

[vancouverwill]

on Gitlab setting virtualenvs.in-project to true worked for me and then caching the .venv directory

The relevant section of the .gitlab-ci.yml file are

...

cache:
  paths:
    - .venv
  key: "${CI_COMMIT_REF_SLUG}"

...
  script:
    - poetry config settings.virtualenvs.in-project true
    - poetry install

[matthieugouel]

I came into a slightly different .travis-ci.yml based on your suggestions :

language: python
dist: xenial
python:
  - "3.7"

stages:
  - Quality
  - Publish

before_install:
  - pip install poetry

install:
  - poetry install

jobs:
  include:
    - stage: Quality
      name: lint
      script: make lint
    - name: type
      script: make type
    - name: tests
      script: make tests
    - stage: Publish
      script: skip
      before_deploy:
        - poetry config http-basic.pypi $PYPI_USERNAME $PYPI_PASSWORD
        - poetry build -f sdist
      deploy:
        provider: script
        script: poetry publish
        skip_cleanup: true
        on:
          tags: true
      if: tag IS present

after_success:
  poetry run coveralls

[FranklinYu]

Is it recommended to install poetry through pip? From README:

Be aware, however, that it will also install poetry's dependencies which might cause conflicts.

In addition #166 may be related.

[piccolbo]

@FranklinYu isn't that true of all dev deps, say pytest? To test a package I need pytest, so I typically install all dev deps with poetry install which may create conflicts or, more often for me, hide a missing dep. I had the same problem before using poetry, just didn't think about it.

[dmontagu]

@piccolbo Bad things happen if you install poetry via pip but have it configured not to create a virtual environment. (As far as I can tell this isn't relevant to the configs discussed above though.)

It might be possible to speed up your builds if you don't have poetry create a virtualenvironment, and rely on the installer script to ensure the vendorized dependencies are present. (This probably depends on your CI system to some degree though.)

[FranklinYu]

@piccolbo I'm lost at what you mean. If you add pytest through Poetry then it shouldn't conflict with other dependencies. This is the whole point of Poetry, isn't it?

[piccolbo]

@FranklinYu I consider a conflict when even Poetry can not satisfy all dep constraints. Maybe I am not using the term in the most appropriate way. More generally, I was referring to the fact the dev environment is more complicated than the regular one but the only one in which we can test. So one may be unable to test the environment a user would get in say an empty env. Let's say mypackage requires pandas >=3.14.15 but installing pytest requires pandas >=4.0.0 (I am making this up). Can't test with pandas <4 anymore. Maybe I am worrying for no reason since one can't test all dep combinations anyway.

[FranklinYu]

Let's say mypackage requires pandas >=3.14.15 but installing pytest requires pandas >=4.0.0 (I am making this up).

In this case Poetry will try to come up with a solution (like finding another pytest version). When this is impossible, it would raise an error before trying to install anything. This happens in version resolution, not during installation. This is the point of using Poetry instead of requirements.txt.

[piccolbo]

@FranklinYu @dmontagu thanks for your answers but they address different questions. Unless I find a better way to explain my question, I guess it's better to leave it at that.