The Journey of Upgrading To Python 3.11 in CircleCI

DALL-E 2 trying to depict my upgrade pain

This week I wanted to upgrade from Python 3.10 to 3.11 within my open source Python project data-file-merge. Whilst upgrading the project itself was trivial, doing to same for my CircleCI pipeline was not.

What Does My CircleCI Pipeline Do?

The project (data-file-merge) provides both a PyPi package and a CLI (Windows, Mac and Linux). How exactly I generate the CLI for 3 separate Operating Systems can be found in another Medium article.

The CircleCI pipeline does the following:

• Runs formatting and unit tests (all branches)
• Builds the CLI .exe/binaries for all three operating systems (release tags on main only).

The bin/.exe is produced by:

• Starting a containerised job using each desired operating system.
• Then utilising Pyinstaller to package everything up and generate the file.

Three bin/.exe files are created in total, each working with the operating system they were created on.

Why Upgrade to Python 3.11?

Two main reasons to upgrade from 3.10 to 3.11:

• Performance improvements: Although the project isn’t intended for high-performance usage, it is always nice to get a “free” performance improvement.
• A Young Project: Data-file-merge is still pre-v1.0 release. If there is ever a time to do a Python major version upgrade, it is now. As the project grows, so will the complexity with this upgrade.

Upgrading the Linux Job

Before:

     docker: 
      - image: cimg/python:3.10

After:

     docker: 
      - image: cimg/python:3.11

Running linux containers in CircleCi feels really smooth. CircleCI provide a docker image with a specific Python version pre-installed and ready for use.

Everything appears to be correctly wired up, the Python Orb worked seamlessly with this new docker image and no further config changes were necessary.

Upgrading the Mac Job

Before:

 - run:
          name: Build Mac CLI
          command: |
            brew install python@3.10
            curl -sSL https://install.python-poetry.org | python3 -
            ...

After:

       - run: 
          name: Build Mac CLI 
          command: | 
            brew install python@3.11 
            curl -sSL https://install.python-poetry.org | python3.11 -
            ...

Upgrading the Mac job was also relatively simple. I was already having to brew install my desired python version then download and install poetry as part of the job. All I had to do was ensure I updated the exact version of Python I was installing via Homebrew. For some reason, I had to be specific with what version of Python I piped my poetry install to.

Upgrading The Windows Job

Before:

      - run:
          name: Build Windows CLI
          shell: cmd.exe
          command: |
            curl -sSL https://install.python-poetry.org | python - && ..\AppData\Roaming\Python\Scripts\poetry install ...

After:

      - run: 
          name: Build Windows CLI 
          shell: cmd.exe 
          command: | 
            choco install pyenv-win -y --force && refreshenv && pyenv install 3.11.0b4 && pyenv global 3.11.0b4 && pyenv shell 3.11.0b4 && curl -sSL https://install.python-poetry.org | python3 - && ..\AppData\Roaming\Python\Scripts\poetry install ...

Well, if all three operating systems worked seamlessly with a simple Python upgrade then the article wouldn’t have been worth writing right?

As usual the culprit of complexity was working with Windows.

It is hard to tell from the code snippets above what exactly has changed because the windows cmd.exe command section within a CircleCI config has a known bug that it can only handle a single line of commands.

I had to prefix the existing command with the following:

choco install pyenv-win -y --force && refreshenv && pyenv install 3.11.0b4 && pyenv global 3.11.0b4 && pyenv shell 3.11.0b4

The circleci/windows@5.0 orb provides you with various “executors”. Essentially windows containers. the default executor uses Windows Server 2022 which comes with Python 3.10.2 preinstalled.

So I had to meddle with Python versioning on a remote Windows container which I had no control over pre-configuring. My idea of hell!

How did I succeed?:

• Install pyenv via chocolatey.
• Refresh the environment.
• Install a specific version of Python 3.11 via pyenv.
• Set that 3.11 as the Python version globally (so the 3.10 pre-installed does not interfere).
• Initialise a pyenv shell using Python 3.11

Conclusion

Yes, ok… Now that I worked through the debugging and endless failed CI/CD, the solutions all look quick, logical and simple. But believe me, the process of getting here included a full day of errors, python mismatching (looking at you Windows) and poor CircleCI documentation. I hope this saves someone else a day of debugging!

P.S. Always welcome to hear from others with better ways of doing this Python upgrade so I know for next time.