Safely installing Python applications and managing additional Python versions

NUIT sometimes see University Ubuntu systems where the graphical desktop is missing because the colleague or student has attempted to remove the system Python, some of the system’s essential Python libraries and applications, or both. Other times, we see system applications not working properly because the colleague or student has installed a different Python version alongside the system Python and then set the new Python version to be the default python3 command system-wide.

This damage leaves the system unusable, preventing the colleague or student from working. The only solution is for NUIT to reinstall the Ubuntu operating system.

What to avoid

  • Do not try to remove or replace the existing Python 3. Python is embedded into Ubuntu so heavily that an attempt to remove the python3 package will remove the Ubuntu desktop and hundreds of other packages, rendering the system unusable.
  • Do not parallel install a second Python and make it answer to python3 instead of the system Python. Redirecting the python3 command to a Python other than the one you get from apt install python3 will stop Ubuntu-specific applications and libraries from working.

Solutions

  • Pyenv allows you to install multiple Python versions into your home directory without affecting the system Python, and switch between them easily. Example: pyenv install 3.8 ; pyenv local 3.8
  • Pipx will install applications written in Python and handle their dependencies elegantly. Limitations are that optional modules that are not marked as dependencies aren’t installed and cannot trivially be added. Example: pipx install --include-deps ipykernel
  • Pip3 is needed to install Python libraries where there is no application component. You should use the --user flag with pip3. Example: pip3 install --user pysqlite3
  • Virtualenv lets you install multiple versions of the same Python application or library to cope with different projects and switch between them. Example: virtualenv myenv ; source myenv/bin/activate

Do not use sudo with any of these tools, they are all designed to be run with ordinary user privileges.

Pyenv doesn’t switch Python versions within a Virtualenv instance, but you can mimic that feature as follows: virtualenv --python ~/.pyenv/versions/3.12.1/bin/python myenv

After installing Pyenv

You need to install some dependencies and set up aliases. Read on…

Install dependencies

Pyenv needs certain libraries and development headers installed via Apt before you can use it to install Pythons:

~ $ pyenv install 3.12
Downloading Python-3.12.1.tar.xz...
-> https://www.python.org/ftp/python/3.12.1/Python-3.12.1.tar.xz
Installing Python-3.12.1...
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/home/campus.ncl.ac.uk/me/.pyenv/versions/3.12.1/lib/python3.12/bz2.py", line 17, in <module>
    from _bz2 import BZ2Compressor, BZ2Decompressor
ModuleNotFoundError: No module named '_bz2'
WARNING: The Python bz2 extension was not compiled. Missing the bzip2 lib?
[many lines omitted]
Installed Python-3.12.1 to /home/campus.ncl.ac.uk/me/.pyenv/versions/3.12.1

You must infer these missing dependencies from the errors that pyenv install 3.12 emits, so what you need may not match the packages listed below. apt search thing and yum search thing will be useful for this.

What I needed to install when writing this guide, to give an example of the kind of dependencies you might need:

sudo apt install libbz2-dev libncurses-dev libffi-dev \
    libreadline-dev tk-dev liblzma-dev

The same example for RedHat:

sudo yum install bzip2-devel ncurses-devel libffi-devel \
    readline-devel tk-devel lzma-devel

Remove the failed Python version before trying again:

rm -rf ~/.pyenv/versions/3.12.1

pyenv install 3.12

On Ubuntu 24.04 systems, you also need to install zlib1g-dev otherwise you get the unhelpfully-opaque:

student@labC1QBRO:~$ pyenv install 3.12
Downloading Python-3.12.5.tar.xz...
-> https://www.python.org/ftp/python/3.12.5/Python-3.12.5.tar.xz
Installing Python-3.12.5...

BUILD FAILED (Ubuntu 24.04 using python-build 20180424)

Inspect or clean up the working tree at /tmp/python-build.20240903154016.7543
Results logged to /tmp/python-build.20240903154016.7543.log

Last 10 log lines:
  File "/tmp/python-build.20240903154016.7543/Python-3.12.5/Lib/ensurepip/__init__.py", line 200, in _bootstrap
    return _run_pip([*args, *_PACKAGE_NAMES], additional_paths)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/tmp/python-build.20240903154016.7543/Python-3.12.5/Lib/ensurepip/__init__.py", line 101, in _run_pip
    return subprocess.run(cmd, check=True).returncode
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/tmp/python-build.20240903154016.7543/Python-3.12.5/Lib/subprocess.py", line 571, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['/tmp/python-build.20240903154016.7543/Python-3.12.5/python', '-W', 'ignore::DeprecationWarning', '-c', '\nimport runpy\nimport sys\nsys.path = [\'/tmp/tmpjoslp4b2/pip-24.2-py3-none-any.whl\'] + sys.path\nsys.argv[1:] = [\'install\', \'--no-cache-dir\', \'--no-index\', \'--find-links\', \'/tmp/tmpjoslp4b2\', \'--root\', \'/\', \'--upgrade\', \'pip\']\nrunpy.run_module("pip", run_name="__main__", alter_sys=True)\n']' returned non-zero exit status 1.
make: *** [Makefile:2027: install] Error 1

Set up the aliases

Pyenv gives you a Python interpreter that answers to the name python. To have it also answer to python3 and switch between the system Python and Pyenv-installed Pythons reliably, run:

sudo apt install python-is-python3

alias python3=python

Then add that alias to your shell’s run control file. The wide variety of shells in use means that we cannot give specific instructions for that.

If you don’t install python-is-python3 and don’t set up the alias, you see outcomes like:

~ $ pyenv local 3.12.1

~ $ python --version
Python 3.12.1

~ $ python3 --version
Python 3.10.12

~ $ pyenv local system  

~ $ python --version   
pyenv: python: command not found

The `python' command exists in these Python versions:
  3.12.1

Pyenv demonstration

The below demonstrates how Pyenv can be used to set up a directory for a project that needs a specific Python version without affecting the Python version used elsewhere:

~ $ pyenv global system

~ $ python3 --version  
Python 3.10.12

~ $ python --version   
Python 3.10.12

~ $ mkdir pyenv-test ; cd pyenv-test

~/pyenv-test $ pyenv local 3.12.1             

~/pyenv-test $ python3 --version               
Python 3.12.1

~/pyenv-test $ python --version               
Python 3.12.1

~/pyenv-test $ cd ..

~ $ python --version
Python 3.10.12

~ $ python3 --version
Python 3.10.12