Month: December 2023

Safely installing Python applications and managing additional Python versions

NUIT sometimes see University Ubuntu systems where the graphical desktop is missing because the colleague or student has attempted to remove the system Python, some of the system’s essential Python libraries and applications, or both. Other times, we see system applications, such as the Software Updater and the ubuntu-drivers driver selection tool, not working properly because the colleague or student has installed a different Python version alongside the system Python and then set the new Python version to be the default python3 command system-wide.

This damage often leaves the system unusable, preventing the colleague or student from working. The only solution is for NUIT to reinstall the Ubuntu operating system.

What to avoid

  • Do not try to remove or replace the existing Python 3. Python is embedded into Ubuntu so heavily that an attempt to remove the python3 package will remove the Ubuntu desktop and hundreds of other packages, rendering the system unusable.
  • Do not parallel install a second Python and make it answer to python3 instead of the system Python. Redirecting the python3 command to a Python other than the one you get from apt install python3 will stop Ubuntu-specific applications and libraries from working.

Solutions

  • Pyenv allows you to install multiple Python versions into your home directory without affecting the system Python, and switch between them easily. Example: pyenv install 3.8 ; pyenv local 3.8
  • Pipx will install applications written in Python and handle their dependencies elegantly. Limitations are that optional modules that are not marked as dependencies aren’t installed and cannot trivially be added. Example: pipx install --include-deps ipykernel
  • Pip3 is needed to install Python libraries where there is no application component. You should use the --user flag with pip3. Example: pip3 install --user pysqlite3
  • Virtualenv lets you install multiple versions of the same Python application or library to cope with different projects and switch between them. Example: virtualenv myenv ; source myenv/bin/activate

Do not use sudo with any of these tools, they are all designed to be run with ordinary user privileges.

Pyenv doesn’t switch Python versions within a Virtualenv instance, but you can mimic that feature as follows: virtualenv --python ~/.pyenv/versions/3.12.1/bin/python myenv

Quickstart for Pyenv

Install dependencies

You, or your NUIT support staff, need to install some dependencies as root:

sudo apt update; sudo apt install make build-essential libssl-dev zlib1g-dev \
libbz2-dev libreadline-dev libsqlite3-dev curl git \
libncursesw5-dev xz-utils tk-dev libxml2-dev libxmlsec1-dev libffi-dev liblzma-dev \
libzstd-dev

On Ubuntu 24.04 systems, you also need to install zlib1g-dev otherwise you get the unhelpfully-opaque:

student@labC1QBRO:~$ pyenv install 3.12
Downloading Python-3.12.5.tar.xz...
-> https://www.python.org/ftp/python/3.12.5/Python-3.12.5.tar.xz
Installing Python-3.12.5...

BUILD FAILED (Ubuntu 24.04 using python-build 20180424)

Inspect or clean up the working tree at /tmp/python-build.20240903154016.7543
Results logged to /tmp/python-build.20240903154016.7543.log

Last 10 log lines:
  File "/tmp/python-build.20240903154016.7543/Python-3.12.5/Lib/ensurepip/__init__.py", line 200, in _bootstrap
    return _run_pip([*args, *_PACKAGE_NAMES], additional_paths)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/tmp/python-build.20240903154016.7543/Python-3.12.5/Lib/ensurepip/__init__.py", line 101, in _run_pip
    return subprocess.run(cmd, check=True).returncode
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/tmp/python-build.20240903154016.7543/Python-3.12.5/Lib/subprocess.py", line 571, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['/tmp/python-build.20240903154016.7543/Python-3.12.5/python', '-W', 'ignore::DeprecationWarning', '-c', '\nimport runpy\nimport sys\nsys.path = [\'/tmp/tmpjoslp4b2/pip-24.2-py3-none-any.whl\'] + sys.path\nsys.argv[1:] = [\'install\', \'--no-cache-dir\', \'--no-index\', \'--find-links\', \'/tmp/tmpjoslp4b2\', \'--root\', \'/\', \'--upgrade\', \'pip\']\nrunpy.run_module("pip", run_name="__main__", alter_sys=True)\n']' returned non-zero exit status 1.
make: *** [Makefile:2027: install] Error 1

Download and configure PyEnv

curl -fsSL https://pyenv.run | bash

Then get the shell configuration for your shell.

Optional shell alias

Pyenv gives you a Python interpreter that answers to the name python. To have it also answer to python3 and switch between the system Python and Pyenv-installed Pythons reliably, run:

sudo apt install python-is-python3

alias python3=python

Then add that alias to your shell’s run control file. The wide variety of shells in use means that we cannot give specific instructions for that.

If you don’t install python-is-python3 and don’t set up the alias, you see outcomes like:

~ $ pyenv local 3.12.1

~ $ python --version
Python 3.12.1

~ $ python3 --version
Python 3.10.12

~ $ pyenv local system  

~ $ python --version   
pyenv: python: command not found

The `python' command exists in these Python versions:
  3.12.1

Pyenv demonstration

The below demonstrates how Pyenv can be used to set up a directory for a project that needs a specific Python version without affecting the Python version used elsewhere:

~ $ pyenv global system

~ $ python3 --version  
Python 3.10.12

~ $ python --version   
Python 3.10.12

~ $ mkdir pyenv-test ; cd pyenv-test

~/pyenv-test $ pyenv local 3.12.1             

~/pyenv-test $ python3 --version               
Python 3.12.1

~/pyenv-test $ python --version               
Python 3.12.1

~/pyenv-test $ cd ..

~ $ python --version
Python 3.10.12

~ $ python3 --version
Python 3.10.12