0x01 Python venv

I wrote this article out of curiosity about the shebang mechanism, then dug into what Python virtual environments are really doing under the hood. If you are short on time, jump directly to the Conclusion 🙂.

The venv module creates lightweight virtual environments, each with its own isolated set of Python packages installed under the site directory.

A virtual environment is built on top of an existing Python installation (the “base Python”). It can optionally isolate itself from the base environment’s packages, so only packages explicitly installed into the virtual environment are available.

When working inside a virtual environment, command-line installers such as pip install packages into that current environment without extra path declarations.

A virtual environment:

• Contains a specific Python interpreter, project dependencies, and related binaries, and is isolated by default from other Python interpreters/libraries/environments on the OS.
• Usually lives in a project folder named .venv or venv, or in a shared container directory like ~/.virtualenvs.
• Should not be committed to version control (for example, Git).
• Should be treated as disposable: easy to remove and recreate. Project source code should not live inside it.
• Should not be treated as portable/copiable between unknown target machines. Recreate it on each target.

Creating virtual environments

python -m venv /path/to/new/virtual/enviroment

This command creates the target directory (including required parent directories), then creates pyvenv.cfg. The home key points to the Python installation used to create the environment.

It also creates a bin subdirectory (Scripts on Windows), containing either copies or symlinks to the Python executable (platform/config-dependent), and a lib/pythonX.Y/site-packages subdirectory (Lib/site-packages on Windows).

If the target directory already exists, it is reused.

pyvenv.cfg also includes include-system-site-packages, which becomes true when --system-site-packages is used.

Unless --without-pip is provided, ensurepip installs pip into the virtual environment.

You can pass multiple paths to venv; the same environment layout will be created under each provided path.

How venv works

When running the Python interpreter from a virtual environment, sys.prefix and sys.exec_prefix point to the venv directory, while sys.base_prefix and sys.base_exec_prefix point to the base Python used to create the venv.

So sys.prefix != sys.base_prefix is enough to detect whether the current interpreter is running from a virtual environment.

Python 3.12.4 (v3.12.4:8e8a4baf65, Jun  6 2024, 17:33:18) [Clang 13.0.0 (clang-1300.0.29.30)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> sys.prefix
'/Users/Side-Projects/ModelTools/.venv'
>>> sys.base_prefix
'/Library/Frameworks/Python.framework/Versions/3.12'
>>> sys.exec_prefix
'/Users/Side-Projects/ModelTools/.venv'
>>> sys.base_exec_prefix
'/Library/Frameworks/Python.framework/Versions/3.12'

You do not have to activate a virtual environment explicitly, because you can always call the full interpreter path inside the venv.

Also, scripts installed inside the venv are typically directly runnable without activation.

Why this works: scripts installed in a venv include a shebang line pointing to that venv interpreter, for example #!/<path-to-venv>/bin/python. This forces the script to run with that interpreter regardless of the current PATH. On Windows, shebangs also work if Python Launcher for Windows is installed properly.

When a venv is activated, VIRTUAL_ENV is set and prepended into PATH. But because explicit activation is optional, VIRTUAL_ENV alone is not a reliable indicator that a virtual environment is in use.

Conclusion

How creation and runtime mechanics fit together:

1. A venv created by the venv module builds a directory containing: activation scripts, a copy/symlink to the Python interpreter used at creation time, and the package install path lib/pythonX.Y/site-packages.

2. Activating the environment is essentially putting VIRTUAL_ENV/bin at the front of PATH and exporting VIRTUAL_ENV.

   VIRTUAL_ENV='/Users/Side-Projects/ModelTools/.venv'
   if ([ "$OSTYPE" = "cygwin" ] || [ "$OSTYPE" = "msys" ]) && $(command -v cygpath &> /dev/null) ; then
       VIRTUAL_ENV=$(cygpath -u "$VIRTUAL_ENV")
   fi
   export VIRTUAL_ENV
   
   _OLD_VIRTUAL_PATH="$PATH"
   PATH="$VIRTUAL_ENV/bin:$PATH"
   export PATH

3. If your shebang is #!/usr/bin/env python3, then /usr/bin/env resolves python3 via the current PATH. After activation, the venv path is first, so the venv interpreter is selected.

4. If you hardcode the interpreter path in shebang, e.g. #!.venv/bin/python, the script can use that venv interpreter even without explicit activation.