The environment: venv, versions and reproducibility

Why it works on my machine is not a result

Half of the time lost in data Python goes not into code but into the environment the code runs in. The classic story: the script worked in March, you run it again in July and get an error inside a library you never touched. The cause is almost always the same. A package version moved, and you had pinned it nowhere.

A virtual environment in two minutes

A virtual environment is a separate folder with its own set of packages for one specific project. Packages of one project stop interfering with another.

python -m venv .venv
source .venv/bin/activate      # macOS and Linux
.venv\Scripts\activate         # Windows
pip install pandas numpy
pip freeze > requirements.txt

After activate, the terminal prompt shows the environment name in brackets. That is your signal that you are not in the system Python. The pip freeze line records the exact version of everything installed. That file is what makes a run repeatable.

How to check where you actually are

When an import cannot find an installed package, the package is almost never the problem. The problem is that pip installed it into one Python and your code runs in another:

import sys
print(sys.executable)    # which Python is executing this code
print(sys.version)       # its version
import pandas; print(pandas.__version__)

Compare the path from sys.executable with what which python prints (or where python on Windows). Different paths mean you have found the cause. In notebooks the same problem shows up as "pip install succeeded in the terminal but the cell raises ModuleNotFoundError": the notebook kernel points at a different environment.

What to pin and what not to

• The Python version. Write it into the project README as a single line. The difference between 3.9 and 3.12 sometimes breaks syntax outright.
• Versions of the key libraries. pandas, numpy and anything else the numbers depend on.
• Do not commit the .venv folder. It is large and tied to your machine. The repository gets requirements.txt, and everyone rebuilds the environment locally.

Restoring the project on a new machine

python -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt
python -c "import pandas; print(pandas.__version__)"

That last line is not a formality. It verifies that what got installed is what you expected. If the version differs from the one the code was written against, you learn it now rather than halfway through a computation.

A separate trap is pip install without an environment, straight into the system Python. On macOS and on recent Linux distributions that usually ends with an externally-managed-environment error. Nothing is broken: the system is asking you to make a venv. Do not route around it with the break-system-packages flag. Spend the two minutes on the environment instead.

Cheat sheet

• One project, one virtual environment.
• Run pip freeze into requirements.txt right after installing packages.
• ModuleNotFoundError with the package installed is almost always the wrong Python.
• The Python version and the key library versions are part of the result.