Motivation

When developing with Python, I encountered many situations where I needed to configure the environment. Sometimes I need to install different versions of Python, sometimes I need to install different Python packages, and sometimes I need to install different versions of the same Python package.

At the beginning, I basically followed the tutorials on the Internet. Different Python projects may use different environment management tools, and I have used pip, conda, mamba, virtualenv, venv, etc. These tools have their own advantages and disadvantages. Sometimes I also mix them, which leads to confusion of environment variables and conflicts.

Since there has been no consistency, the Python environment on my computer is quite messy, and I feel confused myself. So I decided to summarize the usage of these tools and their advantages and disadvantages. Finally, I will give some recommendations.

Classification of Python Management Tools

Python management tools can be divided into two categories:

1. Package management tools: used to install, uninstall, and update Python packages, such as pip, conda, mamba, etc.

2. Environment management tools: used to manage Python environments, such as venv, virtualenv, conda, mamba, etc.

Here we mainly discuss environment management tools, because the usage of package management tools is relatively simple, and in most cases we will use pip, so we will not go into details here.

There is also another post about using module to manage software packages and environment variables on Linux, which can be found at “Use Environment Module to Manage Software Packages and Environment Variables in Linux”

Environment Management Tools

venv

venv is a built-in environment management tool in Python after version 3.3, which is the simplest and least troublesome to use.

Usage

1. Create a new environment:

   1	python -m venv .venv

   This will create a .venv folder in the current directory, which contains a new Python environment. Of course, .venv can be replaced with any name you want. Using .venv will create an automatically hidden folder, which is easy to ignore in git.

   Note that python here is the Python version you want to create the environment. If you have multiple Python versions, you can use python3 or python3.10, etc.

2. Activate this environment:

   If you are using a Linux or MacOS system, you can use the following command:

   1	source .venv/bin/activate

   If you are using a Windows system, you can use the following command:

   1	.venv\Scripts\activate

3. Install Python packages:

   After activating the environment, using pip to install Python packages will install the packages into this environment, not the global environment. The installed packages will be placed in the .venv/lib/python3.10/site-packages directory.

   1	pip install numpy

4. Exit the environment:

   You can exit the environment by using the deactivate command.

   1	deactivate

Advantages and Disadvantages

• Advantages:

  • Native tool, simple and easy to use, no need to install additional software.
  • Fast, because no additional software packages need to be downloaded.
  • Environment isolation, will not affect the global environment.
  • Can be created anywhere, no need for administrator privileges.
  • Each environment in each project is in the project directory, which is physically isolated from other projects and not easy to confuse.

• Disadvantages:

  • Each project needs to create a new environment, which can take up a lot of space if there are many projects.
  • Cannot share environments, need to be recreated if multiple projects use the same environment.
  • Not good at switching Python versions, each environment is a Python version and cannot be switched.
  • Only applicable to Python 3.3 and above.

In short, if you don’t care about space usage, don’t need to switch Python versions, and just want the simplest environment management tool, then venv is a good choice.

virtualenv

Since venv has a large space usage problem, someone developed virtualenv, which is a third-party environment management tool that can solve some of the problems of venv.

The usage of virtualenv is basically the same as venv, but you need to install the virtualenv package.

Usage

1. Install virtualenv:

   1	pip install virtualenv

2. Create a new environment:

   1	virtualenv .venv

   If you want to specify the Python version, you can use the following command:

   1	virtualenv -p python3.10 .venv

3. Activate this environment:

   1	.venv\Scripts\activate

4. Install Python packages:

   1	pip install numpy

5. Exit the environment:

   1	deactivate

But the difference between virtualenv and venv is that virtualenv will reuse the system’s Python library and will not reinstall it, so it will take up much less space. If you don’t want to reuse the system’s Python library, you can use the --no-site-packages option to install a completely new Python library in this project.

Advantages and Disadvantages

• Advantages:

  • Can specify Python versions.
  • Can share environments, no need to recreate.
  • Takes up less space, does not reinstall Python libraries.
  • Applicable to Python 2.7 and above.

• Disadvantages:

  • Non-native tool, need to install additional software.

In short, if you care about space usage and want to share environments, and are using Python 2.7 and above, then virtualenv is a good choice.

conda

conda is a very powerful environment management tool that can manage Python environments as well as environments for other languages such as R, Julia, etc.

Relationship between conda, Anaconda, miniconda, mamba, and micromamba

• conda is part of Anaconda. Anaconda is a tool package commonly used by data scientists, which contains many software packages commonly used by data scientists, such as numpy, pandas, scipy, etc. However, some users have complained that the software packages installed by Anaconda are too many and take up too much space, so someone developed miniconda, which is a slim version of Anaconda that only contains conda and some basic software packages.

• Whether it is Anaconda or miniconda, they use the conda package management tool to manage environments. However, conda is single-threaded when installing software packages, so it is slow. Therefore, someone developed mamba, which is an accelerated version of conda and is much faster than conda.

• mamba and conda have almost the same usage, except that mamba is used instead of conda when installing software packages. Users can basically think that the mamba command is an alias of the conda command.

• micromamba is a slim version of mamba. The base environment of micromamba is empty and does not contain any software packages.

Usage (using mamba as an example)

1. Install conda or mamba:

   You can find the minimal conda or mamba installation package on miniforge, download and install it.

2. Create a new environment:

   1	mamba create -n myenv python=3.10

   Here, myenv is the name of the environment, and python=3.10 specifies the Python version.

3. Activate this environment:

   1	conda activate myenv

4. Install Python packages:

   1	mamba install numpy

5. Exit the environment:

   1	conda deactivate

Advantages and Disadvantages

• Advantages:

  • Can specify Python versions.
  • Can share environments, no need to recreate.
  • Can install environments for other languages.
  • Fast, because there is mamba to accelerate.
  • Can activate the environment anywhere, no need to create the environment in the project directory.
  • Can share software packages, no need to download repeatedly.

• Disadvantages:

  • Non-native tool, need to install additional software.
  • The environment is separate from the specific project, and for environments that have not been used for a long time, you may forget what the environment is for. For projects that have not been used for a long time, you may forget which environment the project uses.

In short, if you want a powerful environment management tool and want to install packages quickly, then conda or mamba is a good choice. But when developing a project, it is best to mark which environment the project uses to avoid forgetting.

Freeze Environment

Regardless of which environment management tool you use, you can use the pip freeze command to freeze the environment, that is, save the list of software packages in the current environment to a file.

1

pip freeze > requirements.txt

This will create a requirements.txt file in the current directory, which contains the list of software packages in the current environment. When you need to install these software packages in another environment, you can use the following command:

1

pip install -r requirements.txt

This will install all the software packages listed in the requirements.txt file.

Summary

• If you don’t care about space usage, use Python versions higher than 3.3, don’t need to switch Python versions, and just want the simplest environment management tool, then venv is a good choice.

• If you care about space usage, only want to install packages when needed, you can install micromamba on your main development computer.

• If you don’t care about space usage, you can use Anaconda on your main development computer.

Recommended Usage

• If you don’t care about space usage, and may use many scientific computing packages, you can install Anaconda on your main development computer.

• If you care about space usage, only want to install packages when needed, you can install micromamba on your main development computer.

• If you don’t care about space usage, you can use venv on your test computer or server.