Use virtualenv

In a conversation recently with a friend, we agreed that “something the instructions tell you to do ‘sudo pip install’…which is good, because then you know to ignore them”.

There is never a need for “sudo pip install”, and doing it is an anti-pattern. Instead, all installation of packages should go into a virtualenv. The only exception is, of course, virtualenv (and arguably, pip and wheel). I got enough questions about this that I wanted to write up an explanation about the how, why and why the counter-arguments are wrong.

What is virtualenv?

The documentation says:

virtualenv is a tool to create isolated Python environments.

The basic problem being addressed is one of dependencies and versions, and indirectly permissions. Imagine you have an application that needs version 1 of LibFoo, but another application requires version 2. How can you use both these applications? If you install everything into/usr/lib/python2.7/site-packages (or whatever your platform’s standard location is), it’s easy to end up in a situation where you unintentionally upgrade an application that shouldn’t be upgraded.

Or more generally, what if you want to install an application and leave it be? If an application works, any change in its libraries or the versions of those libraries can break the application.

The tl:dr; is:

  • virtualenv allows not needing administrator privileges
  • virtualenv allows installing different versions of the same library
  • virtualenv allows installing an application and never accidentally updating a dependency

The first problem is the one the “sudo” comment addresses — but the real issues stem from the second and third: not using a virtual environment leads to the potential of conflicts and dependency hell.

How to use virtualenv?

Creating a virtual environment is easy:

$ virtualenv dirname

will create the directory, if it does not exist, and then create a virtual environment in it. It is possible to use it either activated or unactivated. Activating a virtual environment is done by

$ . dirname/bin/activate

this will make python, as well as any script installed using setuptools’ “console_scripts” option in the virtual environment, on the command-execution path. The most important of those is pip, and so using pip will install into the virtual environment.

It is also possible to use a virtual environment without activating it, by directly calling dirname/bin/python or any other console script. Again, pip is an example of those, and used for installing into the virtual environment.

Installing tools for “general use”

I have seen a couple of times the argument that when installing tools for general use it makes sense to install them into the system install. I do not think that this is a reasonable exception for two reasons:

  • It still forces to use root to install/upgrade those tools
  • It still runs into the dependency/conflict hell problems

There are a few good alternatives for this:

  • Create a (handful of) virtual environments, and add them to users’ path.
  • Use “pex” to install Python tools in a way that isolates them even further from system dependencies.

Exploratory programming

People often use Python for exploratory programming. That’s great! Note that since pip 7, pip is building and caching wheels by default. This means that creating virtual environments is even cheaper: tearing down an environment and building a new one will not require recompilation. Because of that, it is easy to treat virtual environments as disposable except for configuration: activate a virtual environment, explore — and whenever needing to move things into production, ‘pip freeze’ will allow easy recreation of the environment.


4 Responses to Use virtualenv

  1. 1. virtualenvwrapper is pretty good. I find myself using `mktmpenv` a lot when mucking around with other people’s code, playing with dependencies, etc.

    2. If you actually want proper isolation and reproducible environments, use nix, which is fundamentally incompatible with virtualenv.

    3. Would you contend that people should use virtualenv even inside a container?

    • moshez says:

      1. I am firmly agnostic on virtualenvwrapper. I don’t use it, but a lot of people I respect (present company included) seem to enjoy it, so it is probably solving some problems well.

      2. I agree Nix is awesome, but I am not sure why it is “fundamentally incompatible with virtualenv”. Can you clarify? [I use virtualenv and nix on my Mac laptop.]

      3. “Containers” is a weird category. If you want to see how I like to do containers, boredbot is my canonical example of my latest thinking. Notice that in the build container, I do use virtualenv — that probably has more to do with the issue that I have not addressed: isolation from system Python packages. In the runtime container, I run a pex file. I’ll note that if you have a pex-oriented workflow, there is indeed little need for a virtualenv.

      • re Nix: What I should have said is that I tried doing Flocker development in a virtualenv on NixOS and the ensuing mess was way too complicated for me to sort out in a timely fashion.

      • Anonymous says:

        Nix is, indeed, “incompatible” with virtualenv. It’s not “fundamentally incompatible” with virtualenv, the Nix Python package has a flaw in it — it does not package sqlite bindings correctly. There are two ways to fix it, and the maintainers are debating 😦

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: