Daniel's Blog

Comparing Python Dependencies between releases

The Problem

My client tried to do a deploy and they encountered this error:

Traceback (most recent call last):
  File "/usr/local/bin/manage.py", line 10, in <module>
    execute_from_command_line(sys.argv)
  File "/usr/local/lib/python2.7/dist-packages/django/core/management/__init__.py", line 363, in execute_from_command_line
    utility.execute()
  File "/usr/local/lib/python2.7/dist-packages/django/core/management/__init__.py", line 307, in execute
    settings.INSTALLED_APPS
  File "/usr/local/lib/python2.7/dist-packages/django/conf/__init__.py", line 56, in __getattr__
    self._setup(name)
  File "/usr/local/lib/python2.7/dist-packages/django/conf/__init__.py", line 41, in _setup
    self._wrapped = Settings(settings_module)
  File "/usr/local/lib/python2.7/dist-packages/django/conf/__init__.py", line 110, in __init__
    mod = importlib.import_module(self.SETTINGS_MODULE)
  File "/usr/lib/python2.7/importlib/__init__.py", line 37, in import_module
    __import__(name)
  File "/srv/amigoserver/amigoserver/django/amigoserver/amigoserver/__init__.py", line 3, in <module>
    from .celery import app as celery_app
  File "/srv/amigoserver/amigoserver/django/amigoserver/amigoserver/celery.py", line 7, in <module>
    import raven
  File "/usr/local/lib/python2.7/dist-packages/raven/__init__.py", line 54, in <module>
    from raven.base import *  # NOQA
  File "/usr/local/lib/python2.7/dist-packages/raven/base.py", line 37, in <module>
    from raven.conf.remote import RemoteConfig
  File "/usr/local/lib/python2.7/dist-packages/raven/conf/remote.py", line 36, in <module>
    DEFAULT_TRANSPORT = discover_default_transport()
  File "/usr/local/lib/python2.7/dist-packages/raven/conf/remote.py", line 18, in discover_default_transport
    from raven.transport.threaded import ThreadedHTTPTransport
  File "/usr/local/lib/python2.7/dist-packages/raven/transport/__init__.py", line 15, in <module>
    from raven.transport.gevent import *  # NOQA
  File "/usr/local/lib/python2.7/dist-packages/raven/transport/gevent.py", line 14, in <module>
    import gevent
  File "/usr/local/lib/python2.7/dist-packages/gevent/__init__.py", line 86, in <module>
    from gevent._hub_local import get_hub
  File "/usr/local/lib/python2.7/dist-packages/gevent/_hub_local.py", line 101, in <module>
    import_c_accel(globals(), 'gevent.__hub_local')
  File "/usr/local/lib/python2.7/dist-packages/gevent/_util.py", line 148, in import_c_accel
    mod = importlib.import_module(cname)
  File "/usr/lib/python2.7/importlib/__init__.py", line 37, in import_module
    __import__(name)
  File "src/gevent/_hub_local.py", line 1, in init gevent._gevent_c_hub_local
ValueError: greenlet.greenlet size changed, may indicate binary incompatibility. Expected 128 from C header, got 40 from PyObject

This was preventing their deployment so they need a fix quick.

Investigation

First thing is to look into the issue. It seems to be an incompatibility between gevent and greenlet? Looking at their requirements.txt I saw that there was an install of gevent there:

$ cat requirements.txt | gevent
gevent==20.9.0
$

However there is no mention of greenlet

$cat requirements.txt | greenlet
$

They had a previous docker image that was working so I pulled both docker images to my local machine and ran them. Had to override the Entrypoint using --entrypoint '' and then have them run bash in order to get a prompt.

Once in the running container, I can use pip list to check what's installed.

The working version had this:

$ pip list
gevent                           20.9.0
...
greenlet                         1.1.2

The non working version had this:

gevent                           20.9.0
...
greenlet                         2.0.1

Now their production images have a "base" image that has all the requirements installed into it. That was replaced a week ago and there hadn't been a deployment until last night.

It seems when the base image was rebuilt, greenlet installed the latest and that was incompatible with the older version of gevent.

So to quickly fix it, I just added the older version of greenlet into the requirements.txt. Then it was a couple of hours of building and testing images until everything was good. For good measure, I added a test into the build script of the base image to ensure that gevent could be imported so this won't happen in the future.

It's important to keep track of your dependencies and your dependencies' dependencies.

This can be checked using pipdeptree

$ pip install pipdeptree
$ pipdeptree -fl
...
gevent==20.9.0
  greenlet==1.1.2
  setuptools==44.1.1
  zope.event==4.5.0
    setuptools==44.1.1
  zope.interface==5.4.0
    setuptools==44.1.1

Recommendations

Obviously their libraries are out of date and need to be updated. That can happen when their production site isn't down. To get it up in the shortest amount of time it's better to just use the known older library and get it running. Now that there is time for an After Action Report, they should prioritize setting up a requirements.txt using pip freeze that comes from their working deployment. Then keep those versions up to date.

The dev team had done that, but hadn't been using their requirements-freeze.txt when building or updating it when a change to the libraries happened. It was just generated then forgotten.