Pairing problems with solutions
I came across a bug while rebuilding cache_requests. To summarize, python has a built-in
atexit hook used to register clean-up routines. One of my dependencies —
redislite — was using this hook in a way that conflicted with
setuptools. This was causing my tests to incorrectly fail on Tox and Travis.
Not understanding the problem initially, I dumped all my logs and raised an issue with yahoo’s redislite team.
Then, I went on to solve it!
Researching the problem
First I isolated the issue. I used my custom
cookiecutter to scaffold an empty project where I could recreate the bug with the least amount of code.
I did some research and found the exact source of the problem. I found a blog post, where someone ran into the same obscure problem. This author tracked the problem down to
setuptools: between test runs
setuptools deletes all the loaded modules THEN
atexit runs. If your
atexit cleanup function requires modules to run, they will not be there. Really, this seems like a problem with
setuptools, but that set of tools is WAY too complicated for me to get involved with. That blog post even suggested a fix. It was bit convoluted, so I suggested an easier fix and posted a gist to illustrate what my fix would look like. I also demonstrated my solution works in that empty repo I built before.
I reported my new findings to the redislite repo. At this point, my personal obligation to pair problems with solutions is fulfilled.
I went on to experiment with a few solutions. The solution I landed on, conceptually goes like this:
- when registering your cleanup function, also pass in a copy of the current modules;
- when your cleanup function runs, the first thing to do is restore loaded modules using that copy.
This is actually really simple! In python, everything is a
dict — everything, including loaded modules! You access this
sys.modules. So you pass a copy —
sys.modules.copy() — in with your cleanup function, your cleanup function restores the modules.
Several months later, they accepted my PR! I found a problem. I solved. I solved it again. And again. Each time more effective and more permanent than the last. It feels good.
In retrospect, 2 things:
- At the time, I thought of some ways to write tests for this, I should have written those out, and
- A context manager would have been a better way to swap out `sys.modules` — ideally this context manager would swap the copy of modules in to run the function and out when its done.
Next time I revisit those projects, these points will be on my mind.
Over all, this was a fun adventure into the open source community!