Not The Same Pre-fork Worker Model
Running into bugs when forking at the wrong time
Intro At my company, we have a web app that monitors websites of investment firms for any new investment deals they make. The web app runs on Python Flask with Gunicorn and queues scraping jobs to Celery workers. Lately, we ran into an issue where a database query returned empty when it should not have. Since this issue occurs sporadically, I suspected it was due to our multithreading code. The ORM that we’re using, SQLAlchemy, warns that session is not thread-safe, and should not be accessed concurrently from multiple threads.
[Read More]