Asynchronous Task Queues In The Django World

An asynchronous task queue is one where tasks are executed at a different time from when they are created, and possibly not in the same order they were created.

Task Queue vs Asynchronous Task Queue
In the Django world, both terms are used to describe asynchronous task queue. When some- one writes task queue in the context of Django, they usually mean asynchronous task queue.

Before we get into best practices, let’s go over some definitions:

Broker -: The storage for the tasks themselves. This can be implemented using any sort of persistence tool, although in Django the most common ones in use are RabbitMQ and Redis.

Producer :- The code that adds tasks to the queue to be executed later.

Worker :-The code that takes tasks from the broker and performs them.

Do We Need a Task Queue?

It depends. They add complexity but can really help user experience. Arguably it comes down to whether a particular piece of code causes a bottleneck and can be delayed for later when more free CPU cycles are available.
Here is a useful rule of thumb for determining if a task queue should be used:
Results take time to process: Task queue should probably be used.
Users can and should see results immediately: Task queue should not be used.

Let’s go over some possible use cases:

Please keep in mind there are site-traffic driven exceptions to all of these use cases:

Sites with small-to-medium amounts of traffic may never need a task queue for any of these actions.
Sites with larger amounts of traffic may discover that nearly every user action requires use of a task queue.

Determining whether or not a site or action needs a task is a bit of an art.
There is no easy answer we can provide. However, knowing how to use them is a really powerful tool in any developer’s toolkit.

Choosing Task Queue Software

Celery, Redis Queue, django-background-tasks, which to choose? Let’s go over their pros and cons:

Here is the general rule of thumb:

For most high-to-low volume projects, we recommend Redis Queue.
For high-volume projects with the need for complex task management, we recommend Celery.
For small volume projects, especially for running of periodic batch jobs, we recommend django- background-tasks.

Of course, your own experience and knowledge should be used to determine which task queue system you use for a project. For example, if you have a good amount of Celery experience and are comfortable with it, then by all means use it for small volume or toy projects.