Running Background Tasks

In this section, we will learn how to run background tasks in Django using Celery, a powerful task queue that allows you to execute time-consuming operations asynchronously. This is particularly useful for tasks like sending emails, processing files, or performing complex calculations without blocking the main application.

Redis

Redis is used for multiple purposes in a Django application, including as a message broker for Celery and for caching. It is an in-memory data structure store that can be used as a database, cache, and message broker.

Installing Redis

We will use Docker to run Redis locally. Add the following service to your docker-compose.yml file:

1. Add the Redis service to your docker-compose.yml:

   services:
       redis:
           image: redis
           container_name: redis
           restart: always
           ports:
               - 6379:6379
           volumes:
               - ./data/redis:/data
           command: ["redis-server", "--appendonly", "yes"] # Enable data persistence (optional)
   
   
       redisinsight:
           image: redislabs/redisinsight
           container_name: redisinsight
           ports:
               - 8001:8001
           volumes:
               - ./data/redisinsight:/db

2. Start the Redis server using Docker Compose:

   docker-compose up -d redis redisinsight

3. To use Redis in your Django application, install the redis Python client package:

   uv add redis

This will start both the Redis server and RedisInsight, a web-based tool for managing Redis. You can access RedisInsight at http://localhost:8001 to monitor your Redis instance.

Celery

Celery is a powerful task queue that allows you to run background tasks asynchronously. It helps you to offload time-consuming tasks from the main application thread, improving the responsiveness of your application.

Celery Availability

Celery does not work on Windows. If you are using Windows, consider using a virtual machine or Docker to run Celery in a Linux environment.

Installing Celery

To install Celery, follow these steps based on your operating system:

Windows

Best option: Switch to Linux Ubuntu or use GitHub Codespaces to avoid manual configuration headaches. If you prefer to run Celery on Windows, you can use WSL (Windows Subsystem for Linux). Here’s how to set it up:

1. Install Celery in your Django project:

   uv add celery

2. Setup WSL (Windows Subsystem for Linux) and follow the Linux instructions to run Celery in the WSL environment.

   1. Install WSL and a Linux distribution (e.g., Ubuntu) from the Microsoft Store:

      wsl --install

   2. Install Python and required packages in the WSL environment:

      sudo apt update
      sudo apt install python3 python3-pip

   3. Export your project dependencies to a requirements.txt file for the WSL environment:

      uv export --format requirements-txt > requirements.txt
      # or
      uv export --no-hashes > requirements.txt
      # or
      uv export --no-dev > requirements.txt
      # or
      uv pip compile pyproject.toml -o requirements.txt

      Update your .gitignore file to include the requirements.txt and virtual environment directory to avoid committing them to version control:

      # .gitignore
       requirements.txt
       wsl_venv/

   4. Navigate to your project directory in the WSL terminal and create a virtual environment:

      python3 -m venv wsl_venv
      source wsl_venv/bin/activate
      pip install -r requirements.txt

   5. Start the Celery worker in the WSL terminal:

      celery -A your_project_name worker --loglevel=info

   6. In another WSL terminal window, start the Celery beat scheduler:

      celery -A your_project_name beat --loglevel=info

Linux/Mac

1. Install Celery in your Django project:

   uv add celery

2. Start the Celery worker to process tasks:

   celery -A your_project_name worker --loglevel=info

3. In another terminal, start the Celery beat scheduler for periodic tasks:

   celery -A your_project_name beat --loglevel=info

Configuring Celery

To configure Celery in your Django project, follow these steps:

1. Create a celery.py file in your Django project directory (same level as settings.py):

   # your_project_name/celery.py
   import os
   from celery import Celery
   
   os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'your_project_name.settings')
   
   celery = Celery('your_project_name')
   celery.config_from_object('django.conf:settings', namespace='CELERY')
   celery.autodiscover_tasks()

2. Update your Django __init__.py file to include the Celery app:

   # your_project_name/__init__.py
   from .celery import celery as celery_app
   
   __all__ = ('celery_app',)

3. Configure Celery settings in your settings.py file:

   # settings.py
   from celery.schedules import crontab
   
   CELERY_BROKER_URL = 'redis://localhost:6379/1'
   CELERY_RESULT_BACKEND = 'redis://localhost:6379/2'
   CELERY_BEAT_SCHEDULE = {
       'send-notifications-every-minute': {
           # format: 'your_project_name.app_name.tasks.task_name'
           'task': 'your_project_name.tasks.notify_user',
           # crontab docs: https://docs.celeryq.dev/en/stable/userguide/periodic-tasks.html#crontab-schedules
           'schedule': crontab(minute='*'),  # Run every minute
           'args': ["My Notification Message"],
           'kwargs': {}
       },
   }

   Here, we configure Redis as both the message broker and result backend for Celery. In the URLs:

   • redis://localhost:6379/1 uses Redis database 1 as the broker (stores pending tasks)
   • redis://localhost:6379/2 uses Redis database 2 for storing task results

   Using separate databases prevents data conflicts between different Celery components.

4. Create a tasks.py file in one of your Django apps to define your background tasks:

   Shared Task (Recommended)

   # your_project_name/tasks.py
   from celery import shared_task
   
   @shared_task
   def notify_user(message):
       # Simulate a time-consuming task
       import time
       time.sleep(5)
       print(f"Notification sent successfully: {message}")

   Task (Not Recommended)

   # your_project_name/tasks.py
   from your_project_name.celery import celery
   
   @celery.task
   def notify_user(message):
       # Simulate a time-consuming task
       import time
       time.sleep(5)
       print(f"Notification sent successfully: {message}")

5. Start the Celery worker and beat scheduler in separate terminal windows:

   # Terminal 1: Start the worker process (processes queued tasks)
   celery -A your_project_name worker --loglevel=info
   
   # Terminal 2: Start the beat scheduler (triggers periodic tasks)
   celery -A your_project_name beat --loglevel=info

Monitoring Celery Tasks

To monitor Celery tasks, you can use the flower package, which provides a web-based dashboard for monitoring Celery workers and tasks in real-time.

1. Install Flower:

   uv add flower

2. Start the Flower server in a new terminal:

   celery -A your_project_name flower --port=5555

3. Access the Flower dashboard at http://localhost:5555 to monitor your Celery workers, tasks, and execution history.

Summary

By following this guide, you have:

• Set up Redis as a message broker and result backend
• Configured Celery in your Django project
• Created background tasks using the @shared_task decorator
• Learned how to run worker and beat processes
• Enabled monitoring with Flower for visibility into task execution

Background tasks are essential for building scalable Django applications that can handle long-running operations without blocking user requests.

Tips and Best Practices

Using Shared Tasks

Always use the @shared_task decorator for defining tasks. It’s more flexible and allows you to import tasks without requiring access to the Celery app instance.

Error Handling

Add error handling and retry logic to your tasks to ensure reliability:

from celery import shared_task
from celery.exceptions import MaxRetriesExceededError

@shared_task(bind=True, max_retries=3)
def notify_user(self, message):
    try:
        # Your task logic here
        print(f"Notification sent successfully: {message}")
    except Exception as exc:
        try:
            self.retry(exc=exc, countdown=60)  # Retry after 60 seconds
        except MaxRetriesExceededError:
            # Handle final failure
            print(f"Failed to send notification: {message}")

Task Result Storage

Always configure CELERY_RESULT_BACKEND to store task results. This allows you to track task status and retrieve results using task IDs.

Development vs Production

For development, a single worker and beat process is sufficient. For production, run multiple workers for load balancing and fault tolerance.