Running Background Tasks

In this section, we will learn how to run background tasks in Django using Celery. Celery is a powerful task queue that lets you run slow, time-consuming jobs in the background, instead of making the user wait for them to finish. This is really useful for things like sending emails, processing files, or doing heavy calculations, without freezing up your main application.

Redis

Redis gets used for a few different jobs inside a Django app - it works as a message broker for Celery, and it’s also used for caching. It’s an in-memory data store, which means it keeps data in memory (RAM) instead of on disk, and it can be used as a database, a cache, and a message broker, all at once.

Installing Redis

We’ll use Docker to run Redis on our own computer. Add the following service to your docker-compose.yml file:

1. Add the Redis service to your docker-compose.yml:

   services:
       redis:
           image: redis
           container_name: redis
           restart: always
           ports:
               - 6379:6379
           volumes:
               - ./data/redis:/data
           command: ["redis-server", "--appendonly", "yes"] # Enable data persistence (optional)
   
   
       redisinsight:
           image: redislabs/redisinsight
           container_name: redisinsight
           ports:
               - 8001:8001
           volumes:
               - ./data/redisinsight:/db

2. Start the Redis server using Docker Compose:

   docker-compose up -d redis redisinsight

3. To use Redis inside your Django app, install the redis Python client package:

   uv add redis

This will start both the Redis server and RedisInsight, which is a web-based tool you can use to look inside and manage Redis. You can open RedisInsight at http://localhost:8001 to keep an eye on your Redis instance.

Celery

Celery is a powerful task queue that lets you run background tasks without making the main request wait around for them to finish. It helps you move slow, time-consuming jobs away from your main application thread, so your app feels faster and more responsive to users.

Celery Availability

Celery does not work on Windows. If you’re using Windows, it’s best to use a virtual machine or Docker, so you can run Celery inside a Linux environment.

Installing Celery

To install Celery, follow the steps below based on which operating system you’re using:

Windows

Best option: Switch to Linux Ubuntu, or use GitHub Codespaces, to avoid all the manual setup headaches. If you’d still rather run Celery on Windows, you can use WSL (Windows Subsystem for Linux) instead. Here’s how to set it up:

1. Install Celery in your Django project:

   uv add celery

2. Set up WSL (Windows Subsystem for Linux), and then just follow the normal Linux steps to run Celery inside the WSL environment.

   1. Install WSL and a Linux distribution (like Ubuntu) from the Microsoft Store:

      wsl --install

   2. Install Python and the packages you need, inside the WSL environment:

      sudo apt update
      sudo apt install python3 python3-pip

   3. Export your project’s dependencies into a requirements.txt file, so WSL can use them too:

      uv export --format requirements-txt > requirements.txt
      # or
      uv export --no-hashes > requirements.txt
      # or
      uv export --no-dev > requirements.txt
      # or
      uv pip compile pyproject.toml -o requirements.txt

      Update your .gitignore file to include the requirements.txt file and your virtual environment folder, so you don’t accidentally commit them to version control:

      # .gitignore
       requirements.txt
       wsl_venv/

   4. Go to your project folder in the WSL terminal, and create a virtual environment:

      python3 -m venv wsl_venv
      source wsl_venv/bin/activate
      pip install -r requirements.txt

   5. Start the Celery worker inside the WSL terminal:

      celery -A your_project_name worker --loglevel=info

   6. In a second WSL terminal window, start the Celery beat scheduler:

      celery -A your_project_name beat --loglevel=info

Linux/Mac

1. Install Celery in your Django project:

   uv add celery

2. Start the Celery worker, so it can start processing tasks:

   celery -A your_project_name worker --loglevel=info

3. In another terminal, start the Celery beat scheduler, which handles tasks that need to repeat on a schedule:

   celery -A your_project_name beat --loglevel=info

Configuring Celery

To set up Celery in your Django project, follow these steps:

1. Create a celery.py file inside your Django project folder (right next to settings.py):

   # your_project_name/celery.py
   import os
   from celery import Celery
   
   os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'your_project_name.settings')
   
   celery = Celery('your_project_name')
   celery.config_from_object('django.conf:settings', namespace='CELERY')
   celery.autodiscover_tasks()

2. Update your Django __init__.py file, so it includes the Celery app:

   # your_project_name/__init__.py
   from .celery import celery as celery_app
   
   __all__ = ('celery_app',)

3. Set up Celery’s settings inside your settings.py file:

   # settings.py
   from celery.schedules import crontab
   
   CELERY_BROKER_URL = 'redis://localhost:6379/1'
   CELERY_RESULT_BACKEND = 'redis://localhost:6379/2'
   CELERY_BEAT_SCHEDULE = {
       'send-notifications-every-minute': {
           # format: 'your_project_name.app_name.tasks.task_name'
           'task': 'your_project_name.tasks.notify_user',
           # crontab docs: https://docs.celeryq.dev/en/stable/userguide/periodic-tasks.html#crontab-schedules
           'schedule': crontab(minute='*'),  # Run every minute
           'args': ["My Notification Message"],
           'kwargs': {}
       },
   }

   Here, we’re telling Celery to use Redis as both the message broker and the place where it stores task results. Looking at the URLs:

   • redis://localhost:6379/1 uses Redis database 1 as the broker (this is where pending tasks get stored)
   • redis://localhost:6379/2 uses Redis database 2 to store the results of finished tasks

   Using two separate databases like this stops data from different Celery parts from getting mixed up together.

4. Create a tasks.py file in one of your Django apps, where you’ll define your background tasks:

   Shared Task (Recommended)

   # your_project_name/tasks.py
   from celery import shared_task
   
   @shared_task
   def notify_user(message):
       # Simulate a time-consuming task
       import time
       time.sleep(5)
       print(f"Notification sent successfully: {message}")

   Task (Not Recommended)

   # your_project_name/tasks.py
   from your_project_name.celery import celery
   
   @celery.task
   def notify_user(message):
       # Simulate a time-consuming task
       import time
       time.sleep(5)
       print(f"Notification sent successfully: {message}")

5. Start the Celery worker and the beat scheduler, each in its own terminal window:

   # Terminal 1: Start the worker process (processes queued tasks)
   celery -A your_project_name worker --loglevel=info
   
   # Terminal 2: Start the beat scheduler (triggers periodic tasks)
   celery -A your_project_name beat --loglevel=info

Monitoring Celery Tasks

If you want to keep an eye on your Celery tasks, you can use a package called flower. It gives you a web-based dashboard where you can watch your Celery workers and tasks in real time.

1. Install Flower:

   uv add flower

2. Start the Flower server in a new terminal:

   celery -A your_project_name flower --port=5555

3. Open the Flower dashboard at http://localhost:5555 to watch your Celery workers, tasks, and past task history.

Summary

By following this guide, you have:

• Set up Redis as a message broker and a place to store task results
• Set up Celery inside your Django project
• Created background tasks using the @shared_task decorator
• Learned how to run the worker and beat processes
• Turned on monitoring with Flower, so you can see what’s happening with your tasks

Background tasks are a key part of building Django apps that can scale well and handle long, slow jobs without making users wait around.

Tips and Best Practices

Using Shared Tasks

Always use the @shared_task decorator when defining your tasks. It’s more flexible, and it lets you import your tasks from anywhere, without needing direct access to the Celery app instance.

Error Handling

Add error handling and retry logic to your tasks, so they stay reliable even when something goes wrong:

from celery import shared_task
from celery.exceptions import MaxRetriesExceededError

@shared_task(bind=True, max_retries=3)
def notify_user(self, message):
    try:
        # Your task logic here
        print(f"Notification sent successfully: {message}")
    except Exception as exc:
        try:
            self.retry(exc=exc, countdown=60)  # Retry after 60 seconds
        except MaxRetriesExceededError:
            # Handle final failure
            print(f"Failed to send notification: {message}")

Task Result Storage

Always set up CELERY_RESULT_BACKEND, so your task results get stored somewhere. This way, you can check on a task’s status and get its results later using the task’s ID.

Development vs Production

For development, having just one worker and one beat process is enough. For production, it’s better to run several workers, so you get load balancing and your app keeps working even if one worker fails.