Testing APIs

Testing is essential for ensuring your API works correctly, handles edge cases, and prevents regressions. Django provides a built-in testing framework (based on Python’s unittest), but many developers prefer pytest for its cleaner syntax and powerful features. This guide focuses on testing behavior, not implementation details.

In this section we will cover Automated Testing and Performance Testing. We will learn how to write tests that verify the API’s behavior from the user’s perspective, and how to measure and optimize the performance of your API.

Testing Behavior vs. Implementation

The Microwave Analogy

Imagine testing a microwave. When you want to verify it works:

Correct Way (Test Behavior):

• Press the start button
• Observe the monitor displays a running timer for the requested duration
• Verify the heating stops when the timer ends
• You test what the user sees and expects

Wrong Way (Test Implementation):

• Open the microwave casing
• Test every transistor’s electrical signals
• Verify internal circuit board connections
• You’re testing internal details that users don’t care about

The Same Applies to Software

Many developers fail at automated testing because they test implementation details instead of behavior. Here’s the problem:

• Implementation changes frequently: You might replace a function-based view with a class-based view, or split a model into two models, or combine models together
• Tests become brittle: When you test internal details, your tests break every time you refactor
• Maintenance nightmare: You spend time fixing tests instead of fixing actual bugs
• False confidence: Tests that check implementation details don’t guarantee the API actually works for users

The solution? Test the API’s behavior—what users see and experience—not how you implemented it internally.

What to Test?

Focus on Behavior, Not Implementation

When testing your API, ask yourself: “How should this API behave from the user’s perspective?”

Your tests should verify:

• What HTTP status codes are returned
• What data is in the response
• How the API responds to invalid input
• Whether authorization rules are enforced
• Whether the user-facing functionality works

Your tests should NOT verify:

• Which Django view class is used (function-based vs. class-based)
• How data flows through internal helper functions
• Whether you’re using Django ORM, raw SQL, or an external service
• Internal database structure or optimization techniques

Example: Testing an API Endpoint

Let’s say your API has a POST /collections/ endpoint for creating collections. Here’s how to test its behavior (not implementation):

Scenario 1: Unauthenticated Request

• Client sends POST request without authentication token
• Expected behavior: Return 401 Unauthorized
• Why: The API should reject unauthenticated requests

Scenario 2: Authenticated but Unauthorized Request

• Client sends POST request with valid token, but user is not an admin
• Expected behavior: Return 403 Forbidden
• Why: Only admins should create collections

Scenario 3: Missing Required Data

• Admin client sends POST request without collection name
• Expected behavior: Return 400 Bad Request with error message in response body
• Why: The API should validate required fields

Scenario 4: Valid Request

• Admin client sends POST request with collection name
• Expected behavior: Return 201 Created with the new collection’s ID in response body
• Why: The API should successfully create the resource

Notice we didn’t test:

• Whether the view is function-based or class-based
• How the serializer validates data internally
• The exact SQL queries executed
• Where the response is constructed

We only tested how the API behaves from the outside.

Reference Code for Testing Example

# models.py
from django.db import models
class Collection(models.Model):
    name = models.CharField(max_length=255)
    created_at = models.DateTimeField(auto_now_add=True)

class Product(models.Model):
name = models.CharField(max_length=255)
collection = models.ForeignKey(Collection, related_name='products', on_delete=models.CASCADE)
created_at = models.DateTimeField(auto_now_add=True)

# serializers.py
from rest_framework import serializers
from .models import Collection, Product

class CollectionSerializer(serializers.ModelSerializer):
    class Meta:
        model = Collection
        fields = ['id', 'name']

class ProductSerializer(serializers.ModelSerializer):
    class Meta:
        model = Product
        fields = ['id', 'name', 'collection_id']

# permissions.py
from rest_framework import permissions
class IsAdminOrReadOnly(permissions.BasePermission):
    def has_permission(self, request, view):
        if request.method in permissions.SAFE_METHODS:
            return True
        return request.user and request.user.is_staff

# views.py
from rest_framework import status, viewsets
from .serializers import CollectionSerializer
from .models import Collection, Product
from .permissions import IsAdminOrReadOnly

class CollectionViewSet(viewsets.ModelViewSet):
    queryset = Collection.objects.all()
    serializer_class = CollectionSerializer
    permission_classes = [IsAdminOrReadOnly]

class ProductViewSet(viewsets.ModelViewSet):
    queryset = Product.objects.all()
    serializer_class = ProductSerializer
    permission_classes = [IsAdminOrReadOnly]

# urls.py
from django.urls import path, include
from rest_framework.routers import DefaultRouter
from .views import CollectionViewSet, ProductViewSet

router = DefaultRouter()
router.register(r'collections', CollectionViewSet, basename='collection')
router.register(r'products', ProductViewSet, basename='product')
urlpatterns = [
    path('', include(router.urls)),
]
# /collections/
# /collections/{id}/
# /products/
# /products/{id}/

Pytest

While Django’s built-in testing framework is powerful, many developers prefer pytest for its cleaner syntax, better error messages, and powerful features like fixtures and parameterization.

Setting Up Pytest

1. Install pytest:

   uv add --dev pytest pytest-django

2. Create a pytest.ini file in your project root with the following content:

   [pytest]
   DJANGO_SETTINGS_MODULE = your_project_name.settings

3. Follow the Pytest conventions.

   • Directory structure: Place your tests in a tests/ directory inside each app, or in a top-level tests/ directory.
   • Test file naming: Name your test files starting with test_ (e.g., test_views.py, test_models.py).
   • Test function naming: Name your test functions starting with test_ (e.g., test_create_collection(), test_unauthorized_access()).
   • Test classes: You can group related tests in classes that start with Test (e.g., class TestCollectionAPI:), but this is optional.

4. AAA Pattern: Follow the Arrange-Act-Assert pattern in your tests for clarity:

   • Arrange: Set up the test data and environment
   • Act: Perform the action being tested (e.g., make an API request)
   • Assert: Verify the expected outcome (e.g., check response status code and data)

5. Use pytest features:

   • Fixtures: Use @pytest.fixture to create reusable test data and setup code
   • Parameterization: Use @pytest.mark.parametrize to run the same test with different inputs
   • Markers: Use custom markers (e.g., @pytest.mark.slow) to categorize tests and control which ones run

6. Run your tests with pytest:

   uv run pytest
   # or for specific test file
   uv run pytest path/to/test_views.py
   # or for specific test class
   uv run pytest path/to/test_views.py::TestCollectionAPI
   # or for specific test function
   uv run pytest path/to/test_views.py::test_create_collection
   # or for specific test under a class
   uv run pytest path/to/test_views.py::TestCollectionAPI::test_create_collection
   # or a test having a text in its function name
   uv run pytest -k "anonymous"

7. Continuous Testing: Use pytest-watch to automatically run tests when files change. Run it in a separate terminal:

   uv add --dev pytest-watch
   uv run ptw

Example

from rest_framework.test import APIClient
from rest_framework import status
import pytest

@pytest.mark.django_db
class TestCollectionAPI:
    def test_if_user_is_anonymous(self):
        # Arrange
        client = APIClient()

        # Act
        response = client.post('/collections/', {'name': 'My Collection'})

        # Assert
        assert response.status_code == status.HTTP_401_UNAUTHORIZED

Skipping Tests

If you have tests that are not relevant in certain environments (e.g., tests that require external services), you can skip them using the @pytest.mark.skip decorator:

@pytest.mark.skip(reason="Requires external service")
def test_external_service_integration():
    # Test code that interacts with an external service
    pass

VS Code Pytest Integration

VS Code has excellent support for pytest. You can run and debug your tests directly from the editor. To set it up:

1. Install the Python extension for VS Code.
2. Open the command palette (Ctrl+Shift+P) and select “Python: Configure Tests”.
3. Choose pytest as the test framework.
4. Follow the prompts to specify the test directory and pattern (Recommended choice: . Root Directory). Once configured, you can run tests by clicking the “Run Test” or “Debug Test” links that appear above each test function in the editor.

Advantage of using pytest in VS Code:

• Run individual tests or test classes with a single click
• Debug tests with breakpoints and step-through debugging
• View test results and error messages in the integrated terminal

Writing Pytest Tests

Here we use pytest for testing our API endpoints.

DB Access

When writing tests that interact with the database, use the @pytest.mark.django_db decorator to ensure that the test has access to the database. This allows you to create test data and verify database interactions without affecting your production data.

Best Practices

Whenever You write test for first time first try to comment that code which cause to pass the test and then run test if it fails then you can uncomment that code and run test again to see it pass. This way you can be sure that your test is actually testing the code and not just passing by coincidence.

Simple Testing

Here’s an example of how to write a test for the POST /collections/ endpoint using pytest:

from rest_framework.test import APIClient
from rest_framework import status
import pytest

@pytest.mark.django_db
class TestCollectionAPI:
    def test_if_user_is_anonymous(self):
        # Arrange
        client = APIClient()

        # Act
        response = client.post('/collections/', {'name': 'My Collection'})

        # Assert
        assert response.status_code == status.HTTP_401_UNAUTHORIZED

Authenticate User

To test an authenticated endpoint, create a real user and pass that user to the test client:

The django_user_model fixture comes from pytest-django. It gives you the active user model for the project, so you can create real user objects in your tests. This is useful because it makes your tests work with the database and authentication system in the same way a real request would.

from rest_framework import status
import pytest
from rest_framework.test import APIClient

@pytest.mark.django_db
class TestCollectionAPI:
    def test_if_user_is_authenticated_but_not_admin(self, django_user_model):
        # Arrange
        client = APIClient()
        user = django_user_model.objects.create_user(
            username='regular-user',
            password='password123',
        )
        client.force_authenticate(user=user)

        # Act
        response = client.post('/collections/', {'name': 'My Collection'})

        # Assert
        assert response.status_code == status.HTTP_403_FORBIDDEN

Here we create a real user object, then authenticate the client with that user. Because the user is not a staff member, the API should return 403 Forbidden.

Multiple assertions

You can also include multiple assertions in a single test to verify different aspects of the response:

from rest_framework import status
import pytest
from rest_framework.test import APIClient

@pytest.mark.django_db
class TestCollectionAPI:
    def test_create_collection(self, django_user_model):
        # Arrange
        client = APIClient()
        admin_user = django_user_model.objects.create_user(
            username='admin-user',
            password='password123',
            is_staff=True,
        )
        client.force_authenticate(user=admin_user)

        # Act
        response = client.post('/collections/', {'name': 'My Collection'})

        # Assert
        assert response.status_code == status.HTTP_201_CREATED
        assert 'id' in response.data
        assert response.data['id'] > 0
        assert response.data['name'] == 'My Collection'

Here we create a real staff user and authenticate the client with it. That makes the example match a real admin flow and clearly shows why the request should succeed.

Fixtures

Pytest fixtures are a clean way to share setup code between tests. Use them when the same test data, client setup, or login step appears in more than one test. They help you avoid copy-paste code and make tests easier to read.

You pass a fixture into a test by adding it as a parameter. Pytest will run the fixture first and give the result to your test automatically.

Good times to use fixtures:

• When several tests need the same API client
• When many tests need a logged-in user
• When test setup is long or repeated in many places
• When you want to keep test functions short and focused

You do not need a fixture for every small test. If something is used only once, plain setup inside the test is often easier to understand.

Creating a Fixture

# conftest.py
from rest_framework.test import APIClient
import pytest

@pytest.fixture
def api_client():
    return APIClient()

Using a Fixture in a Test

from rest_framework import status
import pytest

@pytest.mark.django_db
class TestCollectionAPI:
    def test_if_user_is_anonymous(self, api_client):
        # Act
        response = api_client.post('/collections/', {'name': 'My Collection'})

        # Assert
        assert response.status_code == status.HTTP_401_UNAUTHORIZED

In this example, the api_client fixture creates one APIClient instance for the test. That keeps the test short and lets you reuse the same setup in other tests too.

Fixture with Dependencies

You can also build one fixture on top of another fixture. This is useful when one setup step depends on another. For example, you might want a helper that returns a logged-in API client:

# conftest.py
from rest_framework.test import APIClient
import pytest

@pytest.fixture
def api_client():
    return APIClient()

@pytest.fixture
def create_collection(api_client):
    def _create_collection(name='My Collection'):
        return api_client.post('/collections/', {'name': name})
    return _create_collection

@pytest.fixture
def authenticate(api_client, django_user_model):
    def _authenticate(is_staff=False):
        user = django_user_model.objects.create_user(
            username='admin-user' if is_staff else 'regular-user',
            password='password123',
            is_staff=is_staff,
        )
        api_client.force_authenticate(user=user)
        return api_client
    return _authenticate

from rest_framework import status
import pytest

@pytest.mark.django_db
class TestCollectionAPI:
    def test_if_user_is_anonymous(self, create_collection):
        # Act
        response = create_collection(name='Test Collection')

        # Assert
        assert response.status_code == status.HTTP_401_UNAUTHORIZED

    def test_create_collection_as_admin(self, authenticate, create_collection):
        # Arrange
        authenticate(is_staff=True)

        # Act
        response = create_collection(name='Test Collection')

        # Assert
        assert response.status_code == status.HTTP_201_CREATED
        assert 'id' in response.data
        assert response.data['name'] == 'Test Collection'

This pattern is helpful when one test step needs another. Here, authenticate depends on api_client and django_user_model, so the fixture can create a real user and then log the client in before the test runs.

Testing GET Endpoints with model_bakery

When testing a GET endpoint, you usually need some data in the database first. Instead of creating that data manually each time, you can use model_bakery. It quickly creates model instances with sensible defaults, which makes your tests shorter and easier to maintain.

Docs: https://model-bakery.readthedocs.io/en/latest/

uv add --dev model_bakery

Example: Retrieve a Single Collection

from rest_framework import status
from model_bakery import baker
import pytest
from some_app.models import Collection

@pytest.mark.django_db
class TestCollectionAPI:
    def test_get_collection(self, api_client, authenticate):
        # Arrange
        authenticate(is_staff=True)
        collection = baker.make(Collection)

        # Act
        response = api_client.get(f'/collections/{collection.id}/')

        # Assert
        assert response.status_code == status.HTTP_200_OK
        assert response.data == {
            'id': collection.id,
            'name': collection.name,
        }

Understanding _quantity in baker.make

from rest_framework import status
from model_bakery import baker
import pytest
from some_app.models import Collection, Product

@pytest.mark.django_db
class TestProductAPI:
    def test_get_product(self, api_client, authenticate):
        # Arrange
        authenticate(is_staff=True)
        collection = baker.make(Collection)
        products = baker.make(Product, collection=collection, _quantity=10)
        product = products[0]

        # Act
        response = api_client.get(f'/products/{product.id}/')

        # Assert
        assert response.status_code == status.HTTP_200_OK
        assert response.data == {
            'id': product.id,
            'name': product.name,
            'collection_id': collection.id,
        }

If you do not pass collection in baker.make(Product, _quantity=10), model_bakery may create related Collection objects automatically, which can result in different collections for different products.

If you pass collection=collection, all generated products are linked to the same collection. This is useful when you want to test relationships and filtering behavior with consistent related data.

Performance Testing with Locust

Locust is a powerful, Python-based performance testing tool used to simulate real users interacting with your API. Instead of writing complex test scripts in a DSL (Domain-Specific Language), you define user behavior using normal Python code. This makes it especially useful for backend developers (like you working with Django/DRF), because you already understand Python.

At a high level, Locust helps you answer questions like:

• How many users can my API handle at the same time?
• What happens when traffic suddenly increases?
• Which endpoints are slow or failing under load?
• Where are the bottlenecks (database, network, code logic)?

How Locust Works

Think of Locust as spawning many virtual users, where each user repeatedly performs actions (tasks) such as:

User
 ├── wait (thinking time)
 ├── call API endpoint (/collections/)
 ├── wait
 ├── call API endpoint (/collections/{id}/)
 ├── wait
 └── repeat...

Each user behaves independently, and Locust runs many of them concurrently.

Knowing What to Test

Before writing any performance test, you must decide what matters most in your system.

For a Django REST API, typical important areas are:

1. Read-heavy endpoints

   • Listing collections
   • Viewing a single resource
   • Searching/filtering

2. Write-heavy endpoints

   • Creating collections/products
   • Updating data

3. Authentication endpoints

   • Login
   • Token refresh

4. Complex queries

   • Endpoints involving joins, filtering, pagination

Example Strategy

If your API has:

• /collections/ → list
• /collections/{id}/ → detail
• /collections/ (POST) → create

Then realistic usage looks like:

80% → read (GET)
20% → write (POST)

So your test should reflect that distribution.

Installing Locust

uv add --dev locust

Conventions for Locust in Django

To keep your tests clean, scalable, and easy to debug, follow these:

• one file per scenario

  locust/
      ├── browse_collections.py
      ├── create_collection.py

• one action per task: Each @task should represent one user action only.

• use wait_time: Simulates real user thinking delay:

  wait_time = between(1, 5)

• use on_start: It is a lifecycle hook that runs once when a user starts. Use for user setuo tasks like login or fetching tokens.

  Runs once per user when they start:

  • login
  • setup tokens

• use name for grouping: Helps clean reporting in UI:

  name="/collections/{id}/"

Running Locust

1. Run Locust:

   uv run locust -f locust/browse_collections.py

2. Open browser: http://localhost:8089

3. Fill fields:

   • Users: 50
   • Spawn rate: 5
   • Host: http://localhost:8000

4. Start test

Creating Locust Tasks

Below is a corrected and beginner-friendly version of your code.

# locust/browse_collections.py
from locust import HttpUser, task, between
from random import randint


class BrowseCollectionsUser(HttpUser):
    """
    This user simulates read-heavy behavior:
    - Listing collections
    - Viewing a single collection
    """

    wait_time = between(1, 5)

    @task(3)  # higher weight → more frequent
    def browse_collections(self):
        """
        Simulates user opening collection list page
        """
        self.client.get("/collections/", name="/collections/")

    @task(1)
    def browse_collection(self):
        """
        Simulates user opening a single collection
        """
        collection_id = randint(1, 20)  # adjust to your DB
        self.client.get(
            f"/collections/{collection_id}/",
            name="/collections/{id}/"
        )

# locust/create_collection.py
from locust import HttpUser, task, between
import random
import string


class CreateCollectionUser(HttpUser):
    """
    This user simulates write-heavy behavior:
    - Logging in
    - Creating collections
    """

    wait_time = between(1, 5)

    def on_start(self):
        """
        Runs once when a user starts.
        Used for authentication.
        """

        response = self.client.post(
            "/auth/login/",
            json={
                "username": "admin",
                "password": "password123"
            }
        )

        # Basic safety check
        if response.status_code != 200:
            raise Exception("Login failed")

        data = response.json()
        access_token = data.get("access")

        if not access_token:
            raise Exception("No access token received")

        # Attach token to all future requests
        self.client.headers.update({
            "Authorization": f"Bearer {access_token}"
        })

    @task
    def create_collection(self):
        """
        Simulates user creating a collection
        """

        random_name = "".join(
            random.choices(string.ascii_letters, k=10)
        )

        self.client.post(
            "/collections/",
            json={
                "name": f"Test {random_name}"
            },
            name="/collections/"
        )

Understanding the Code

1. HttpUser

It is the base class for defining a user behavior. Each instance of HttpUser represents a single simulated user that will perform tasks against your API.

HttpUser
   ├── task()
   ├── task()
   └── wait_time

Each user runs tasks in a loop.

2. @task(weight)

We use the @task decorator to define what actions the user performs. The optional weight parameter controls how often a task runs relative to other tasks.

@task(3) → runs 3x more often
@task(1) → runs normally

So:

browse_collections : browse_collection
         3         :        1

3. wait_time

Simulates human delay:

wait_time = between(1, 5)

Without this:

• your API gets unrealistically hammered
• results become misleading

4. on_start()

Lifecycle hook: It runs once when a user starts. Use it for setup tasks like login or fetching tokens. It also used some time to generating dummy necessary data for testing.

User starts
   ↓
on_start() runs once
   ↓
tasks start running

Used for:

• login
• fetching tokens
• setting headers

5. client

It is a wrapper over HTTP requests that Locust provides. It allows you to make requests to your API endpoints as if you were a real user.

self.client.get(...)
self.client.post(...)

It automatically:

• tracks response time
• tracks failures
• reports metrics to UI

6. name parameter

The name parameter in self.client.get() is used to group similar requests together in the Locust UI. This is especially important when your API has dynamic URLs (like /collections/{id}/) that would otherwise be treated as separate endpoints.

self.client.get(f"/collections/{id}/", name="/collections/{id}/")

Without name, Locust treats each ID separately:

/collections/1/
/collections/2/
/collections/3/

With name, it groups:

/collections/{id}/

7. json parameter

When making POST requests, you can use the json parameter to send JSON data in the request body. This is a convenient way to send structured data to your API.

self.client.post(
    "/collections/",
    json={
        "name": "Test Collection"
    }
)

How to Use Locust

1. Run:

uv run locust -f locust/browse_collections.py

2. Open:

http://localhost:8089

3. Fill:

• Users: 10
• Spawn rate: 2
• Host: http://localhost:8000

4. Click Start Swarming

Understanding Metrics

Requests per second (RPS)

Higher = better throughput

Response time percentiles

• 50% → average user
• 95% → slow users
• 99% → worst case

Example:

50% → 120ms
95% → 800ms   ← WARNING

Failures

Any non-2xx response:

500 → server crash
401 → auth issue
400 → bad request

Testing Strategy

1. Start small:

   5 users

2. Verify correctness
3. Increase gradually:

   10 → 20 → 50 → 100 → 1000

4. Watch:
   • response time spikes
   • failure increase

Test in Production?

Testing in production is risky but sometimes necessary. If you do it, follow these guidelines:

• Use a separate test environment if possible to avoid affecting real users
• If testing in production, use a small percentage of real traffic (e.g., 1%) to avoid overwhelming the system
• Monitor closely for any issues
• Communicate with your team and users about the testing to avoid confusion and panic

Django Silk (Profiling Tool)

Django Silk is a profiling tool that tracks every request, showing you why your API is slow by inspecting queries, execution time, and bottlenecks.

Note

Locust vs Silk:

• Locust: Shows which endpoints are slow
• Silk: Shows why they are slow (too many queries, slow queries, inefficient ORM)

Docs: Django Silk GitHub

Installation & Setup

uv add django-silk

1. Add to INSTALLED_APPS:

if DEBUG:
    INSTALLED_APPS += ["silk"]

2. Add Middleware:

if DEBUG:
    MIDDLEWARE += ["silk.middleware.SilkyMiddleware"]

Place Silk middleware towards the end so it observes the full request lifecycle.

3. Add URL Route:

# main urls.py
from django.urls import path, include
from django.conf import settings

if settings.DEBUG:
    urlpatterns += [path("silk/", include("silk.urls", name="silk"))]

4. Run Migrations:

python manage.py migrate

Using Silk

1. Start the server: python manage.py runserver
2. Open dashboard: http://localhost:8000/silk/
3. Generate requests (e.g., with Locust)
4. Analyze requests, queries, and timing in the dashboard

Identifying N+1 Query Problems

Here’s a practical example of detecting N+1 issues with Silk:

Without optimization (N+1 problem):

# Bad: Multiple queries
users = User.objects.all()  # Query 1
for user in users:
    print(user.profile.bio)  # Query N+1 (one query per user)

Silk shows:

Query count: 11 queries (1 for users + 10 for profiles)
Total time: 450ms

With optimization:

# Good: Single query with prefetch_related
users = User.objects.prefetch_related('profile').all()
for user in users:
    print(user.profile.bio)  # No additional queries

Silk shows:

Query count: 2 queries
Total time: 45ms

Tip

Use Silk’s query inspector to see:

• Exact SQL queries executed
• Time spent on each query
• Query count per request

This makes N+1 problems immediately obvious.

Workflow

1. Run your performance test (with Locust)
2. Find slow endpoints in Silk dashboard
3. Inspect queries to identify issues
4. Disable Silk in settings (it adds overhead)
5. Optimize your code (use select_related, prefetch_related, add database indexes)
6. Re-run tests without Silk to measure real improvement

Caution

Always disable Silk before measuring real performance. Silk adds middleware overhead, making results inaccurate. Clear Silk data before each new test run for clean results.

Optimizing ORM Queries

• Use select_related() for ForeignKey relations (single query join)
• Use prefetch_related() for ManyToMany and reverse ForeignKey (cached queries)
• Check the raw SQL with .query to understand what Django generates
• Monitor query count: fewer queries = faster API

Note

Quick Diagnosis:

Slow API → Run Locust → Find slow endpoint → Use Silk →
Inspect queries → Optimize ORM → Test again

Performance Optimisation

Performance optimisation in a Django application is about reducing unnecessary work at every layer:

Request → Django ORM → SQL Query → Database → Response

If any layer is inefficient, your entire API becomes slow. The goal is to minimize database load, memory usage, and response time, especially for endpoints that are hit frequently.

Optimising Django ORM Queries

Django ORM is powerful, but it can generate inefficient SQL if used carelessly.

1. Avoid N+1 Queries

This happens when Django executes one query for the main object and additional queries for related objects.

# BAD (N+1 problem)
products = Product.objects.all()
for p in products:
    print(p.category.name)  # triggers extra query per product

Fix using select_related (for ForeignKey / OneToOne)

products = Product.objects.select_related("category").all()

• Performs a SQL JOIN
• Fetches related object in the same query

---

Fix using prefetch_related (for ManyToMany / reverse FK)

products = Product.objects.prefetch_related("tags").all()

• Executes separate queries
• Combines results in Python
• Efficient for many-to-many relationships

2. Load Only Required Fields

Avoid loading unnecessary data.

Using only

products = Product.objects.only("id", "title")

• Loads only specified fields
• Other fields are deferred automatically

Using defer

products = Product.objects.defer("description")

• Skips large or unused fields
• Opposite of only

---

3. Use values() and values_list()

products = Product.objects.values("id", "title")

Returns:

[
  {"id": 1, "title": "A"},
  {"id": 2, "title": "B"}
]

products = Product.objects.values_list("id", "title")

Returns:

[(1, "A"), (2, "B")]

Why faster?

Model instance → heavy (methods, state, ORM overhead)
Dict/List → lightweight (less memory, faster)

Use these when:

• You don’t need model methods (save, delete)
• You only need raw data

---

4. Efficient Counting

# BAD
len(Product.objects.all())

• Loads all records into memory

# GOOD
Product.objects.count()

• Executes SELECT COUNT(*) in database
• Much faster and memory efficient

5. Bulk Operations

Bulk Create

Product.objects.bulk_create([
    Product(title="A"),
    Product(title="B"),
])

Bulk Update

Product.objects.bulk_update(products, ["title"])

Why important?

Loop create → N database queries
Bulk create → 1 database query

When ORM Is Not Enough

Sometimes Django ORM generates inefficient SQL.

Rewrite Query Using Raw SQL

from django.db import connection

with connection.cursor() as cursor:
    cursor.execute("SELECT ...")
    rows = cursor.fetchall()

Use this when:

• Query is complex
• ORM generates slow SQL
• You understand SQL well

Database Optimisation

If queries are still slow, the issue is often in the database design.

1. Add Indexes

class Product(models.Model):
    title = models.CharField(max_length=255, db_index=True)

• Speeds up filtering/search
• Essential for large tables

---

2. Optimise Schema

• Normalize data properly
• Avoid unnecessary joins
• Use correct field types

---

Caching (Use Carefully)

Caching stores results in memory to avoid repeated database queries.

First request → slow (DB hit)
Next requests → fast (cache hit)

Example Flow

Request → Check cache
         ├── Hit → return data
         └── Miss → query DB → store in cache → return

Important Note

Caching is not always faster:

Cache server (Redis) → network call
Database → local query

If query is simple, DB might be faster than cache.

Use caching for:

• Expensive queries
• Frequently accessed data

Scaling the Application

If optimisation is done but performance still drops under load:

1. Vertical Scaling

Upgrade server:

More CPU
More RAM
Faster disk

2. Horizontal Scaling

Add more servers:

Load Balancer
   ├── Server 1
   ├── Server 2
   └── Server 3

• Handles more concurrent users
• More complex setup
• Higher cost

Practical Strategy

Do not optimise everything.

Focus on:

High traffic endpoints
Critical user paths
Slow queries

Avoid wasting time on:

Admin reports used rarely
Low-impact features

Stress Testing with Locust

Stress testing pushes your API beyond its normal limits to observe how it behaves under extreme conditions. It helps you identify breaking points, bottlenecks, and how your system recovers from failure.

While not strictly required, stress testing is a best practice after completing performance testing and optimization. This allows you to verify that your optimizations hold up under heavy load and identify any remaining weaknesses.

In this section, you will use Locust to simulate many concurrent users accessing your API simultaneously and observe its performance under stress.

By the end, you will understand your API’s breaking points and how it handles high-traffic scenarios.

Important Note

Development server is not designed for production use and may not handle high traffic well. For accurate stress testing, use a production-like environment (e.g., staging server) that closely mimics your real deployment setup.