TIOBE Index News (April 2024): PHP’s Popularity Declining

How to Handle Django Forms within Modal Dialogs

I like django-crispy-forms. You can use it for stylish uniform HTML forms with Bootstrap, TailwindCSS, or even your custom template pack. But when it comes to custom widgets and dynamic form handling, it was always a challenge. Recently I discovered htmx. It's a JavaScript framework that handles Ajax communication based on custom HTML attributes. In this article, I will explore how you can use django-crispy-forms with htmx to provide a form with server-side validation in a modal dialog.

The setup

For this experiment, I will be using these PyPI packages:

• Django - my beloved Python web framework.
• django-crispy-forms - library for stylized forms.
• crispy-bootstrap5 - Bootstrap 5 template pack for django-crispy-forms.
• django-htmx - some handy htmx helpers for Django projects.

Also, I will use the CDN versions of Bootstrap5 and htmx.

The form

I decided to add some crispy style to the login form by extending Django's authentication form and attaching a crispy helper to it.

from django.contrib.auth.forms import AuthenticationForm
from crispy_forms.helper import FormHelper
from crispy_forms.layout import Layout
from crispy_bootstrap5 import bootstrap5


class LoginForm(AuthenticationForm):
    def __init__(self, *args, **kwargs):
        super().__init__(*args, **kwargs)

        self.helper = FormHelper()
        self.helper.form_tag = False
        self.helper.include_media = False
        self.helper.layout = Layout(
            bootstrap5.FloatingField("username", autocomplete="username"),
            bootstrap5.FloatingField("password", autocomplete="current-password"),
        )

Here I set form_tag to False to skip the <form> tag from the rendered form elements because I want to customize it in the templates. And I set include_media to False because I want to manually handle the media inclusion, as you will see later, instead of automatically including them in the form.

The views

I will have two views:

• The home view will have a button to open a modal dialog for login.
• The login form view will handle the form and return the HTML markup for the dialog.

from django.shortcuts import render, redirect
from django.contrib.auth import login as auth_login, logout as auth_logout
from django_htmx.http import HttpResponseClientRefresh
from .forms import LoginForm


def home(request):
    if request.user.is_authenticated:
        return render(request, "dashboard.html")
    return render(request, "home.html", context)


def login(request):
    if request.method == "POST":
        form = LoginForm(request=request, data=request.POST, prefix="login")
        template_name = "login_form.html"
        if form.is_valid():
            user = form.get_user()
            auth_login(request, user)
            return HttpResponseClientRefresh()
    else:
        form = LoginForm(request=request, prefix="login")
        template_name = "login_dialog.html"
    context = {"form": form}
    return render(request, template_name, context)

The home view just returns different templates based on whether the user is logged in or not.

The login view handles a login form and renders different templates based on whether the view was accessed by GET or POST method. If the login succeeds, a special HttpResponseClientRefresh response is returned, which tells htmx to refresh the page from where the login form was loaded and submitted. It's an empty response with the HX-Refresh: true header.

Then I plug those two views into my urls.py rules.

The templates and javascript

At the end of the base.html template, I include htmx from CDN and my custom dialog.js:

<script src="https://unpkg.com/htmx.org@1.8.2" integrity="sha384-+8ISc/waZcRdXCLxVgbsLzay31nCdyZXQxnsUy++HJzJliTzxKWr0m1cIEMyUzQu" crossorigin="anonymous"></script>
<script src="{% static 'js/dialog.js' %}"></script>

In the home.html template, I add a button which will open the dialog:

<button
    data-hx-get="{% url 'login' %}"
    data-hx-target="main"
    data-hx-swap="beforeend"
    type="button"
    class="btn btn-primary"
>Log in</button>

Note that htmx allows either hx-* syntax or data-hx-* for its HTML attributes and I prefer the latter because data-* attributes make a valid HTML document.

In the snippet above, I tell htmx to load the login page on the button click and insert it before the end of the <main> HTML tag.

Let's have a look at my dialog.js file:

function init_widgets_for_htmx_element(target) {
    // init other widgets

    // init modal dialogs
    if (target.tagName === 'DIALOG') {
        target.showModal();
        htmx.on('.close-dialog', 'click', function(event) {
            var dialog = htmx.find('dialog[open]');
            dialog.close();
            htmx.remove(dialog);
        });
    }
}

htmx.onLoad(init_widgets_for_htmx_element);

Here, when a page loads or a htmx inserts a snippet, the init_widgets_for_htmx_element function will be called with the <body> or the inserted element as the target. One can use this function to initialize widgets, such as rich text fields, autocompletes, tabs, custom frontend validators, etc. Also, I use this function to open the loaded modal dialogs and add event handlers to close them.

Now the login_dialog.html template looks like this (I just stripped the styling markup):

<dialog id="htmx-dialog">
    <h1>Login</h1>
    <button type="button" class="close-dialog btn-close" aria-label="Close"></button>
    {% include "login_form.html" %}
    <button type="submit" form="htmx-dialog-form">Log in</button>
</dialog>

And the login_form.html looks like this:

<form
    id="htmx-dialog-form"
    novalidate
    data-hx-post="{% url 'login' %}"
    data-hx-swap="outerHTML"
>
    {% load crispy_forms_tags %}
    {% crispy form %}
</form>

The htmx attributes tell htmx to submit the form data to the login view by Ajax and replace the <form> HTML tag with the response received. That is used for form validation. If the response has the HX-Refresh: true header, as mentioned before, then the home page is refreshed.

Here is what the result looks like:

The form media

If you noticed before, we excluded the media from the form. Otherwise, the home page would load the media of its own forms (for example, a search form) and the media of the dialog forms. And that would cause double executions of shared scripts, for example, the ones for autocompletes and rich text fields.

You can nicely combine media from different forms by concatenating the media instances. This way, each CSS and Javascript file is included just once.

As the login form isn't part of the home page but instead included on demand, I need to have its media in the home view or any other view where the login button is shown.

Here comes this custom context processor for help:

def login_dialog(request):
    from .forms import LoginForm

    if not request.user.is_authenticated:
        form = LoginForm(request=request)
        if hasattr(request, "combined_media"):
            request.combined_media += form.media
        else:
            request.combined_media = form.media
    return {}

It checks for any request.combined_media and attaches the form media from the login form.

Lastly, I attach this context processor to the template settings and render the value of combined media before </body>:

{{ request.combined_media }}

The final words

Get the code to play with from Github. As you can see, htmx makes Ajax communications pretty straightforward, even when it's about dynamically loading and reloading parts of the content or initializing custom widgets.

Cover photo by Pixabay

How to Rename a Django App

When I initially created my MVP (minimal viable product) for 1st things 1st, I considered the whole Django project to be about prioritization. After a few years, I realized that the Django project is about SaaS (software as a service), and prioritization is just a part of all functionalities necessary for a SaaS to function. I ended up needing to rename apps to have clean and better-organized code. Here is how I did that.

0. Get your code and database up to date

Ensure you have the latest git pull and execute all database migrations.

1. Install django-rename-app

Put django-rename-app into pip requirements and install them or just run:

(venv)$ pip install django-rename-app

Put the app into INSTALLED_APPS in your settings:

INSTALLED_APPS = [
    # …
    "django_rename_app",
]

2. Rename the app directories

Rename the oldapp as newapp in your apps and templates.

3. Rename the app name occurrences in the code

Rename the app in all your imports, relations, migrations, and template paths.

You can do a global search for oldapp and then check case by case where you need to rename that term to newapp, and where not.

4. Run the management command rename_app

Run the management command rename_app:

(env)$ python manage.py rename_app oldapp newapp

This command renames the app prefix the app tables and the records in django_content_type and django_migrations tables.

If you plan to update staging or production servers, add the rename_app command before running migrations in your deployment scripts (Ansible, Docker, etc.)

5. Update indexes and constraints

Lastly, create an empty database migration for the app with custom code to update indexes and foreign-key constraints.

(env)$ python manage.py makemigrations newapp --empty --name rename_indexes

Fill the migration with the following code:

# newapp/migrations/0002_rename_indexes.py
from django.db import migrations


def named_tuple_fetch_all(cursor):
    "Return all rows from a cursor as a namedtuple"
    from collections import namedtuple

    desc = cursor.description
    Result = namedtuple("Result", [col[0] for col in desc])
    return [Result(*row) for row in cursor.fetchall()]


def rename_indexes(apps, schema_editor):
    from django.db import connection

    with connection.cursor() as cursor:
        cursor.execute(
            """SELECT indexname FROM pg_indexes 
            WHERE tablename LIKE 'newapp%'"""
        )
        for result in named_tuple_fetch_all(cursor):
            old_index_name = result.indexname
            new_index_name = old_index_name.replace(
                "oldapp_", "newapp_", 1
            )
            cursor.execute(
                f"""ALTER INDEX IF EXISTS {old_index_name} 
                RENAME TO {new_index_name}"""
            )


def rename_foreignkeys(apps, schema_editor):
    from django.db import connection

    with connection.cursor() as cursor:
        cursor.execute(
            """SELECT table_name, constraint_name 
            FROM information_schema.key_column_usage
            WHERE constraint_catalog=CURRENT_CATALOG 
            AND table_name LIKE 'newapp%'
            AND position_in_unique_constraint notnull"""
        )
        for result in named_tuple_fetch_all(cursor):
            table_name = result.table_name
            old_foreignkey_name = result.constraint_name
            new_foreignkey_name = old_foreignkey_name.replace(
                "oldapp_", "newapp_", 1
            )
            cursor.execute(
                f"""ALTER TABLE {table_name} 
                RENAME CONSTRAINT {old_foreignkey_name} 
                TO {new_foreignkey_name}"""
            )


class Migration(migrations.Migration):

    dependencies = [
        ("newapp", "0001_initial"),
    ]

    operations = [
        migrations.RunPython(rename_indexes, migrations.RunPython.noop),
        migrations.RunPython(rename_foreignkeys, migrations.RunPython.noop),
    ]

Run the migrations:

(env)$ python manage.py migrate

If something doesn't work as wanted, migrate back, fix the code, and migrate again. You can unmigrate by migrating to one step before the last migration, for example:

(env)$ python manage.py migrate 0001

6. Cleanup

After applying the migration in all necessary environments, you can clean them up by removing django-rename-app from your pip requirements and deployment scripts.

Final words

It's rarely possible to build a system that meets all your needs from the beginning. Proper systems always require continuous improvement and refactoring. Using a combination of Django migrations and django-rename-app, you can work on your websites in an Agile, clean, and flexible way.

Happy coding!

Cover photo by freestocks.

How I Integrated Zapier into my Django Project

As you might know, I have been developing, providing, and supporting the prioritization tool 1st things 1st. One of the essential features to implement was exporting calculated priorities to other productivity tools. Usually, building an export from one app to another takes 1-2 weeks for me. But this time, I decided to go a better route and use Zapier to export priorities to almost all possible apps in a similar amount of time. Whaaat!?? In this article, I will tell you how.

What is Zapier and how it works?

The no-code tool Zapier takes input from a wide variety of web apps and outputs it to many other apps. Optionally you can filter the input based on conditions. Or format the input differently (for example, convert HTML to Markdown). In addition, you can stack the output actions one after the other. Usually, people use 2-3 steps for their automation, but there are power users who create 50-step workflows.

The input is managed by Zapier's triggers. The output is controlled by Zapier's actions. These can be configured at the website UI or using a command-line tool. I used the UI as this was my first integration. Trigger events accept a JSON feed of objects with unique IDs. Each new item there is treated as a new input item. With a free tier, the triggers are checked every 15 minutes. Multiple triggers are handled in parallel, and the sorting order of execution is not guaranteed. As it is crucial to have the sorting order correct for 1st things 1st priorities, people from Zapier support suggested providing each priority with a 1-minute interval to make sure the priorities get listed in the target app sequentially.

The most challenging part of Zapier integration was setting up OAuth 2.0 provider. Even though I used a third-party Django app django-oauth-toolkit for that. Zapier accepts other authentication options too, but this one is the least demanding for the end-users.

Authentication

OAuth 2.0 allows users of one application to use specific data of another application while keeping private information private. You might have used the OAuth 2.0 client directly or via a wrapper for connecting to Twitter apps. For Zapier, one has to set OAuth 2.0 provider.

The official tutorial for setting up OAuth 2.0 provider with django-oauth-toolkit is a good start. However, one problem with it is that by default, any registered user can create OAuth 2.0 applications at your Django website, where in reality, you need just one global application.

First of all, I wanted to allow OAuth 2.0 application creation only for superusers.

For that, I created a new Django app oauth2_provider_adjustments with modified views and URLs to use instead of the ones from django-oauth-toolkit.

The views related to OAuth 2.0 app creation extended this SuperUserOnlyMixin instead of LoginRequiredMixin:

from django.contrib.auth.mixins import AccessMixin

class SuperUserOnlyMixin(AccessMixin):
    def dispatch(self, request, *args, **kwargs):
        if not request.user.is_superuser:
            return self.handle_no_permission()
        return super().dispatch(request, *args, **kwargs)

Then I replaced the default oauth2_provider URLs:

urlpatterns = [
    # …
    path("o/", include("oauth2_provider.urls", namespace="oauth2_provider")),
]

with my custom ones:

urlpatterns = [
    # …
    path("o/", include("oauth2_provider_adjustments.urls", namespace="oauth2_provider")),
]

I set the new OAuth 2.0 application by going to /o/applications/register/ and filling in this info:

Name: Zapier
Client type: Confidential
Authorization grant type: Authorization code
Redirect uris: https://zapier.com/dashboard/auth/oauth/return/1stThings1stCLIAPI/ (copied from Zapier)
Algorithm: No OIDC support

If you have some expertise in the setup choices and see any flaws, let me know.

Zapier requires creating a test view that will return anything to check if there are no errors authenticating a user with OAuth 2.0. So I made a simple JSON view like this:

from django.http.response import JsonResponse


def user_info(request, *args, **kwargs):
    if not request.user.is_authenticated:
        return JsonResponse(
            {
                "error": "User not authenticated",
            },
            status=200,
        )
    return JsonResponse(
        {
            "first_name": request.user.first_name,
            "last_name": request.user.last_name,
        },
        status=200,
    )

Also, I had to have login and registration views for those cases when the user's session was not present.

Lastly, at Zapier, I had to set these values for OAuth 2.0:

Client ID: The Client ID from registered app
Client Secret: The Client Secret from registered app

Authorization URL: https://apps.1st-things-1st.com/o/authorize/
Scope: read write
Access Token Request: https://apps.1st-things-1st.com/o/token/
Refresh Token Request: https://apps.1st-things-1st.com/o/token/
I want to automatically refresh on unauthorized error: Checked
Test: https://apps.1st-things-1st.com/user-info/
Connection Label: {{first_name}} {{last_name}}

Trigger implementation

There are two types of triggers in Zapier:

• (A) Ones for providing new things to other apps, for example, sending priorities from 1st things 1st to other productivity apps.
• (B) Ones for listing things in drop boxes at the former triggers, for example, letting Zapier users choose the 1st things 1st project from which to import priorities.

The feeds for triggers should (ideally) be paginated. But without meta information for the item count, page number, following page URL, etc., you would usually have with django-rest-framework or other REST frameworks. Provide only an array of objects with unique IDs for each page. The only field name that matters is "id" – others can be anything. Here is an example:

[
    {
        "id": "39T7NsgQarYf",
        "project": "5xPrQbPZNvJv",
        "title": "01. Custom landing pages for several project types (83%)",
        "plain_title": "Custom landing pages for several project types",
        "description": "",
        "score": 83,
        "priority": 1,
        "category": "Choose"
    },
    {
        "id": "4wBSgq3spS49",
        "project": "5xPrQbPZNvJv",
        "title": "02. Zapier integration (79%)",
        "plain_title": "Zapier integration",
        "description": "",
        "score": 79,
        "priority": 2,
        "category": "Choose"
    },
    {
        "id": "6WvwwB7QAnVS",
        "project": "5xPrQbPZNvJv",
        "title": "03. Electron.js desktop app for several project types (42%)",
        "plain_title": "Electron.js desktop app for several project types",
        "description": "",
        "score": 41,
        "priority": 3,
        "category": "Consider"
    }
]

The feeds should list items in reverse order for the (A) type of triggers: the newest things go at the beginning. The pagination is only used to cut the number of items: the second and further pages of the paginated list are ignored by Zapier.

In my specific case of priorities, the order matters, and no items should be lost in the void. So I listed the priorities sequentially (not newest first) and set the number of items per page unrealistically high so that you basically get all the things on the first page of the feed.

The feeds for the triggers of (B) type are normally paginated from the first page until the page returns empty results. The order should be alphabetical, chronological, or by sorting order field, whatever makes sense. There you need just two fields, the ID and the title of the item (but more fields are allowed too), for example:

[
    {
        "id": "5xPrQbPZNvJv",
        "title": "1st things 1st",
        "owner": "Aidas Bendoraitis"
    },
    {
        "id": "VEXGzThxL6Sr",
        "title": "Make Impact",
        "owner": "Aidas Bendoraitis"
    },
    {
        "id": "WoqQbuhdUHGF",
        "title": "DjangoTricks website",
        "owner": "Aidas Bendoraitis"
    },
]

I used django-rest-framework to implement the API because of the batteries included, such as browsable API, permissions, serialization, pagination, etc.

For the specific Zapier requirements, I had to write a custom pagination class, SimplePagination, to use with my API lists. It did two things: omitted the meta section and showed an empty list instead of a 404 error for pages that didn't have any results:

from django.core.paginator import InvalidPage

from rest_framework.pagination import PageNumberPagination
from rest_framework.response import Response


class SimplePagination(PageNumberPagination):
    page_size = 20

    def get_paginated_response(self, data):
        return Response(data)  # <-- Simple pagination without meta

    def get_paginated_response_schema(self, schema):
        return schema  # <-- Simple pagination without meta

    def paginate_queryset(self, queryset, request, view=None):
        """
        Paginate a queryset if required, either returning a
        page object, or `None` if pagination is not configured for this view.
        """
        page_size = self.get_page_size(request)
        if not page_size:
            return None

        paginator = self.django_paginator_class(queryset, page_size)
        page_number = self.get_page_number(request, paginator)

        try:
            self.page = paginator.page(page_number)
        except InvalidPage as exc:
            msg = self.invalid_page_message.format(
                page_number=page_number, message=str(exc)
            )
            return []  # <-- If no items found, don't raise NotFound error

        if paginator.num_pages > 1 and self.template is not None:
            # The browsable API should display pagination controls.
            self.display_page_controls = True

        self.request = request
        return list(self.page)

To preserve the order of items, I had to make the priorities appear one by one at 1-minute intervals. I did that by having a Boolean field exported_to_zapier at the priorities. The API showed priorities only if that field was set to True, which wasn't the case by default. Then, background tasks were scheduled 1 minute after each other, triggered by a button click at 1st things 1st, which set the exported_to_zapier to True for each next priority. I was using huey, but the same can be achieved with Celery, cron jobs, or other background task manager:

# zapier_api/tasks.py
from django.conf import settings
from django.utils.translation import gettext
from huey.contrib.djhuey import db_task


@db_task()
def export_next_initiative_to_zapier(project_id):
    from evaluations.models import Initiative

    next_initiatives = Initiative.objects.filter(
        project__pk=project_id,
        exported_to_zapier=False,
    ).order_by("-total_weight", "order")
    count = next_initiatives.count()
    if count > 0:
        next_initiative = next_initiatives.first()
        next_initiative.exported_to_zapier = True
        next_initiative.save(update_fields=["exported_to_zapier"])

        if count > 1:
            result = export_next_initiative_to_zapier.schedule(
                kwargs={"project_id": project_id},
                delay=settings.ZAPIER_EXPORT_DELAY,
            )
            result(blocking=False)

One gotcha: Zapier starts pagination from 0, whereas django-rest-framework starts pagination from 1. To make them work together, I had to modify the API request (written in JavaScript) at Zapier trigger configuration:

const options = {
  url: 'https://apps.1st-things-1st.com/api/v1/projects/',
  method: 'GET',
  headers: {
    'Accept': 'application/json',
    'Authorization': `Bearer ${bundle.authData.access_token}`
  },
  params: {
    'page': bundle.meta.page + 1  // <-- The custom line for pagination
  }
}

return z.request(options)
  .then((response) => {
    response.throwForStatus();
    const results = response.json;

    // You can do any parsing you need for results here before returning them

    return results;
  });

Final Words

For the v1 of Zapier integration, I didn't need any Zapier actions, so they are yet something to explore, experiment with, and learn about. But the Zapier triggers seem already pretty helpful and a big win compared to individual exports without this tool.

If you want to try the result, do this:

• Create an account and a project at 1st things 1st
• Prioritize something
• Head to Zapier integrations and connect your prioritization project to a project of your favorite to-do list or project management app
• Then click on "Export via Zapier" at 1st things 1st.

Cover photo by Anna Nekrashevich

Generic Functionality without Generic Relations

When you have some generic functionality like anything commentable, likable, or upvotable, it’s common to use Generic Relations in Django. The problem with Generic Relations is that they create the relationships at the application level instead of the database level, and that requires a lot of database queries if you want to aggregate content that shares the generic functionality. There is another way that I will show you in this article.

I learned this technique at my first job in 2002 and then rediscovered it again with Django a few years ago. The trick is to have a generic Item model where every other autonomous model has a one-to-one relation to the Item. Moreover, the Item model has an item_type field, allowing you to recognize the backward one-to-one relationship.

Then whenever you need to have some generic categories, you link them to the Item. Whenever you create generic functionality like media gallery, comments, likes, or upvotes, you attach them to the Item. Whenever you need to work with permissions, publishing status, or workflows, you deal with the Item. Whenever you need to create a global search or trash bin, you work with the Item instances.

Let’s have a look at some code.

Items

First, I'll create the items app with two models: the previously mentioned Item and the abstract model ItemBase with the one-to-one relation for various models to inherit:

# items/models.py
import sys

from django.db import models
from django.apps import apps

if "makemigrations" in sys.argv:
    from django.utils.translation import gettext_noop as _
else:
    from django.utils.translation import gettext_lazy as _


class Item(models.Model):
    """
    A generic model for all autonomous models to link to.
    
    Currently these autonomous models are available:
    - content.Post
    - companies.Company
    - accounts.User
    """
    ITEM_TYPE_CHOICES = (
        ("content.Post", _("Post")),
        ("companies.Company", _("Company")),
        ("accounts.User", _("User")),
    )
    item_type = models.CharField(
        max_length=200, choices=ITEM_TYPE_CHOICES, editable=False, db_index=True
    )

    class Meta:
        verbose_name = _("Item")
        verbose_name_plural = _("Items")

    def __str__(self):
        content_object_title = (
            str(self.content_object) if self.content_object else "BROKEN REFERENCE"
        )
        return (
            f"{content_object_title} ({self.get_item_type_display()})"
        )

    @property
    def content_object(self):
        app_label, model_name = self.item_type.split(".")
        model = apps.get_model(app_label, model_name)
        return model.objects.filter(item=self).first()


class ItemBase(models.Model):
    """
    An abstract model for the autonomous models that will link to the Item.
    """
    item = models.OneToOneField(
        Item,
        verbose_name=_("Item"),
        editable=False,
        blank=True,
        null=True,
        on_delete=models.CASCADE,
        related_name="%(app_label)s_%(class)s",
    )

    class Meta:
        abstract = True

    def save(self, *args, **kwargs):
        if not self.item:
            model = type(self)
            item = Item.objects.create(
                item_type=f"{model._meta.app_label}.{model.__name__}"
            )
            self.item = item
        super().save()

    def delete(self, *args, **kwargs):
        if self.item:
            self.item.delete()
        super().delete(*args, **kwargs)

Then let's create some autonomous models that will have one-to-one relations with the Item. By "autonomous models," I mean those which are enough by themselves, such as posts, companies, or accounts. Models like types, categories, tags, or likes, wouldn't be autonomous.

Posts

Second, I create the content app with the Post model. This model extends ItemBase which will create the one-to-one relation on save, and will define the item_type as content.Post:

# content/models.py
import sys

from django.contrib.auth.base_user import BaseUserManager
from django.db import models
from django.contrib.auth.models import AbstractUser

if "makemigrations" in sys.argv:
    from django.utils.translation import gettext_noop as _
else:
    from django.utils.translation import gettext_lazy as _

from items.models import ItemBase


class Post(ItemBase):
    title = models.CharField(_("Title"), max_length=255)
    slug = models.SlugField(_("Slug"), max_length=255)
    content = models.TextField(_("Content"))

    class Meta:
        verbose_name = _("Post")
        verbose_name_plural = _("Posts")

Companies

Third, I create the companies app with the Company model. This model also extends ItemBase which will create the one-to-one relation on save, and will define the item_type as companies.Company:

# companies/models.py
import sys

from django.contrib.auth.base_user import BaseUserManager
from django.db import models
from django.contrib.auth.models import AbstractUser

if "makemigrations" in sys.argv:
    from django.utils.translation import gettext_noop as _
else:
    from django.utils.translation import gettext_lazy as _

from items.models import ItemBase


class Company(ItemBase):
    name = models.CharField(_("Name"), max_length=255)
    slug = models.SlugField(_("Slug"), max_length=255)
    description = models.TextField(_("Description"))

    class Meta:
        verbose_name = _("Company")
        verbose_name_plural = _("Companies")

Accounts

Fourth, I'll have a more extensive example with the accounts app containing the User model. This model extends AbstractUser from django.contrib.auth as well as ItemBase for the one-to-one relation. The item_type set at the Item model will be accounts.User:

# accounts/models.py
import sys

from django.db import models
from django.contrib.auth.base_user import BaseUserManager
from django.contrib.auth.models import AbstractUser

if "makemigrations" in sys.argv:
    from django.utils.translation import gettext_noop as _
else:
    from django.utils.translation import gettext_lazy as _

from items.models import ItemBase


class UserManager(BaseUserManager):
    def create_user(self, username="", email="", password="", **extra_fields):
        if not email:
            raise ValueError("Enter an email address")
        email = self.normalize_email(email)
        user = self.model(username=username, email=email, **extra_fields)
        user.set_password(password)
        user.save(using=self._db)
        return user

    def create_superuser(self, username="", email="", password=""):
        user = self.create_user(email=email, password=password, username=username)
        user.is_superuser = True
        user.is_staff = True
        user.save(using=self._db)
        return user


class User(AbstractUser, ItemBase):
    # change username to non-editable non-required field
    username = models.CharField(
        _("Username"), max_length=150, editable=False, blank=True
    )
    # change email to unique and required field
    email = models.EmailField(_("Email address"), unique=True)
    bio = models.TextField(_("Bio"))

    USERNAME_FIELD = "email"
    REQUIRED_FIELDS = []

    objects = UserManager()

Creating new items

I will use the Django shell to create several autonomous model instances and the related Items too:

>>> from content.models import Post
>>> from companies.models import Company
>>> from accounts.models import User
>>> from items.models import Item
>>> post = Post.objects.create(
...     title="Hello, World!",
...     slug="hello-world",
...     content="Lorem ipsum…",
... )
>>> company = Company.objects.create(
...     name="Aidas & Co",
...     slug="aidas-co",
...     description="Lorem ipsum…",
... )
>>> user = User.objects.create_user(
...     username="aidas",
...     email="aidas@example.com",
...     password="jdf234oha&6sfhasdfh",
... )
>>> Item.objects.count()
3

Aggregating content from all those relations

Lastly, here is an example of having posts, companies, and users in a single view. For that, we will use the Item queryset with annotations:

from django import forms
from django.db import models
from django.shortcuts import render
from django.utils.translation import gettext, gettext_lazy as _

from .models import Item


class SearchForm(forms.Form):
    q = forms.CharField(label=_("Search"), required=False)
    

def all_items(request):
    qs = Item.objects.annotate(
        title=models.Case(
            models.When(
                item_type="content.Post", 
                then="content_post__title",
            ),
            models.When(
                item_type="companies.Company", 
                then="companies_company__name",
            ),
            models.When(
                item_type="accounts.User",
                then="accounts_user__email",
            ),
            default=models.Value(gettext("<Untitled>")),
        ),
        description=models.Case(
            models.When(
                item_type="content.Post",
                then="content_post__content",
            ),
            models.When(
                item_type="companies.Company",
                then="companies_company__description",
            ),
            models.When(
                item_type="accounts.User", 
                then="accounts_user__bio",
                ),
            default=models.Value(""),
        ),
    )
    
    form = SearchForm(data=request.GET, prefix="search")
    if form.is_valid():
        query = form.cleaned_data["q"]
        if query:
            qs = qs.annotate(
                search=SearchVector(
                    "title",
                    "description",
                )
            ).filter(search=query)

    context = {
        "queryset": qs,
        "search_form": form,
    }
    return render(request, "items/all_items.html", context)

Final words

You can have generic functionality and still avoid multiple hits to the database by using the Item one-to-one approach instead of generic relations.

The name of the Item model can be different, and you can even have multiple such models for various purposes, for example, TaggedItem for tags only.

Do you use anything similar in your projects?

Do you see how this approach could be improved?

Let me know in the comments!

Cover picture by Pixabay

17 Django Project Ideas that can Make a Positive Impact around You

For more than a decade, I was focused only on the technical part of website building with Django. In the process, I have built a bunch of interesting cultural websites. But I always felt that those sleepless nights were not worthy of the impact.

They say, "Don’t work hard, work smart!" I agree with that phrase, and for me it's not about working less hours. For me, it's working as much as necessary, but on things that matter most.

So after years of collecting facts about life, I connected the dots and came up with make-impact.org – a social donation platform, which became one of the most important long-term projects. All my planning goes around this project.

And I believe I am not the only programmer who sometimes feels that they want to make a positive impact with their skills. So I brainstormed 17 Django project ideas. You can choose one and realize it as a hobby project, open-source platform, startup, or non-profit organization; alone, with a team of developers, or collaborating with some non-technical people.

Idea #1: Low Qualification Job Search

The job market is pretty competitive, and not all people can keep up with the train. You could build a job search website for jobs that don't require high education or lots of working experience. It could be helpful for people with language barriers, harsh living conditions, or those who are very young or very old. You could build it for your city, region, or country.

Idea #2: Discounted Meals and Products

Get inspired from Too Good To Go and build a progressive web app for your city about discounted restaurant meals and shop products whose expiration date is close to the end, but they are still good to eat.

Idea #3: Personal Health Advisor and Tracker

Build a website for setting your personal health improvement goals and tracking the progress. For example, maybe one wants to start eating more particular vegetables every week, jogging daily, lose or gain weight, or get rid of unhealthy addictions. Let people choose their health goals and check in with each progressive step. Allow using the website anonymously.

Idea #4: Online Primary and Elementary School Materials

Some people don't have access to schools in general or miss some classes because of illnesses. You could build a global and open wiki-based primary and elementary school education website for children and adults. It should be translatable and localizable. It would also be interesting to compare the same subject teachings in different countries side-by-side.

Idea #5: Psychological Support for Women

You could build a website with a video chat providing psychological support to discriminated or violently abused women. The help could be given by professionals or emphatic volunteers. The technical part can be implemented using django-channels, WebSockets, and WebRTC.

Idea #6: Rain-harvesting Companies around the World

Rain harvesting is one of the available ways to solve the problem of the lack of drinking water. There could be a platform comparing rain-harvesting companies all around the world. What are the installation prices? What are the countries they are working with? How many people have they saved? This website would allow people to find the most optimal company to build a rain harvesting system for them.

Idea #7: Closest Electric Car Charging Stations

Use the Open Charge Map API and create a progressive web app that shows the nearest electric car charging station and how to get there.

Idea #8: Escrow-based Remote Job Search

As remote jobs are getting more and more popular, there is still a matter of trust between the employees and employers. "Will the job taker complete their job in a good quality?" "Will the company pay the employee on time?" There are Escrow services to fix this issue. These are third parties that take and hold the money until the job is done. You could build a remote job search website promoting the usage of Escrow.com or another escrow service provider.

Idea #9: Open Work Locations

You could build a website listing coworking spaces and cafes with free wifi in your city. It should include the map, price ranges, details if registration is required, and other information necessary for remote workers.

Idea #10: Most Admired Companies

There could be a social website listing the most admired companies to work for in your country. Companies could be rated by working conditions, salary equality, growth opportunities, work relations, and other criteria. Anyone could suggest such a company, and they would be rated by their current and former employees anonymously.

Idea #11: Tiny Houses

The cost of accommodation is a critical problem in many locations of the world. You could develop a website that lists examples of tiny houses and their building schemas and instructions.

Idea #12: Catalog of Recycled Products

You could work on a product catalog with links to online shops, selling things produced from collected plastic. For example, these sunglasses are made of plastic collected from the ocean. Where available, you could use affiliate marketing links.

Idea #13: Information for Climate-change Migrants

You could work on a website for climate-change migrants with information about getting registered, housing, education, and jobs in a new city or country with better climate conditions.

Idea #14: Fishes, Fishing, and Overfishing

Scrape parts of FishBase and create a website about fishes, fishing, and overfishing in your region or the world. Engage people about the marine world and inform them about the damage done by overfishing.

Idea #15: Plant Trees

Create an E-commerce shop or Software as a Service and integrate RaaS (Reforestation as a Service). Let a tree be planted for every sale.

Idea #16: Positive Parenting

Create a progressive web app about positive parenting. For inspiration and information check this article.

Idea #17: Constructive Forum

Create a forum with topic voting and automatic hate speech detection and flagging. For example, maybe you could use a combination of Sentiment analysis from text-processing.com and usage of profanity words to find negativity in forum posts.

It's your turn

I hope this post inspired you. If you decided to start a startup with one of those ideas, don't forget to do your research at first. What are the competitors in your area? What would be your unique selling point? Etc.

Also, it would be interesting to hear your thoughts. Which of the projects would seem to you the most crucial? Which of them would you like to work on?

Cover photo by Joshua Fuller

What You Should Know About The Django User Model

The goal of this article is to discuss the caveats of the default Django user model implementation and also to give you some advice on how to address them. It is important to know the limitations of the current implementation so to avoid the most common pitfalls.

Something to keep in mind is that the Django user model is heavily based on its initial implementation that is at least 16 years old. Because user and authentication is a core part of the majority of the web applications using Django, most of its quirks persisted on the subsequent releases so to maintain backward compatibility.

The good news is that Django offers many ways to override and customize its default implementation so to fit your application needs. But some of those changes must be done right at the beginning of the project, otherwise it would be too much of a hassle to change the database structure after your application is in production.

Below, the topics that we are going to cover in this article:

• User Model Limitations
  • The username field is case-sensitive
  • The username field validates against unicode letters
  • The email field is not unique
  • The email field is not mandatory
  • A user without password cannot initiate a password reset
  • Swapping the default user model is very difficult after you created the initial migrations
• Detailed Solutions
  • Workarounds
    • Making username field case-insensitive
    • Fixing the username validation to use accept ASCII letters only
    • Fixing the email uniqueness and making it mandatory
  • Replacing the default User model
    • Handling case-insensitive without PostgreSQL
• Conclusions

User Model Limitations

First, let’s explore the caveats and next we discuss the options.

The username field is case-sensitive

Even though the username field is marked as unique, by default it is not case-sensitive. That means the username john.doe and John.doe identifies two different users in your application.

This can be a security issue if your application has social aspects that builds around the username providing a public URL to a profile like Twitter, Instagram or GitHub for example.

It also delivers a poor user experience because people doesn’t expect that john.doe is a different username than John.Doe, and if the user does not type the username exactly in the same way when they created their account, they might be unable to log in to your application.

Possible Solutions:

• If you are using PostgreSQL, you can replace the username CharField with the CICharField instead (which is case-insensitive)
• You can override the method get_by_natural_key from the UserManager to query the database using iexact
• Create a custom authentication backend based on the ModelBackend implementation

The username field validates against unicode letters

This is not necessarily an issue, but it is important for you to understand what that means and what are the effects.

By default the username field accepts letters, numbers and the characters: @, ., +, -, and _.

The catch here is on which letters it accepts.

For example, joão would be a valid username. Similarly, Джон or 約翰 would also be a valid username.

Django ships with two username validators: ASCIIUsernameValidator and UnicodeUsernameValidator. If the intended behavior is to only accept letters from A-Z, you may want to switch the username validator to use ASCII letters only by using the ASCIIUsernameValidator.

Possible Solutions:

• Replace the default user model and change the username validator to ASCIIUsernameValidator
• If you can’t replace the default user model, you can change the validator on the form you use to create/update the user

The email field is not unique

Multiple users can have the same email address associated with their account.

By default the email is used to recover a password. If there is more than one user with the same email address, the password reset will be initiated for all accounts and the user will receive an email for each active account.

It also may not be an issue but this will certainly make it impossible to offer the option to authenticate the user using the email address (like those sites that allow you to login with username or email address).

Possible Solutions:

• Replace the default user model using the AbstractBaseUser to define the email field from scratch
• If you can’t replace the user model, enforce the validation on the forms used to create/update

The email field is not mandatory

By default the email field does not allow null, however it allow blank values, so it pretty much allows users to not inform a email address.

Also, this may not be an issue for your application. But if you intend to allow users to log in with email it may be a good idea to enforce the registration of this field.

When using the built-in resources like user creation forms or when using model forms you need to pay attention to this detail if the desired behavior is to always have the user email.

Possible Solutions:

• Replace the default user model using the AbstractBaseUser to define the email field from scratch
• If you can’t replace the user model, enforce the validation on the forms used to create/update

A user without password cannot initiate a password reset

There is a small catch on the user creation process that if the set_password method is called passing None as a parameter, it will produce an unusable password. And that also means that the user will be unable to start a password reset to set the first password.

You can end up in that situation if you are using social networks like Facebook or Twitter to allow the user to create an account on your website.

Another way of ending up in this situation is simply by creating a user using the User.objects.create_user() or User.objects.create_superuser() without providing an initial password.

Possible Solutions:

• If in you user creation flow you allow users to get started without setting a password, remember to pass a random (and lengthy) initial password so the user can later on go through the password reset flow and set an initial password.

Swapping the default user model is very difficult after you created the initial migrations

Changing the user model is something you want to do early on. After your database schema is generated and your database is populated it will be very tricky to swap the user model.

The reason why is that you are likely going to have some foreign key created referencing the user table, also Django internal tables will create hard references to the user table. And if you plan to change that later on you will need to change and migrate the database by yourself.

Possible Solutions:

• Whenever you are starting a new Django project, always swap the default user model. Even if the default implementation fit all your needs. You can simply extend the AbstractUser and change a single configuration on the settings module. This will give you a tremendous freedom and it will make things way easier in the future should the requirements change.

Detailed Solutions

To address the limitations we discussed in this article we have two options: (1) implement workarounds to fix the behavior of the default user model; (2) replace the default user model altogether and fix the issues for good.

What is going to dictate what approach you need to use is in what stage your project currently is.

• If you have an existing project running in production that is using the default django.contrib.auth.models.User, go with the first solution implementing the workarounds;
• If you are just starting your Django, start with the right foot and go with the solution number 2.

Workarounds

First let’s have a look on a few workarounds that you can implement if you project is already in production. Keep in mind that those solutions assume that you don’t have direct access to the User model, that is, you are currently using the default User model importing it from django.contrib.auth.models.

If you did replace the User model, then jump to the next section to get better tips on how to fix the issues.

Making username field case-insensitive

Before making any changes you need to make sure you don’t have conflicting usernames on your database. For example, if you have a User with the username maria and another with the username Maria you have to plan a data migration first. It is difficult to tell you what to do because it really depends on how you want to handle it. One option is to append some digits after the username, but that can disturb the user experience.

Now let’s say you checked your database and there are no conflicting usernames and you are good to go.

First thing you need to do is to protect your sign up forms to not allow conflicting usernames to create accounts.

Then on your user creation form, used to sign up, you could validate the username like this:

def clean_username(self):
    username = self.cleaned_data.get("username")
    if User.objects.filter(username__iexact=username).exists():
        self.add_error("username", "A user with this username already exists.")
    return username

If you are handling user creation in a rest API using DRF, you can do something similar in your serializer:

def validate_username(self, value):
    if User.objects.filter(username__iexact=value).exists():
        raise serializers.ValidationError("A user with this username already exists.")
    return value

In the previous example the mentioned ValidationError is the one defined in the DRF.

The iexact notation on the queryset parameter will query the database ignoring the case.

Now that the user creation is sanitized we can proceed to define a custom authentication backend.

Create a module named backends.py anywhere in your project and add the following snippet:

backends.py

from django.contrib.auth import get_user_model
from django.contrib.auth.backends import ModelBackend


class CaseInsensitiveModelBackend(ModelBackend):
    def authenticate(self, request, username=None, password=None, **kwargs):
        UserModel = get_user_model()
        if username is None:
            username = kwargs.get(UserModel.USERNAME_FIELD)
        try:
            case_insensitive_username_field = '{}__iexact'.format(UserModel.USERNAME_FIELD)
            user = UserModel._default_manager.get(**{case_insensitive_username_field: username})
        except UserModel.DoesNotExist:
            # Run the default password hasher once to reduce the timing
            # difference between an existing and a non-existing user (#20760).
            UserModel().set_password(password)
        else:
            if user.check_password(password) and self.user_can_authenticate(user):
                return user

Now switch the authentication backend in the settings.py module:

settings.py

AUTHENTICATION_BACKENDS = ('mysite.core.backends.CaseInsensitiveModelBackend', )

Please note that 'mysite.core.backends.CaseInsensitiveModelBackend' must be changed to the valid path, where you created the backends.py module.

It is important to have handled all conflicting users before changing the authentication backend because otherwise it could raise a 500 exception MultipleObjectsReturned.

Fixing the username validation to use accept ASCII letters only

Here we can borrow the built-in UsernameField and customize it to append the ASCIIUsernameValidator to the list of validators:

from django.contrib.auth.forms import UsernameField
from django.contrib.auth.validators import ASCIIUsernameValidator

class ASCIIUsernameField(UsernameField):
    def __init__(self, *args, **kwargs):
        super().__init__(*args, **kwargs)
        self.validators.append(ASCIIUsernameValidator())

Then on the Meta of your User creation form you can replace the form field class:

class UserCreationForm(forms.ModelForm):
    # field definitions...

    class Meta:
        model = User
        fields = ("username",)
        field_classes = {'username': ASCIIUsernameField}

Fixing the email uniqueness and making it mandatory

Here all you can do is to sanitize and handle the user input in all views where you user can modify its email address.

You have to include the email field on your sign up form/serializer as well.

Then just make it mandatory like this:

class UserCreationForm(forms.ModelForm):
    email = forms.EmailField(required=True)
    # other field definitions...

    class Meta:
        model = User
        fields = ("username",)
        field_classes = {'username': ASCIIUsernameField}

    def clean_email(self):
        email = self.cleaned_data.get("email")
        if User.objects.filter(email__iexact=email).exists():
            self.add_error("email", _("A user with this email already exists."))
        return email

You can also check a complete and detailed example of this form on the project shared together with this post: userworkarounds

Replacing the default User model

Now I’m going to show you how I usually like to extend and replace the default User model. It is a little bit verbose but that is the strategy that will allow you to access all the inner parts of the User model and make it better.

To replace the User model you have two options: extending the AbstractBaseUser or extending the AbstractUser.

To illustrate what that means I draw the following diagram of how the default Django model is implemented:

The green circle identified with the label User is actually the one you import from django.contrib.auth.models and that is the implementation that we discussed in this article.

If you look at the source code, its implementation looks like this:

class User(AbstractUser):
    class Meta(AbstractUser.Meta):
        swappable = 'AUTH_USER_MODEL'

So basically it is just an implementation of the AbstractUser. Meaning all the fields and logic are implemented in the abstract class.

It is done that way so we can easily extend the User model by creating a sub-class of the AbstractUser and add other features and fields you like.

But there is a limitation that you can’t override an existing model field. For example, you can re-define the email field to make it mandatory or to change its length.

So extending the AbstractUser class is only useful when you want to modify its methods, add more fields or swap the objects manager.

If you want to remove a field or change how the field is defined, you have to extend the user model from the AbstractBaseUser.

The best strategy to have full control over the user model is creating a new concrete class from the PermissionsMixin and the AbstractBaseUser.

Note that the PermissionsMixin is only necessary if you intend to use the Django admin or the built-in permissions framework. If you are not planning to use it you can leave it out. And in the future if things change you can add the mixin and migrate the model and you are ready to go.

So the implementation strategy looks like this:

Now I’m going to show you my go-to implementation. I always use PostgreSQL which, in my opinion, is the best database to use with Django. At least it is the one with most support and features anyway. So I’m going to show an approach that use the PostgreSQL’s CITextExtension. Then I will show some options if you are using other database engines.

For this implementation I always create an app named accounts:

django-admin startapp accounts

Then before adding any code I like to create an empty migration to install the PostgreSQL extensions that we are going to use:

python manage.py makemigrations accounts --empty --name="postgres_extensions"

Inside the migrations directory of the accounts app you will find an empty migration called 0001_postgres_extensions.py.

Modify the file to include the extension installation:

migrations/0001_postgres_extensions.py

from django.contrib.postgres.operations import CITextExtension
from django.db import migrations

class Migration(migrations.Migration):

    dependencies = [
    ]

    operations = [
        CITextExtension()
    ]

Now let’s implement our model. Open the models.py file inside the accounts app.

I always grab the initial code directly from Django’s source on GitHub, copying the AbstractUser implementation, and modify it accordingly:

accounts/models.py

from django.contrib.auth.base_user import AbstractBaseUser
from django.contrib.auth.models import PermissionsMixin, UserManager
from django.contrib.auth.validators import ASCIIUsernameValidator
from django.contrib.postgres.fields import CICharField, CIEmailField
from django.core.mail import send_mail
from django.db import models
from django.utils import timezone
from django.utils.translation import gettext_lazy as _


class CustomUser(AbstractBaseUser, PermissionsMixin):
    username_validator = ASCIIUsernameValidator()

    username = CICharField(
        _("username"),
        max_length=150,
        unique=True,
        help_text=_("Required. 150 characters or fewer. Letters, digits and @/./+/-/_ only."),
        validators=[username_validator],
        error_messages={
            "unique": _("A user with that username already exists."),
        },
    )
    first_name = models.CharField(_("first name"), max_length=150, blank=True)
    last_name = models.CharField(_("last name"), max_length=150, blank=True)
    email = CIEmailField(
        _("email address"),
        unique=True,
        error_messages={
            "unique": _("A user with that email address already exists."),
        },
    )
    is_staff = models.BooleanField(
        _("staff status"),
        default=False,
        help_text=_("Designates whether the user can log into this admin site."),
    )
    is_active = models.BooleanField(
        _("active"),
        default=True,
        help_text=_(
            "Designates whether this user should be treated as active. Unselect this instead of deleting accounts."
        ),
    )
    date_joined = models.DateTimeField(_("date joined"), default=timezone.now)

    objects = UserManager()

    EMAIL_FIELD = "email"
    USERNAME_FIELD = "username"
    REQUIRED_FIELDS = ["email"]

    class Meta:
        verbose_name = _("user")
        verbose_name_plural = _("users")

    def clean(self):
        super().clean()
        self.email = self.__class__.objects.normalize_email(self.email)

    def get_full_name(self):
        """
        Return the first_name plus the last_name, with a space in between.
        """
        full_name = "%s %s" % (self.first_name, self.last_name)
        return full_name.strip()

    def get_short_name(self):
        """Return the short name for the user."""
        return self.first_name

    def email_user(self, subject, message, from_email=None, **kwargs):
        """Send an email to this user."""
        send_mail(subject, message, from_email, [self.email], **kwargs)

Let’s review what we changed here:

• We switched the username_validator to use ASCIIUsernameValidator
• The username field now is using CICharField which is not case-sensitive
• The email field is now mandatory, unique and is using CIEmailField which is not case-sensitive

On the settings module, add the following configuration:

settings.py

AUTH_USER_MODEL = "accounts.CustomUser"

Now we are ready to create our migrations:

python manage.py makemigrations 

Apply the migrations:

python manage.py migrate

And you should get a similar result if you are just creating your project and if there is no other models/apps:

Operations to perform:
  Apply all migrations: accounts, admin, auth, contenttypes, sessions
Running migrations:
  Applying contenttypes.0001_initial... OK
  Applying contenttypes.0002_remove_content_type_name... OK
  Applying auth.0001_initial... OK
  Applying auth.0002_alter_permission_name_max_length... OK
  Applying auth.0003_alter_user_email_max_length... OK
  Applying auth.0004_alter_user_username_opts... OK
  Applying auth.0005_alter_user_last_login_null... OK
  Applying auth.0006_require_contenttypes_0002... OK
  Applying auth.0007_alter_validators_add_error_messages... OK
  Applying auth.0008_alter_user_username_max_length... OK
  Applying auth.0009_alter_user_last_name_max_length... OK

If you check your database scheme you will see that there is no auth_user table (which is the default one), and now the user is stored on the table accounts_customuser:

And all the Foreign Keys to the user model will be created pointing to this table. That’s why it is important to do it right in the beginning of your project, before you created the database scheme.

Now you have all the freedom. You can replace the first_name and last_name and use just one field called name. You could remove the username field and identify your User model with the email (then just make sure you change the property USERNAME_FIELD to email).

You can grab the source code on GitHub: customuser

Handling case-insensitive without PostgreSQL

If you are not using PostgreSQL and want to implement case-insensitive authentication and you have direct access to the User model, a nice hack is to create a custom manager for the User model, like this:

accounts/models.py

from django.contrib.auth.models import AbstractUser, UserManager

class CustomUserManager(UserManager):
    def get_by_natural_key(self, username):
        case_insensitive_username_field = '{}__iexact'.format(self.model.USERNAME_FIELD)
        return self.get(**{case_insensitive_username_field: username})

class CustomUser(AbstractBaseUser, PermissionsMixin):
    # all the fields, etc...

    objects = CustomUserManager()

    # meta, methods, etc...

Then you could also sanitize the username field on the clean() method to always save it as lowercase so you don’t have to bother having case variant/conflicting usernames:

def clean(self):
    super().clean()
    self.email = self.__class__.objects.normalize_email(self.email)
    self.username = self.username.lower()

Conclusions

In this tutorial we discussed a few caveats of the default User model implementation and presented a few options to address those issues.

The takeaway message here is: always replace the default User model.

If your project is already in production, don’t panic: there are ways to fix those issues following the recommendations in this post.

I also have two detailed blog posts on how to make the username field case-insensitive and other about how to extend the django user model:

• How to Implement Case-Insensitive Username
• How to Extend Django User Model

You can also explore the source code presented in this post on GitHub:

• sibtc/django-user-caveats

How to Start a Production-Ready Django Project

In this tutorial I’m going to show you how I usually start and organize a new Django project nowadays. I’ve tried many different configurations and ways to organize the project, but for the past 4 years or so this has been consistently my go-to setup.

Please note that this is not intended to be a “best practice” guide or to fit every use case. It’s just the way I like to use Django and that’s also the way that I found that allow your project to grow in healthy way.

Index

• Premises
• Environments/Modes
  • Local
  • Tests
  • Production
• Project Configuration
  • Requirements
  • Settings
  • Version
• Apps Configuration
  • App Structure
• Code style and formatting
  • Editor Config
  • Flake8
  • isort
  • Black
• Conclusions

Premises

Usually those are the premises I take into account when setting up a project:

• Separation of code and configuration
• Multiple environments (production, staging, development, local)
• Local/development environment first
• Internationalization and localization
• Testing and documentation
• Static checks and styling rules
• Not all apps must be pluggable
• Debugging and logging

Environments/Modes

Usually I work with three environment dimensions in my code: local, tests and production. I like to see it as a “mode” how I run the project. What dictates which mode I’m running the project is which settings.py I’m currently using.

Local

The local dimension always come first. It is the settings and setup that a developer will use on their local machine.

All the defaults and configurations must be done to attend the local development environment first.

The reason why I like to do it that way is that the project must be as simple as possible for a new hire to clone the repository, run the project and start coding.

The production environment usually will be configured and maintained by experienced developers and by those who are more familiar with the code base itself. And because the deployment should be automated, there is no reason for people being re-creating the production server over and over again. So it is perfectly fine for the production setup require a few extra steps and configuration.

Tests

The tests environment will be also available locally, so developers can test the code and run the static checks.

But the idea of the tests environment is to expose it to a CI environment like Travis CI, Circle CI, AWS Code Pipeline, etc.

It is a simple setup that you can install the project and run all the unit tests.

Production

The production dimension is the real deal. This is the environment that goes live without the testing and debugging utilities.

I also use this “mode” or dimension to run the staging server.

A staging server is where you roll out new features and bug fixes before applying to the production server.

The idea here is that your staging server should run in production mode, and the only difference is going to be your static/media server and database server. And this can be achieved just by changing the configuration to tell what is the database connection string for example.

But the main thing is that you should not have any conditional in your code that checks if it is the production or staging server. The project should run exactly in the same way as in production.

Project Configuration

Right from the beginning it is a good idea to setup a remote version control service. My go-to option is Git on GitHub. Usually I create the remote repository first then clone it on my local machine to get started.

Let’s say our project is called simple, after creating the repository on GitHub I will create a directory named simple on my local machine, then within the simple directory I will clone the repository, like shown on the structure below:

simple/
└── simple/  (git repo)

Then I create the virtualenv outside of the Git repository:

simple/
├── simple/
└── venv/

Then alongside the simple and venv directories I may place some other support files related to the project which I do not plan to commit to the Git repository.

The reason I do that is because it is more convenient to destroy and re-create/re-clone both the virtual environment or the repository itself.

It is also good to store your virtual environment outside of the git repository/project root so you don’t need to bother ignoring its path when using libs like flake8, isort, black, tox, etc.

You can also use tools like virtualenvwrapper to manage your virtual environments, but I prefer doing it that way because everything is in one place. And if I no longer need to keep a given project on my local machine, I can delete it completely without leaving behind anything related to the project on my machine.

The next step is installing Django inside the virtualenv so we can use the django-admin commands.

source venv/bin/activate
pip install django

Inside the simple directory (where the git repository was cloned) start a new project:

django-admin startproject simple .

Attention to the . in the end of the command. It is necessary to not create yet another directory called simple.

So now the structure should be something like this:

simple/                   <- (1) Wrapper directory with all project contents including the venv
├── simple/               <- (2) Project root and git repository
│   ├── .git/
│   ├── manage.py
│   └── simple/           <- (3) Project package, apps, templates, static, etc
│       ├── __init__.py
│       ├── asgi.py
│       ├── settings.py
│       ├── urls.py
│       └── wsgi.py
└── venv/

At this point I already complement the project package directory with three extra directories for templates, static and locale.

Both templates and static we are going to manage at a project-level and app-level. Those are refer to the global templates and static files.

The locale is necessary in case you are using i18n to translate your application to other languages. So here is where you are going to store the .mo and .po files.

So the structure now should be something like this:

simple/
├── simple/
│   ├── .git/
│   ├── manage.py
│   └── simple/
│       ├── locale/
│       ├── static/
│       ├── templates/
│       ├── __init__.py
│       ├── asgi.py
│       ├── settings.py
│       ├── urls.py
│       └── wsgi.py
└── venv/

Requirements

Inside the project root (2) I like to create a directory called requirements with all the .txt files, breaking down the project dependencies like this:

• base.txt: Main dependencies, strictly necessary to make the project run. Common to all environments
• tests.txt: Inherits from base.txt + test utilities
• local.txt: Inherits from tests.txt + development utilities
• production.txt: Inherits from base.txt + production only dependencies

Note that I do not have a staging.txt requirements file, that’s because the staging environment is going to use the production.txt requirements so we have an exact copy of the production environment.

simple/
├── simple/
│   ├── .git/
│   ├── manage.py
│   ├── requirements/
│   │   ├── base.txt
│   │   ├── local.txt
│   │   ├── production.txt
│   │   └── tests.txt
│   └── simple/
│       ├── locale/
│       ├── static/
│       ├── templates/
│       ├── __init__.py
│       ├── asgi.py
│       ├── settings.py
│       ├── urls.py
│       └── wsgi.py
└── venv/

Now let’s have a look inside each of those requirements file and what are the python libraries that I always use no matter what type of Django project I’m developing.

base.txt

dj-database-url==0.5.0
Django==3.2.4
psycopg2-binary==2.9.1
python-decouple==3.4
pytz==2021.1

• dj-database-url: This is a very handy Django library to create an one line database connection string which is convenient for storing in .env files in a safe way
• Django: Django itself
• psycopg2-binary: PostgreSQL is my go-to database when working with Django. So I always have it here for all my environments
• python-decouple: A typed environment variable manager to help protect sensitive data that goes to your settings.py module. It also helps with decoupling configuration from source code
• pytz: For timezone aware datetime fields

tests.txt

-r base.txt

black==21.6b0
coverage==5.5
factory-boy==3.2.0
flake8==3.9.2
isort==5.9.1
tox==3.23.1

The -r base.txt inherits all the requirements defined in the base.txt file

• black: A Python auto-formatter so you don’t have to bother with styling and formatting your code. It let you focus on what really matters while coding and doing code reviews
• coverage: Lib to generate test coverage reports of your project
• factory-boy: A model factory to help you setup complex test cases where the code you are testing rely on multiple models being set in a certain way
• flake8: Checks for code complexity, PEPs, formatting rules, etc
• isort: Auto-formatter for your imports so all imports are organized by blocks (standard library, Django, third-party, first-party, etc)
• tox: I use tox as an interface for CI tools to run all code checks and unit tests

local.txt

-r tests.txt

django-debug-toolbar==3.2.1
ipython==7.25.0

The -r tests.txt inherits all the requirements defined in the base.txt and tests.txt file

• django-debug-toolbar: 99% of the time I use it to debug the query count on complex views so you can optimize your database access
• ipython: Improved Python shell. I use it all the time during the development phase to start some implementation or to inspect code

production.txt

-r base.txt

gunicorn==20.1.0
sentry-sdk==1.1.0

The -r base.txt inherits all the requirements defined in the base.txt file

• gunicorn: A Python WSGI HTTP server for production used behind a proxy server like Nginx
• sentry-sdk: Error reporting/logging tool to catch exceptions raised in production

Settings

Also following the environments and modes premise I like to setup multiple settings modules. Those are going to serve as the entry point to determine in which mode I’m running the project.

Inside the simple project package, I create a new directory called settings and break down the files like this:

simple/                       (1)
├── simple/                   (2)
│   ├── .git/
│   ├── manage.py
│   ├── requirements/
│   │   ├── base.txt
│   │   ├── local.txt
│   │   ├── production.txt
│   │   └── tests.txt
│   └── simple/              (3)
│       ├── locale/
│       ├── settings/
│       │   ├── __init__.py
│       │   ├── base.py
│       │   ├── local.py
│       │   ├── production.py
│       │   └── tests.py
│       ├── static/
│       ├── templates/
│       ├── __init__.py
│       ├── asgi.py
│       ├── urls.py
│       └── wsgi.py
└── venv/

Note that I removed the settings.py that used to live inside the simple/ (3) directory.

The majority of the code will live inside the base.py settings module.

Everything that we can set only once in the base.py and change its value using python-decouple we should keep in the base.py and never repeat/override in the other settings modules.

After the removal of the main settings.py a nice touch is to modify the manage.py file to set the local.py as the default settings module so we can still run commands like python manage.py runserver without any further parameters:

manage.py

#!/usr/bin/env python
"""Django's command-line utility for administrative tasks."""
import os
import sys


def main():
    """Run administrative tasks."""
    os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'simple.settings.local')  # <- here!
    try:
        from django.core.management import execute_from_command_line
    except ImportError as exc:
        raise ImportError(
            "Couldn't import Django. Are you sure it's installed and "
            "available on your PYTHONPATH environment variable? Did you "
            "forget to activate a virtual environment?"
        ) from exc
    execute_from_command_line(sys.argv)


if __name__ == '__main__':
    main()

Now let’s have a look on each of those settings modules.

base.py

from pathlib import Path

import dj_database_url
from decouple import Csv, config

BASE_DIR = Path(__file__).resolve().parent.parent


# ==============================================================================
# CORE SETTINGS
# ==============================================================================

SECRET_KEY = config("SECRET_KEY", default="django-insecure$simple.settings.local")

DEBUG = config("DEBUG", default=True, cast=bool)

ALLOWED_HOSTS = config("ALLOWED_HOSTS", default="127.0.0.1,localhost", cast=Csv())

INSTALLED_APPS = [
    "django.contrib.admin",
    "django.contrib.auth",
    "django.contrib.contenttypes",
    "django.contrib.sessions",
    "django.contrib.messages",
    "django.contrib.staticfiles",
]

DEFAULT_AUTO_FIELD = "django.db.models.BigAutoField"

ROOT_URLCONF = "simple.urls"

INTERNAL_IPS = ["127.0.0.1"]

WSGI_APPLICATION = "simple.wsgi.application"


# ==============================================================================
# MIDDLEWARE SETTINGS
# ==============================================================================

MIDDLEWARE = [
    "django.middleware.security.SecurityMiddleware",
    "django.contrib.sessions.middleware.SessionMiddleware",
    "django.middleware.common.CommonMiddleware",
    "django.middleware.csrf.CsrfViewMiddleware",
    "django.contrib.auth.middleware.AuthenticationMiddleware",
    "django.contrib.messages.middleware.MessageMiddleware",
    "django.middleware.clickjacking.XFrameOptionsMiddleware",
]


# ==============================================================================
# TEMPLATES SETTINGS
# ==============================================================================

TEMPLATES = [
    {
        "BACKEND": "django.template.backends.django.DjangoTemplates",
        "DIRS": [BASE_DIR / "templates"],
        "APP_DIRS": True,
        "OPTIONS": {
            "context_processors": [
                "django.template.context_processors.debug",
                "django.template.context_processors.request",
                "django.contrib.auth.context_processors.auth",
                "django.contrib.messages.context_processors.messages",
            ],
        },
    },
]


# ==============================================================================
# DATABASES SETTINGS
# ==============================================================================

DATABASES = {
    "default": dj_database_url.config(
        default=config("DATABASE_URL", default="postgres://simple:simple@localhost:5432/simple"),
        conn_max_age=600,
    )
}


# ==============================================================================
# AUTHENTICATION AND AUTHORIZATION SETTINGS
# ==============================================================================

AUTH_PASSWORD_VALIDATORS = [
    {
        "NAME": "django.contrib.auth.password_validation.UserAttributeSimilarityValidator",
    },
    {
        "NAME": "django.contrib.auth.password_validation.MinimumLengthValidator",
    },
    {
        "NAME": "django.contrib.auth.password_validation.CommonPasswordValidator",
    },
    {
        "NAME": "django.contrib.auth.password_validation.NumericPasswordValidator",
    },
]


# ==============================================================================
# I18N AND L10N SETTINGS
# ==============================================================================

LANGUAGE_CODE = config("LANGUAGE_CODE", default="en-us")

TIME_ZONE = config("TIME_ZONE", default="UTC")

USE_I18N = True

USE_L10N = True

USE_TZ = True

LOCALE_PATHS = [BASE_DIR / "locale"]


# ==============================================================================
# STATIC FILES SETTINGS
# ==============================================================================

STATIC_URL = "/static/"

STATIC_ROOT = BASE_DIR.parent.parent / "static"

STATICFILES_DIRS = [BASE_DIR / "static"]

STATICFILES_FINDERS = (
    "django.contrib.staticfiles.finders.FileSystemFinder",
    "django.contrib.staticfiles.finders.AppDirectoriesFinder",
)


# ==============================================================================
# MEDIA FILES SETTINGS
# ==============================================================================

MEDIA_URL = "/media/"

MEDIA_ROOT = BASE_DIR.parent.parent / "media"



# ==============================================================================
# THIRD-PARTY SETTINGS
# ==============================================================================


# ==============================================================================
# FIRST-PARTY SETTINGS
# ==============================================================================

SIMPLE_ENVIRONMENT = config("SIMPLE_ENVIRONMENT", default="local")

A few comments on the overall base settings file contents:

• The config() are from the python-decouple library. It is exposing the configuration to an environment variable and retrieving its value accordingly to the expected data type. Read more about python-decouple on this guide: How to Use Python Decouple
• See how configurations like SECRET_KEY, DEBUG and ALLOWED_HOSTS defaults to local/development environment values. That means a new developer won’t need to set a local .env and provide some initial value to run locally
• On the database settings block we are using the dj_database_url to translate this one line string to a Python dictionary as Django expects
• Note that how on the MEDIA_ROOT we are navigating two directories up to create a media directory outside the git repository but inside our project workspace (inside the directory simple/ (1)). So everything is handy and we won’t be committing test uploads to our repository
• In the end of the base.py settings I reserve two blocks for third-party Django libraries that I may install, such as Django Rest Framework or Django Crispy Forms. And the first-party settings refer to custom settings that I may create exclusively for our project. Usually I will prefix them with the project name, like SIMPLE_XXX

local.py

# flake8: noqa

from .base import *

INSTALLED_APPS += ["debug_toolbar"]

MIDDLEWARE.insert(0, "debug_toolbar.middleware.DebugToolbarMiddleware")


# ==============================================================================
# EMAIL SETTINGS
# ==============================================================================

EMAIL_BACKEND = "django.core.mail.backends.console.EmailBackend"

Here is where I will setup Django Debug Toolbar for example. Or set the email backend to display the sent emails on console instead of having to setup a valid email server to work on the project.

All the code that is only relevant for the development process goes here.

You can use it to setup other libs like Django Silk to run profiling without exposing it to production.

tests.py

# flake8: noqa

from .base import *

PASSWORD_HASHERS = ["django.contrib.auth.hashers.MD5PasswordHasher"]


class DisableMigrations:
    def __contains__(self, item):
        return True

    def __getitem__(self, item):
        return None


MIGRATION_MODULES = DisableMigrations()

Here I add configurations that help us run the test cases faster. Sometimes disabling the migrations may not work if you have interdependencies between the apps models so Django may fail to create a database without the migrations.

In some projects it is better to keep the test database after the execution.

production.py

# flake8: noqa

import sentry_sdk
from sentry_sdk.integrations.django import DjangoIntegration

import simple
from .base import *

# ==============================================================================
# SECURITY SETTINGS
# ==============================================================================

CSRF_COOKIE_SECURE = True
CSRF_COOKIE_HTTPONLY = True

SECURE_HSTS_SECONDS = 60 * 60 * 24 * 7 * 52  # one year
SECURE_HSTS_INCLUDE_SUBDOMAINS = True
SECURE_SSL_REDIRECT = True
SECURE_BROWSER_XSS_FILTER = True
SECURE_CONTENT_TYPE_NOSNIFF = True
SECURE_PROXY_SSL_HEADER = ("HTTP_X_FORWARDED_PROTO", "https")

SESSION_COOKIE_SECURE = True


# ==============================================================================
# THIRD-PARTY APPS SETTINGS
# ==============================================================================

sentry_sdk.init(
    dsn=config("SENTRY_DSN", default=""),
    environment=SIMPLE_ENVIRONMENT,
    release="simple@%s" % simple.__version__,
    integrations=[DjangoIntegration()],
)

The most important part here on the production settings is to enable all the security settings Django offer. I like to do it that way because you can’t run the development server with most of those configurations on.

The other thing is the Sentry configuration.

Note the simple.__version__ on the release. Next we are going to explore how I usually manage the version of the project.

Version

I like to reuse Django’s get_version utility for a simple and PEP 440 complaint version identification.

Inside the project’s __init__.py module:

simple/
├── simple/
│   ├── .git/
│   ├── manage.py
│   ├── requirements/
│   └── simple/
│       ├── locale/
│       ├── settings/
│       ├── static/
│       ├── templates/
│       ├── __init__.py     <-- here!
│       ├── asgi.py
│       ├── urls.py
│       └── wsgi.py
└── venv/

You can do something like this:

from django import get_version

VERSION = (1, 0, 0, "final", 0)

__version__ = get_version(VERSION)

The only down side of using the get_version directly from the Django module is that it won’t be able to resolve the git hash for alpha versions.

A possible solution is making a copy of the django/utils/version.py file to your project, and then you import it locally, so it will be able to identify your git repository within the project folder.

But it also depends what kind of versioning you are using for your project. If the version of your project is not really relevant to the end user and you want to keep track of it for internal management like to identify the release on a Sentry issue, you could use a date-based release versioning.

Apps Configuration

A Django app is a Python package that you “install” using the INSTALLED_APPS in your settings file. An app can live pretty much anywhere: inside or outside the project package or even in a library that you installed using pip.

Indeed, your Django apps may be reusable on other projects. But that doesn’t mean it should. Don’t let it destroy your project design or don’t get obsessed over it. Also, it shouldn’t necessarily represent a “part” of your website/web application.

It is perfectly fine for some apps to not have models, or other apps have only views. Some of your modules doesn’t even need to be a Django app at all. I like to see my Django projects as a big Python package and organize it in a way that makes sense, and not try to place everything inside reusable apps.

The general recommendation of the official Django documentation is to place your apps in the project root (alongside the manage.py file, identified here in this tutorial by the simple/ (2) folder).

But actually I prefer to create my apps inside the project package (identified in this tutorial by the simple/ (3) folder). I create a module named apps and then inside the apps I create my Django apps. The main reason why is that it creates a nice namespace for the app. It helps you easily identify that a particular import is part of your project. Also this namespace helps when creating logging rules to handle events in a different way.

Here is an example of how I do it:

simple/                      (1)
├── simple/                  (2)
│   ├── .git/
│   ├── manage.py
│   ├── requirements/
│   └── simple/              (3)
│       ├── apps/            <-- here!
│       │   ├── __init__.py
│       │   ├── accounts/
│       │   └── core/
│       ├── locale/
│       ├── settings/
│       ├── static/
│       ├── templates/
│       ├── __init__.py
│       ├── asgi.py
│       ├── urls.py
│       └── wsgi.py
└── venv/

In the example above the folders accounts/ and core/ are Django apps created with the command django-admin startapp.

Those two apps are also always in my project. The accounts app is the one that I use the replace the default Django User model and also the place where I eventually create password reset, account activation, sign ups, etc.

The core app I use for general/global implementations. For example to define a model that will be used across most of the other apps. I try to keep it decoupled from other apps, not importing other apps resources. It usually is a good place to implement general purpose or reusable views and mixins.

Something to pay attention when using this approach is that you need to change the name of the apps configuration, inside the apps.py file of the Django app:

accounts/apps.py

from django.apps import AppConfig

class AccountsConfig(AppConfig):
    default_auto_field = 'django.db.models.BigAutoField'
    name = 'accounts'  # <- this is the default name created by the startapp command

You should rename it like this, to respect the namespace:

from django.apps import AppConfig

class AccountsConfig(AppConfig):
    default_auto_field = 'django.db.models.BigAutoField'
    name = 'simple.apps.accounts'  # <- change to this!

Then on your INSTALLED_APPS you are going to create a reference to your models like this:

INSTALLED_APPS = [
    "django.contrib.admin",
    "django.contrib.auth",
    "django.contrib.contenttypes",
    "django.contrib.sessions",
    "django.contrib.messages",
    "django.contrib.staticfiles",
    
    "simple.apps.accounts",
    "simple.apps.core",
]

The namespace also helps to organize your INSTALLED_APPS making your project apps easily recognizable.

App Structure

This is what my app structure looks like:

simple/                              (1)
├── simple/                          (2)
│   ├── .git/
│   ├── manage.py
│   ├── requirements/
│   └── simple/                      (3)
│       ├── apps/
│       │   ├── accounts/            <- My app structure
│       │   │   ├── migrations/
│       │   │   │   └── __init__.py
│       │   │   ├── static/
│       │   │   │   └── accounts/
│       │   │   ├── templates/
│       │   │   │   └── accounts/
│       │   │   ├── tests/
│       │   │   │   ├── __init__.py
│       │   │   │   └── factories.py
│       │   │   ├── __init__.py
│       │   │   ├── admin.py
│       │   │   ├── apps.py
│       │   │   ├── constants.py
│       │   │   ├── models.py
│       │   │   └── views.py
│       │   ├── core/
│       │   └── __init__.py
│       ├── locale/
│       ├── settings/
│       ├── static/
│       ├── templates/
│       ├── __init__.py
│       ├── asgi.py
│       ├── urls.py
│       └── wsgi.py
└── venv/

The first thing I do is create a folder named tests so I can break down my tests into several files. I always add a factories.py to create my model factories using the factory-boy library.

For both static and templates always create first a directory with the same name as the app to avoid name collisions when Django collect all static files and try to resolve the templates.

The admin.py may be there or not depending if I’m using the Django Admin contrib app.

Other common modules that you may have is a utils.py, forms.py, managers.py, services.py etc.

Code style and formatting

Now I’m going to show you the configuration that I use for tools like isort, black, flake8, coverage and tox.

Editor Config

The .editorconfig file is a standard recognized by all major IDEs and code editors. It helps the editor understand what is the file formatting rules used in the project.

It tells the editor if the project is indented with tabs or spaces. How many spaces/tabs. What’s the max length for a line of code.

I like to use Django’s .editorconfig file. Here is what it looks like:

.editorconfig

# https://editorconfig.org/

root = true

[*]
indent_style = space
indent_size = 4
insert_final_newline = true
trim_trailing_whitespace = true
end_of_line = lf
charset = utf-8

# Docstrings and comments use max_line_length = 79
[*.py]
max_line_length = 119

# Use 2 spaces for the HTML files
[*.html]
indent_size = 2

# The JSON files contain newlines inconsistently
[*.json]
indent_size = 2
insert_final_newline = ignore

[**/admin/js/vendor/**]
indent_style = ignore
indent_size = ignore

# Minified JavaScript files shouldn't be changed
[**.min.js]
indent_style = ignore
insert_final_newline = ignore

# Makefiles always use tabs for indentation
[Makefile]
indent_style = tab

# Batch files use tabs for indentation
[*.bat]
indent_style = tab

[docs/**.txt]
max_line_length = 79

[*.yml]
indent_size = 2

Flake8

Flake8 is a Python library that wraps PyFlakes, pycodestyle and Ned Batchelder’s McCabe script. It is a great toolkit for checking your code base against coding style (PEP8), programming errors (like “library imported but unused” and “Undefined name”) and to check cyclomatic complexity.

To learn more about flake8, check this tutorial I posted a while a go: How to Use Flake8.

setup.cfg

[flake8]
exclude = .git,.tox,*/migrations/*
max-line-length = 119

isort

isort is a Python utility / library to sort imports alphabetically, and automatically separated into sections.

To learn more about isort, check this tutorial I posted a while a go: How to Use Python isort Library.

setup.cfg

[isort]
force_grid_wrap = 0
use_parentheses = true
combine_as_imports = true
include_trailing_comma = true
line_length = 119
multi_line_output = 3
skip = migrations
default_section = THIRDPARTY
known_first_party = simple
known_django = django
sections=FUTURE,STDLIB,DJANGO,THIRDPARTY,FIRSTPARTY,LOCALFOLDER

Pay attention to the known_first_party, it should be the name of your project so isort can group your project’s imports.

Black

Black is a life changing library to auto-format your Python applications. There is no way I’m coding with Python nowadays without using Black.

Here is the basic configuration that I use:

pyproject.toml

[tool.black]
line-length = 119
target-version = ['py38']
include = '\.pyi?$'
exclude = '''
  /(
      \.eggs
    | \.git
    | \.hg
    | \.mypy_cache
    | \.tox
    | \.venv
    | _build
    | buck-out
    | build
    | dist
    | migrations
  )/
'''

Conclusions

In this tutorial I described my go-to project setup when working with Django. That’s pretty much how I start all my projects nowadays.

Here is the final project structure for reference:

simple/
├── simple/
│   ├── .git/
│   ├── .gitignore
│   ├── .editorconfig
│   ├── manage.py
│   ├── pyproject.toml
│   ├── requirements/
│   │   ├── base.txt
│   │   ├── local.txt
│   │   ├── production.txt
│   │   └── tests.txt
│   ├── setup.cfg
│   └── simple/
│       ├── __init__.py
│       ├── apps/
│       │   ├── accounts/
│       │   │   ├── migrations/
│       │   │   │   └── __init__.py
│       │   │   ├── static/
│       │   │   │   └── accounts/
│       │   │   ├── templates/
│       │   │   │   └── accounts/
│       │   │   ├── tests/
│       │   │   │   ├── __init__.py
│       │   │   │   └── factories.py
│       │   │   ├── __init__.py
│       │   │   ├── admin.py
│       │   │   ├── apps.py
│       │   │   ├── constants.py
│       │   │   ├── models.py
│       │   │   └── views.py
│       │   ├── core/
│       │   │   ├── migrations/
│       │   │   │   └── __init__.py
│       │   │   ├── static/
│       │   │   │   └── core/
│       │   │   ├── templates/
│       │   │   │   └── core/
│       │   │   ├── tests/
│       │   │   │   ├── __init__.py
│       │   │   │   └── factories.py
│       │   │   ├── __init__.py
│       │   │   ├── admin.py
│       │   │   ├── apps.py
│       │   │   ├── constants.py
│       │   │   ├── models.py
│       │   │   └── views.py
│       │   └── __init__.py
│       ├── locale/
│       ├── settings/
│       │   ├── __init__.py
│       │   ├── base.py
│       │   ├── local.py
│       │   ├── production.py
│       │   └── tests.py
│       ├── static/
│       ├── templates/
│       ├── asgi.py
│       ├── urls.py
│       └── wsgi.py
└── venv/

You can also explore the code on GitHub: django-production-template.

Guest Post: Django Crispy Forms Advanced Usage Example

This is a guest post by Serhii Kushchenko, Python/Django developer and data analyst. He is skillful in SQL, Python, Django, RESTful APIs, statistics, and machine learning.

This post aims to demonstrate the creation and processing of the large and complex form in Django using django-crispy-forms. The form contains several buttons that require executing different actions during the processing phase. Also, in the form, the number of rows and columns is not static but varies depending on some conditions. In our case, the set of rows and columns changes depending on the number of instances (columns) associated with the main object (data schema) in a many-to-one relationship.

The django-crispy-forms documentation includes a page Updating layouts on the go. That page is quite helpful. However, it does not contain any detailed example of a working project. IMHO such an example is much needed. I hope my project serves as a helpful addition to the official Django Crispy Forms documentation. Feel free to copy-paste the pieces of code that you find applicable.

Please see the full codebase here. It is ready for Heroku deployment.

Task

Suppose we have a database with information about people: name, surname, phone numbers, place of work, companies owned, etc. The task is not to process the data but to work with meta-information about that data.

Different users may need to extract varying information from the database. These users do not want to write and run SQL queries. They demand some simple and more visual solutions. The assignment is to make it possible for users to create the data schemas visually. In other words, to develop such a form and make it fully functioning.

Using their schemas, users will be able to CRUD data in the database. However, these operations are beyond the scope of the current project.

Different columns can have their specific parameters. For example, integer columns have lower and upper bounds. It is necessary to develop functionality for editing those parameters for all types of columns. For that editing, forms are used that arise after clicking the "Edit details" button on the main form.

Moreover, we have to develop a form "Create new schema" and a page with a list of all available schemas.

Solution

Described below:

1. Data models used.
2. How the required forms are generated.
3. Recognizing and handling the user-initiated state changes.

The task described above can be better solved using JavaScript together with Django forms. It would reduce the number of requests to the server and increase the speed of the application. So the user experience would improve. However, the project aimed to create an advanced example of working with Django Crispy Forms.

Here you can learn the following tricks:

1. Compose a complex Django Crispy Form using the Layout, Row, Column, Fieldset, Field, and other available classes.
2. During form generation, iterate over the Foreign Key related objects and create a form row for each of them.
3. When it makes sense to use an abstract base class and when it doesn't.
4. How to encode the required action and other data in the button name during its creation.
5. Determine which button the user clicked and implement the analog of switch-case statement to perform the required action.
6. Automatically populate the newly generated form with the request.POST data if you want and if that data is available.
7. Validation of user-entered data (phone number) using a regular expression.
8. If you have many similar models, use metaprogramming to generate ModelForm classes for those models without violating the DRY principle.

Models

According to the task, the number of columns in the schemas can be different. The users add new columns and delete existing columns. Also, they can change the type and order of columns. The columns and schemas have a many-to-one relationship that is described using the Foreign Key in Django models.

The picture shows that every schema has its name, 'Column separator' field, and 'String character' field. Also, it would be nice to save the date of the last schema modification. The following code from schemas\models.py file is pretty simple.

INTEGER_CH = "IntegerColumn"
FULLNAME_CH = "FullNameColumn"
JOB_CH = "JobColumn"
PHONE_CH = "PhoneColumn"
COMPANY_CH = "CompanyColumn"
COLUMN_TYPE_CHOICES = [
    (INTEGER_CH, "Integer"),
    (FULLNAME_CH, "Full Name"),
    (JOB_CH, "Job"),
    (PHONE_CH, "Phone"),
    (COMPANY_CH, "Company"),
]

DOUBLE_QUOTE = '"'
SINGLE_QUOTE = "'"
STRING_CHARACTER_CHOICES = [
    (DOUBLE_QUOTE, 'Double-quote(")'),
    (SINGLE_QUOTE, "Single-quote(')"),
]

COMMA = ","
SEMICOLON = ";"
COLUMN_SEPARATOR_CHOICES = [(COMMA, "Comma(,)"), (SEMICOLON, "Semicolon(;)")]


class DataSchemas(models.Model):

    name = models.CharField(max_length=100)
    column_separator = models.CharField(
        max_length=1,
        choices=COLUMN_SEPARATOR_CHOICES,
        default=COMMA,
    )
    string_character = models.CharField(
        max_length=1,
        choices=STRING_CHARACTER_CHOICES,
        default=DOUBLE_QUOTE,
    )
    modif_date = models.DateField(auto_now=True)

    def get_absolute_url(self):
        return reverse("schema_add_update", args=[str(self.id)])

Each column has a name, type, and order. All of these fields are in the base SchemaColumn(models.Model) class. This class cannot be abstract because in such a case, the code schema.schemacolumn_set.all() would not work.

Columns of type integer, first and last name, job, company, and phone number are implemented as classes derived from the base class SchemaColumn.

class SchemaColumn(models.Model):
    name = models.CharField(max_length=100)
    schema = models.ForeignKey(DataSchemas, on_delete=models.CASCADE)
    order = models.PositiveIntegerField()

    class Meta:
        unique_together = [["schema", "name"], ["schema", "order"]]

    def save(self, *args, **kwargs):
        self.validate_unique()
        super(SchemaColumn, self).save(*args, **kwargs)


class IntegerColumn(SchemaColumn):
    range_low = models.IntegerField(blank=True, null=True, default=-20)
    range_high = models.IntegerField(blank=True, null=True, default=40)


class FullNameColumn(SchemaColumn):
    first_name = models.CharField(max_length=10, blank=True, null=True)
    last_name = models.CharField(max_length=15, blank=True, null=True)


class JobColumn(SchemaColumn):
    job_name = models.CharField(max_length=100, blank=True, null=True)


class CompanyColumn(SchemaColumn):
    company_name = models.CharField(max_length=100, blank=True, null=True)


class PhoneColumn(SchemaColumn):
    phone_regex = RegexValidator(
        regex=r"^\+?1?\d{9,15}$",
        message="Phone number must be entered in the format: '+999999999'. Up to 15 digits allowed.",
    )
    phone_number = models.CharField(
        validators=[phone_regex], max_length=17, blank=True, null=True
    )  # validators should be a list

Forms

The schema editing form is quite complex. We do not use the Django built-in ModelForm class here because it is not flexible enough. Our class DataSchemaForm is a derivative of the forms.Form class. Of course, django-crispy-forms was very helpful and even essential.

from crispy_forms.layout import (
    Layout,
    Submit,
    Row,
    Column,
    Fieldset,
    Field,
    Hidden,
    ButtonHolder,
    HTML,
)

The type of column in the form depends on the class of the column. How to determine that class? The problems can arise if we use the built-in isinstance() function for derived classes such as our various column types. The following code demonstrates how the subclass check was implemented in the forms.py file when generating the form.

INTEGER_CH = "IntegerColumn"
FULLNAME_CH = "FullNameColumn"
JOB_CH = "JobColumn"
PHONE_CH = "PhoneColumn"
COMPANY_CH = "CompanyColumn"
COLUMN_TYPE_CHOICES = [
    (INTEGER_CH, "Integer"),
    (FULLNAME_CH, "Full Name"),
    (JOB_CH, "Job"),
    (PHONE_CH, "Phone"),
    (COMPANY_CH, "Company"),
]

subclasses = [
    str(subclass).split(".")[-1][:-2].lower()
    for subclass in SchemaColumn.__subclasses__()
]

# yes, somewhat redundant
column_type_switcher = {
    "integercolumn": INTEGER_CH,
    "fullnamecolumn": FULLNAME_CH,
    "jobcolumn": JOB_CH,
    "companycolumn": COMPANY_CH,
    "phonecolumn": PHONE_CH,
}

column_type_field_name = "col_type_%s" % (column.pk,)
self.fields[column_type_field_name] = forms.ChoiceField(
    label="Column type", choices=COLUMN_TYPE_CHOICES
)
for subclass in subclasses:
    if hasattr(column, subclass):
        self.fields[column_type_field_name].initial = [
            column_type_switcher.get(subclass)
        ]
        break

How new schemas are created

The function that generates the schema editing form may get the primary key of the existing schema. If such a key is not available, then the function creates a new schema and its first column. After that, the user can change the parameters of the schema, as well as add new columns.

if schema_pk:
    schema = DataSchemas.objects.get(pk=schema_pk)
else:
    # no existing schema primary key passed from the caller,
    # so create new schema and its first column
    # with default parameters
    schema = DataSchemas.objects.create(name="New Schema")
    int1 = IntegerColumn.objects.create(
        name="First Column",
        schema=schema,
        order=1,
        range_low=-20,
        range_high=40,
    )

self.fields["name"].initial = schema.name
self.fields["column_separator"].initial = schema.column_separator
self.fields["string_character"].initial = schema.string_character

In addition to the schema editing form, the application also contains a list of all created schemas.

There is nothing special about that page, so I will not describe it in detail here. Pleas see the full code and templates at https://github.com/s-kust/django-advanced-forms.

Requests processing

The picture shows that the schema editing form contains several types of buttons:

• Submit form
• Add new column
• Delete column
• Edit column details

We need to determine which button the user pressed and perform the required action.

During the form creation, the required action is encoded in the names of all the buttons. Also, the column primary key or schema primary key is encoded there. For example, delete_btn = 'delete_col_%s' % (column.pk,) or submit_form_btn = 'submit_form_%s' % (schema.pk,).

The name of the button can be found in the request.POST data only if the user has pressed that button. The following code from the views.py file searches for the button name in the request.POST data and calls the required function. A well-known method is used to implement the Python analog of switch-case statement.

btn_functions = {
    "add_new_col": process_btn_add_column,
    "delete_col": process_btn_delete_column,
    "edit_col": process_btn_edit_column_details,
    "submit_form": process_btn_submit_form,
    "save_column_chng": process_btn_save_chng_column,
}
btn_pressed = None

# source of key.startswith idea - https://stackoverflow.com/questions/13101853/select-post-get-parameters-with-regular-expression
for key in request.POST:
    if key.startswith("delete_col_"):
        btn_pressed = "delete_col"
    if key.startswith("edit_col_"):
        btn_pressed = "edit_col"
    if key.startswith("add_column_btn_"):
        btn_pressed = "add_new_col"
    if key.startswith("submit_form_"):
        btn_pressed = "submit_form"
    if key.startswith("save_column_chng_btn_"):
        btn_pressed = "save_column_chng"

    if btn_pressed is not None:
        func_to_call = self.btn_functions.get(btn_pressed)
        self.pk, form = func_to_call(self, key, form_data=request.POST)
        break

Editing parameters of different types of columns

Different types of columns have their specific parameters. For example, integer columns have lower and upper bounds. Phone columns have the phone number field that requires validation before saving. The number of different types of columns can increase over time.

How to handle the Edit Details button click? The straightforward solution is to make individual ModelForm classes for each type of column. However, it would violate the DRY principle. Perhaps, in this case, the use of metaprogramming is justified.

def get_general_column_form(self, model_class, column_pk):
    class ColumnFormGeneral(ModelForm):
        def __init__(self, *args, **kwargs):
           super(ColumnFormGeneral, self).__init__(*args, **kwargs)
           self.helper = FormHelper(self)
           save_chng_btn = "save_column_chng_btn_%s" % (column_pk,)
           self.helper.layout.append(Submit(save_chng_btn, "Save changes"))

        class Meta:
           model = model_class
           exclude = ["schema", "order"]

   return ColumnFormGeneral       

First, the type of column is determined using its primary key. After that, the get_general_column_form function is called. It returns the customized ModelForm class. Next, an instance of that class is created and used.

column = get_object_or_404(SchemaColumn, pk=column_pk)
for subclass in self.subclasses:
    if hasattr(column, subclass):
        column_model = apps.get_model("schemas", subclass)
        column = get_object_or_404(column_model, pk=column_pk)
        form_class = self.get_general_column_form(column_model, column_pk)
        form = form_class(
            initial=model_to_dict(
                column, fields=[field.name for field in column._meta.fields]
            )
        )
        break

Handling of the column type change

The user may change the type of one or several columns. If it happens, it means that the class of these columns has changed. Here, it is not enough to change the value of some attribute of the object. We have to delete the old object and create a new object belonging to the new class instead. How do we handle it:

1. First, a form is generated using the schema's primary key.
2. Then, in the newly created form, the data from the database is replaced with the request.POST data, if that data is available. It happens automatically.
3. In the next step, the form is validated. For that, we have to call the form.is_valid() method explicitly.
4. If the validation is successful, then we process every column of the schema. For each column, its type from the database is compared with its type from the form. It means that its database type is compared with its request.POST type. If these types differ, the old column is deleted, and a new one is created instead.

# elem is a 'Submit Form' button that the user pressed.
# schema primary key is encoded in its name
# first, let's decode it
self.pk = [int(s) for s in elem.split("_") if s.isdigit()][0]

# form_data is request.POST
form = DataSchemaForm(form_data, schema_pk=self.pk)
if form.is_valid():
    schema = get_object_or_404(DataSchemas, pk=self.pk)
    schema.name = form.cleaned_data["name"]
    schema.column_separator = form.cleaned_data["column_separator"]
    schema.string_character = form.cleaned_data["string_character"]
    schema.save()

    # the following code is in the save_schema_columns(self, schema, form) function
    schema_columns = schema.schemacolumn_set.all()
    for column in schema_columns:
        column_name_field_name = "col_name_%s" % (column.pk,)
        column_order_field_name = "col_order_%s" % (column.pk,)
        column_type_field_name = "col_type_%s" % (column.pk,)

        type_form = form.cleaned_data[column_type_field_name]
        type_changed = False
        for subclass in self.subclasses:
            if hasattr(column, subclass):
                type_db = self.column_type_switcher.get(subclass)
                if type_db != type_form:
                    new_class = globals()[type_form]
                    new_column = new_class()
                    new_column.name = form.cleaned_data[column_name_field_name]
                    new_column.order = form.cleaned_data[column_order_field_name]
                    new_column.schema = schema
                    column.delete()
                    new_column.save()
                    type_changed = True
                    break
        if not type_changed:
            column.name = form.cleaned_data[column_name_field_name]
            column.order = form.cleaned_data[column_order_field_name]
            column.save()

Final Words

I hope this post serves as a helpful addition to the official Django Crispy Forms documentation. Feel free to copy-paste the pieces of code that you find applicable. Also, write comments and share your experience of using Django Crispy Forms.

Cover photo by Juan Pablo Malo.

How I Tested ReactJS-based Webapp with Selenium

For quite some time, I have been building a SaaS product - strategic prioritizer 1st things 1st. It's using Django in the backend and ReactJS in the frontend and communicating between those ends by REST API. Every week I try to make progress with this project, be it a more prominent feature, some content changes, or small styling tweaks. In the past week, I implemented frontend testing with Selenium, and I want to share my journey with you.

What can you do with 1st things 1st

1st things 1st allows you to evaluate a list of items by multiple criteria and calculates priorities for you to follow and take action. The service has 4 main steps:

1. Defining criteria.
2. Listing out things.
3. Evaluating things by each criterion.
4. Exploring the priorities.

Selenium is a testing tool that mimics user interaction in the browser: you can fill in fields, trigger events, or read out information from the HTML tags. To test the frontend of 1st things 1st with Selenium, I had to

1. enter the user credentials and login,
2. create a project from a blank project template,
3. add some criteria,
4. add some things to do,
5. evaluate each thing by each criterion, and
6. see if the generated list of priorities was correct.

Let's see how I did it.

Preparation

In 2020, Chrome is the most popular browser, and it's my default browser, so I decided to develop tests using it.

I had to install Selenium with pip into my virtual environment:

(venv)$ pip install selenium

Also, I needed a binary chromedriver, which makes Selenium talk to your Chrome browser. I downloaded it and placed it under myproject/drivers/chromedriver.

In the Django project configuration, I needed a couple of settings. I usually have separate settings-file for each of the environments, such as:

• myproject.settings.local for the local development,
• myproject.settings.staging for the staging server,
• myproject.settings.test for testing, and
• myproject.settings.production for production.

All of them import defaults from a common base, and I have to set only the differences for each environment.

In the myproject.settings.test I added these settings:

WEBSITE_URL = 'http://my.1st-things-1st.127.0.0.1.xip.io:8080'  # no trailing slash

TESTS_SHOW_BROWSER = True

Here for the WEBSITE_URL, I was using the xip.io service. It allows you to create domains dynamically pointing to the localhost or any other IP. The Selenium tests will use this URL.

The TEST_SHOW_BROWSER was my custom setting, telling whether to show a browser while testing the frontend or just to run the tests in the background.

The test case

In one of my apps, myproject.apps.evaluations, I created a tests package, and there I placed a test case test_evaluations_frontend.py with the following content:

import os
from time import sleep
from datetime import timedelta

from django.conf import settings
from django.test import LiveServerTestCase
from django.test import override_settings
from django.contrib.auth import get_user_model
from django.utils import timezone

from selenium import webdriver
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.support.ui import WebDriverWait


User = get_user_model()

SHOW_BROWSER = getattr(settings, "TESTS_SHOW_BROWSER", False)


@override_settings(DEBUG=True)
class EvaluationTest(LiveServerTestCase):
    host = settings.WEBSITE_URL.rsplit(":", 1)[0].replace(
        "http://", ""
    )  # domain before port
    port = int(settings.WEBSITE_URL.rsplit(":", 1)[1])  # port
    USER1_USERNAME = "user1"
    USER1_FIRST_NAME = "user1"
    USER1_LAST_NAME = "user1"
    USER1_EMAIL = "user1@example.com"
    USER1_PASSWORD = "change-me"

    @classmethod
    def setUpClass(cls):
        # …

    @classmethod
    def tearDownClass(cls):
        # …

    def wait_until_element_found(self, xpath):
        # …

    def wait_a_little(self, seconds=2):
        # …

    def test_evaluations(self):
        # …

It's a live-server test case, which runs a Django development server under the specified IP and port and then runs the Chrome browser via Selenium and navigates through the DOM and fills in forms.

By default, the LiveServerTestCase runs in non-debug mode, but I want to have the debug mode on so that I could see any causes of server errors. With the @override_settings decorator, I could change the DEBUG setting to True.

The host and port attributes define on which host and port the test server will be running (instead of a 127.0.0.1 and a random port). I extracted those values from the WEBSITE_URL setting.

The test case also had some attributes for the user who will be navigating through the web app.

Let's dig deeper into the code for each method.

Test-case setup and teardown

Django test cases can have class-level setup and teardown, which run before and after all methods whose names start with test_:

    @classmethod
    def setUpClass(cls):
        super().setUpClass()
        cls.user1 = User.objects.create_user(
            cls.USER1_USERNAME, cls.USER1_EMAIL, cls.USER1_PASSWORD
        )
        # … add subscription for this new user …

        driver_path = os.path.join(settings.BASE_DIR, "drivers", "chromedriver")
        chrome_options = Options()
        if not SHOW_BROWSER:
            chrome_options.add_argument("--headless")
        chrome_options.add_argument("--window-size=1200,800")

        cls.browser = webdriver.Chrome(
            executable_path=driver_path, options=chrome_options
        )
        cls.browser.delete_all_cookies()

    @classmethod
    def tearDownClass(cls):
        super().tearDownClass()
        cls.browser.quit()
        # … delete subscription for the user …
        cls.user1.delete()

In the setup, I created a new user, added a subscription to them, and prepared the Chrome browser to use.

If the TEST_SHOW_BROWSER setting was False, Chrome was running headless, that is, in the background without displaying a browser window.

When the tests were over, the browser closed, and the subscription, as well as the user, were deleted.

Utility methods

I created two utility methods for my Selenium test: wait_until_element_found() and wait_a_little():

    def wait_until_element_found(self, xpath):
        WebDriverWait(self.browser, timeout=10).until(
            lambda x: self.browser.find_element_by_xpath(xpath)
        )

    def wait_a_little(self, seconds=2):
        if SHOW_BROWSER:
            sleep(seconds)

I used the wait_until_element_found(xpath) method to keep the test running while pages switched.

I used the wait_a_little(seconds) method to stop the execution for 2 or more seconds so that I could follow what's on the screen, make some screenshots, or even inspect the DOM in the Web Developer Inspector.

XPath

Selenium allows to select DOM elements by ID, name, CSS class, tag name, and other ways, but the most flexible approach, in my opinion, is selecting elements by XPath (XML Path Language).

Contrary to jQuery, ReactJS doesn't use IDs or CSS classes in the markup to update the contents of specific widgets. So the straightforward Selenium's methods for finding elements by IDs or classes won't always work.

XPath is a very flexible and powerful tool. For example, you can:

• Select elements by ID: "//input[@id='id_title']"
• Select elements by any other attribute: "//div[@aria-label='Blank']"
• Select elements by innerText: "//button[.='Save']"
• Select elements by CSS class and innerText: "//button[contains(@class,'btn-primary')][.='Save']"
• Select the first element by innerText: "(//button[.='yes'])[1]"

You can try out XPath syntax and capabilities in Web Developer Console in Chrome and Firefox, using the $x() function, for example:

»  $x("//h1[.='Projects']")
←  Array [ h1.display-4.mb-4 ]

Login and adding a project

I started with opening a login page, dismissing cookie consent notification, filling in user credentials into the login form, creating a new project from a blank template, setting title and description, etc.

    def test_evaluations(self):
        self.browser.get(f"{self.live_server_url}/")
        self.wait_until_element_found("//h1[.='Log in or Sign up']")
        # Accept Cookie Consent
        self.wait_until_element_found("//a[.='Got it!']")
        self.browser.find_element_by_xpath("//a[.='Got it!']").click()
        # Log in
        self.browser.find_element_by_id("id_email").send_keys(self.USER1_EMAIL)
        self.browser.find_element_by_id("id_password").send_keys(self.USER1_PASSWORD)
        self.browser.find_element_by_xpath('//button[text()="Log in"]').send_keys(
            "\n"
        )  # submit the form

        self.wait_until_element_found("//h1[.='Projects']")

        # Click on "Add new project"
        self.wait_until_element_found("//a[.='Add new project']")

        self.wait_a_little()
        self.browser.find_element_by_xpath("//a[.='Add new project']").send_keys("\n")

        self.wait_until_element_found("//div[@aria-label='Blank']")

        # Create a project from the project template "Blank"
        self.wait_a_little()
        self.browser.find_element_by_xpath("//div[@aria-label='Blank']").send_keys("\n")

        # Enter project title and description
        self.wait_until_element_found("//input[@id='id_title']")
        self.browser.find_element_by_xpath("//input[@id='id_title']").send_keys(
            "Urgent and Important Activities"
        )
        self.browser.find_element_by_xpath(
            "//textarea[@id='id_description']"
        ).send_keys("I want to find which things to do and which to skip.")
        self.browser.find_element_by_xpath("//button[.='Next']").send_keys("\n")

        # Keep the default verbose names for the criteria and initiatives
        self.wait_until_element_found("//input[@id='id_initiative_verbose_name_plural']")
        self.wait_a_little()
        self.browser.find_element_by_xpath("//button[.='Next']").send_keys("\n")

If TESTS_SHOW_BROWSER was set to True, we would see all this workflow in an opened browser window.

I was creating the test by carefully inspecting the markup in Web Developer Inspector and creating appropriate DOM navigation with XPath. For most of the navigation, I was using send_keys() method, which triggers keyboard events. During the testing, I also noticed that my cookie consent only worked with a mouse click, and I couldn't approve it by the keyboard. That's some room for improving accessibility.

I ran the test with the following command each time I added some more lines:

(venv)$ python manage.py test myproject.apps.evaluations --settings=myproject.settings.test

The test case failed if any command in the test failed. I didn't even need asserts.

Adding criteria

Now it was time to add some criteria:

        self.wait_until_element_found("//h2[.='Criteria']")

        # Add new criterion "Urgent" with the evaluation type Yes/No/Maybe
        self.wait_until_element_found("//a[.='Add new criterion']")
        self.browser.find_element_by_xpath("//a[.='Add new criterion']").send_keys("\n")
        self.wait_until_element_found("//input[@id='id_title']")
        self.browser.find_element_by_xpath("//input[@id='id_title']").send_keys(
            "Urgent"
        )
        self.browser.find_element_by_xpath("//input[@id='widget_y']").send_keys(" ")
        self.browser.find_element_by_xpath("//button[.='Save']").send_keys("\n")

        # Add new criterion "Important" with the evaluation type Yes/No/Maybe
        self.wait_until_element_found("//a[.='Add new criterion']")
        self.browser.find_element_by_xpath("//a[.='Add new criterion']").send_keys("\n")
        self.wait_until_element_found("//input[@id='id_title']")
        self.browser.find_element_by_xpath("//input[@id='id_title']").send_keys(
            "Important"
        )
        self.browser.find_element_by_xpath("//input[@id='widget_y']").send_keys(" ")
        self.browser.find_element_by_xpath("//button[.='Save']").send_keys("\n")

        # Click on the button "Done"
        self.wait_until_element_found("//a[.='Done']")
        self.browser.find_element_by_xpath("//a[.='Done']").send_keys("\n")

I added two criteria, "Urgent" and "Important", with evaluation type "Yes/No/Maybe".

Adding things

Then I created some activities to evaluate:

        self.wait_until_element_found("//h2[.='Things']")

        # Add new thing "Write a blog post"
        self.wait_until_element_found("//a[.='Add new thing']")
        self.browser.find_element_by_xpath("//a[.='Add new thing']").send_keys("\n")
        self.wait_until_element_found("//input[@id='id_title']")
        self.browser.find_element_by_xpath("//input[@id='id_title']").send_keys(
            "Write a blog post"
        )
        self.browser.find_element_by_xpath("//textarea[@id='id_description']").send_keys(
            "I have an idea of a blog post that I want to write."
        )
        self.browser.find_element_by_xpath("//button[.='Save']").send_keys("\n")

        # Add new thing "Fix a bug"
        self.wait_until_element_found("//a[.='Add new thing']")
        self.browser.find_element_by_xpath("//a[.='Add new thing']").send_keys("\n")
        self.wait_until_element_found("//input[@id='id_title']")
        self.browser.find_element_by_xpath("//input[@id='id_title']").send_keys(
            "Fix a bug"
        )
        self.browser.find_element_by_xpath("//textarea[@id='id_description']").send_keys(
            "There is a critical bug that bothers our clients."
        )
        self.browser.find_element_by_xpath("//button[.='Save']").send_keys("\n")

        # Add new thing "Binge-watch a series"
        self.wait_until_element_found("//a[.='Add new thing']")
        self.browser.find_element_by_xpath("//a[.='Add new thing']").send_keys("\n")
        self.wait_until_element_found("//input[@id='id_title']")
        self.browser.find_element_by_xpath("//input[@id='id_title']").send_keys(
            "Binge-watch a series"
        )
        self.browser.find_element_by_xpath("//textarea[@id='id_description']").send_keys(
            "There is an exciting series that I would like to watch."
        )
        self.browser.find_element_by_xpath("//button[.='Save']").send_keys("\n")

        # Click on the button "Done"
        self.wait_until_element_found("//a[.='Done']")
        self.browser.find_element_by_xpath("//a[.='Done']").send_keys("\n")

These were three activities: "Write a blog post", "Fix a bug", and "Binge-watch a series" with their descriptions:

Evaluating things

In this step, there was a list of widgets to evaluate each thing by each criterion with answers "No", "Maybe", or "Yes". The buttons for those answers had no specific id or CSS class, but I could target them by the text on the button using XPath like "//button[.='maybe']":

        self.wait_until_element_found("//h2[.='Evaluations']")
        self.wait_until_element_found("//button[.='maybe']")

        # Evaluate all things by Urgency
        self.browser.find_element_by_xpath("(//button[.='no'])[1]").send_keys("\n")
        self.wait_until_element_found("//footer[.='Evaluation saved.']")
        self.browser.find_element_by_xpath("(//button[.='yes'])[2]").send_keys("\n")
        self.wait_until_element_found("//footer[.='Evaluation saved.']")
        self.browser.find_element_by_xpath("(//button[.='no'])[3]").send_keys("\n")
        self.wait_until_element_found("//footer[.='Evaluation saved.']")

        # Evaluate all things by Importance
        self.browser.find_element_by_xpath("(//button[.='yes'])[4]").send_keys("\n")
        self.wait_until_element_found("//footer[.='Evaluation saved.']")
        self.browser.find_element_by_xpath("(//button[.='yes'])[5]").send_keys("\n")
        self.wait_until_element_found("//footer[.='Evaluation saved.']")
        self.browser.find_element_by_xpath("(//button[.='maybe'])[6]").send_keys("\n")
        self.wait_until_element_found("//footer[.='Evaluation saved.']")

        # Click on the button "Done"
        self.browser.find_element_by_xpath("//a[.='Done']").send_keys("\n")

These were my evaluations:

• "Write a blog post" was not urgent, but important.
• "Fix a bug" was urgent and important.
• "Binge-watch a series" was not urgent and maybe important (because one has to have rest and feed imagination too).

Checking priorities

So in the last step, I got the calculated priorities:

        self.wait_until_element_found("//h2[.='Priorities']")

        self.wait_until_element_found("//h5[.='1. Fix a bug (100%)']")
        self.wait_until_element_found("//h5[.='2. Write a blog post (50%)']")
        self.wait_until_element_found("//h5[.='3. Binge-watch a series (25%)']")
        self.wait_a_little()

The results looked correct:

• "Fix a bug" was of the 100% priority.
• "Write a blog post" was of the 50% priority.
• "Binge-watch a series was of the 25% priority.

Final words

• Selenium needs a binary browser driver that lets you manipulate DOM in the browser from Python.
• You can set a specific host and port for a LiveServerTestCase.
• The Chrome browser can be displayed or executed in the background, depending on your settings.
• XPath is a flexible and powerful tool to address DOM elements by any attributes or even inner text.
• Selenium can trigger keyboard or mouse events that are handled by JavaScript functions.

I hope that my journey was useful to you too.

Happy coding!

Thanks a lot to Adam Johnson for the review.
Cover photo by Science in HD.

How to Upload a File Using Django REST Framework

When you develop a web app or a mobile app with Django, it is common to use the Django REST Framework for communication with the server-side. The client-side makes GET, POST, PUT, and DELETE requests to the REST API to read, create, update, or delete data there. The communication by Ajax is pretty uncomplicated, but how would you upload an image or another file to the server? I will show you that in this article by creating user avatar upload via REST API. Find the full code for this feature on Github.

Extend Django User model

We will start by installing Pillow for image handling to the virtual environment using the standard pip command:

(venv)$ pip install Pillow

Create accounts app with a custom User model:

# myproject/apps/accounts/models.py
import os
import sys
from django.db import models
from django.contrib.auth.models import AbstractUser
from django.utils import timezone
from django.utils.translation import gettext_lazy as _

def upload_to(instance, filename):
    now = timezone.now()
    base, extension = os.path.splitext(filename.lower())
    milliseconds = now.microsecond // 1000
    return f"users/{instance.pk}/{now:%Y%m%d%H%M%S}{milliseconds}{extension}"

class User(AbstractUser):
    # …
    avatar = models.ImageField(_("Avatar"), upload_to=upload_to, blank=True)

You can add there as many fields as you need, but the noteworthy part there is the avatar field.

Update the settings and add the accounts app to INSTALLED_APPS, set the AUTH_USER_MODEL, and the configuration for the static and media directories:

# myproject/settings.py
INSTALLED_APPS = [
    # …
    "myproject.apps.accounts",
]

AUTH_USER_MODEL = "accounts.User"

STATICFILES_DIRS = [os.path.join(BASE_DIR, "myproject", "site_static")]
STATIC_ROOT = os.path.join(BASE_DIR, "myproject", "static")
STATIC_URL = "/static/"
MEDIA_ROOT = os.path.join(BASE_DIR, "myproject", "media")
MEDIA_URL = "/media/"

Next small steps:

• Create and run migrations with the makemigrations and migrate management commands.
• Set up the custom model administration for the new User model.
• Create the superuser with the createsuperuser management command.

Install and configure Django REST Framework

Install Django REST Framework for the REST APIs to your virtual environment, as always, using pip:

(venv)$ pip install djangorestframework

We'll be using authentication by tokens in this example. So add Django REST Framework to INSTALLED_APPS in the settings and set TokenAuthentication as the default authentication in the REST_FRAMEWORK configuration:

# myproject/settings.py
INSTALLED_APPS = [
    # …
    "rest_framework",
    "rest_framework.authtoken",
    # …
]

REST_FRAMEWORK = {
    'DEFAULT_AUTHENTICATION_CLASSES': [
        'rest_framework.authentication.TokenAuthentication',
    ]
}

Prepare the serializer and the view

In Django REST Framework, serializers are used for data validation, rendering, and saving. They are similar to Django forms. Prepare UserAvatarSerializer for avatar uploads:

# myproject/apps/accounts/serializers.py
from django.contrib.auth import get_user_model
from rest_framework.serializers import ModelSerializer
User = get_user_model()

class UserAvatarSerializer(ModelSerializer):
    class Meta:
        model = User
        fields = ["avatar"]

    def save(self, *args, **kwargs):
        if self.instance.avatar:
            self.instance.avatar.delete()
        return super().save(*args, **kwargs)

Now create an API view UserAvatarUpload for avatar uploads.

# myproject/apps/accounts/views.py
from rest_framework import status
from rest_framework.parsers import MultiPartParser, FormParser
from rest_framework.permissions import IsAuthenticated
from rest_framework.response import Response
from rest_framework.views import APIView
from .serializers import UserAvatarSerializer

class UserAvatarUpload(APIView):
    parser_classes = [MultiPartParser, FormParser]
    permission_classes = [IsAuthenticated]

    def post(self, request, format=None):
        serializer = UserAvatarSerializer(data=request.data, instance=request.user)
        if serializer.is_valid():
            serializer.save()
            return Response(serializer.data, status=status.HTTP_200_OK)
        else:
            return Response(serializer.errors, status=status.HTTP_400_BAD_REQUEST)

Make sure that the view uses MultiPartParser as one of the parser classes. That's necessary for the file transfers.

Prepare the URL configuration

In the URL configuration, we will need those URL rules:

• The path for the index page. Let's make it a direct TemplateView.
• The path for logging in by user credentials and obtaining the authentication token.
• The path for user avatar upload.
• The path for model administration.
• The path for static URLs.
• And finally, the path for media URLs.

# myroject/urls.py
from django.conf.urls.static import static
from django.contrib import admin
from django.urls import path
from django.views.generic import TemplateView
from django.conf import settings
from myproject.accounts.views import UserAvatarUpload
from rest_framework.authtoken.views import obtain_auth_token

urlpatterns = [
    path("", TemplateView.as_view(template_name="index.html")),
    path("api/auth-token/", obtain_auth_token, name="rest_auth_token"),
    path("api/user-avatar/", UserAvatarUpload.as_view(), name="rest_user_avatar_upload"),
    path("admin/", admin.site.urls),
]

urlpatterns += static(settings.STATIC_URL, document_root=settings.STATIC_ROOT)
urlpatterns += static(settings.MEDIA_URL, document_root=settings.MEDIA_ROOT)

Prepare the frontend HTML and JavaScript

I will illustrate the frontend using Bootstrap HTML and Vanilla JavaScript. Of course, you can implement the same using ReactJS, Vue, Angular, or other JavaScript framework and any other CSS framework.

The template for the index page has one login form with username and password or email and password fields (depending on your implementation), and one avatar upload form with a file selection field. Also, it includes a JavaScript file avatar.js for Ajax communication.

{# myproject/templates/index.html #}
<!doctype html>
{% load static %}
<html lang="en">
<head>
    <meta charset="utf-8">
    <meta name="viewport" content="width=device-width, initial-scale=1, shrink-to-fit=no">

    <link rel="stylesheet" href="https://stackpath.bootstrapcdn.com/bootstrap/4.4.1/css/bootstrap.min.css"
          integrity="sha384-Vkoo8x4CGsO3+Hhxv8T/Q5PaXtkKtu6ug5TOeNV6gBiFeWPGFN9MuhOf23Q9Ifjh" crossorigin="anonymous">

    <title>Hello, World!</title>
</head>
<body>
<div class="container">
    <div class="row">
        <div class="col-md-8">
            <p class="text-muted my-3"><small>Open Developer Console for information about responses.</small></p>
            <h1 class="my-3">1. Log in</h1>
            <form id="login_form">
                <div class="form-group">
                    <label for="id_email">Email address</label>
                    <input type="email" class="form-control" id="id_email" aria-describedby="emailHelp"
                           placeholder="Enter email"/>
                </div>
                <div class="form-group">
                    <label for="id_password">Password</label>
                    <input type="password" class="form-control" id="id_password" placeholder="Password"/>
                </div>
                <button type="submit" class="btn btn-primary">Log in</button>
            </form>

            <h1 class="my-3">2. Upload an avatar</h1>
            <form id="avatar_form">
                <div class="form-group">
                    <label for="id_avatar">Choose an image for your avatar</label>
                    <input type="file" class="form-control-file" id="id_avatar"/>
                </div>
                <button type="submit" class="btn btn-primary">Upload</button>
            </form>

        </div>
    </div>
</div>
<script src="{% static 'site/js/avatar.js' %}"></script>
</body>
</html>

Last but not least, create the JavaScript file avatar.js. It contains these things:

• a global variable to store the user token. In the real-world application, you would probably save the token in a cookie or local storage.
• a login-form submit handler which posts user credentials to the server and retrieves the authentication token.
• an avatar-form submit handler which posts the selected file and the token to the server and retrieves the path of the saved file on the server.

// myproject/site_static/site/js/avatar.js
let userToken;

document.getElementById('login_form').addEventListener('submit', function(event) {
    event.preventDefault();
    let email = document.getElementById('id_email').value;
    let password = document.getElementById('id_password').value;

    fetch('http://127.0.0.1:8000/api/auth-token/', {
        method: 'POST',
        headers: {
            'Content-Type': 'application/json',
        },
        body: JSON.stringify({
            "username": email,
            "password": password,
        })
    }).then( response => {
        return response.json();
    }).then(data => {
        console.log(data);
        userToken = data.token;
        console.log('Logged in. Got the token.');
    }).catch((error) => {
        console.error('Error:', error);
    });
});

document.getElementById('avatar_form').addEventListener('submit', function(event) {
    event.preventDefault();
    let input = document.getElementById('id_avatar');

    let data = new FormData();
    data.append('avatar', input.files[0]);

    fetch('http://127.0.0.1:8000/api/user-avatar/', {
        method: 'POST',
        headers: {
            'Authorization': `Token ${userToken}`
        },
        body: data
    }).then(response => {
        return response.json();
    }).then(data => {
        console.log(data);
    }).catch((error) => {
        console.error('Error:', error);
    });
});

In the JavaScript file, we are using fetch API for the REST API requests. The noteworthy part there is the FormData class that we use to send the file to the server.

Now run the local development server and go to the http://127.0.0.1:8000. There you will have something like this:

Final Thoughts

As more than a half Internet usage happens on mobile devices, there is a demand to switch from usual HTML websites and platforms to mobile apps. Whether you create a native mobile app, a hybrid app, or Progressive Web App, you will likely have to communicate with the server via REST API or GraphQL. It is pretty clear how to transfer textual data from and to a remote server. But after this exercise, we can also transfer binary files like images, PDF or Word documents, music, and videos.

Happy coding!

Cover Photo by Dan Silva

Use the Google Analytics API with Django

Google Analytics is a powerful tool that gives you useful insight about the traffic that arrives in your website. But together with Django, you will be able to retrieve data of your Analytics account and display it to the user, on demand.

In this post you’ll learn how to put together Django, the oauth2 protocol and the Google Analytics API.

Let’s Start!

First of all, you might want to check this post if you’re not familiar with the Google Analytics API with Python.

Moreover, it assumes that you are familiar with Python and pip, and that you know Django and have a working application. If you navigate into the Python and Django sections of this website you’ll learn more about those

Create a Google Application

I assume you have a working Django application, and that you want to extract data from your Google Analytics account using an existing Project. If you didn’t create a Project in the Google Developers Console, check this post and create one. Then, come back

But if you have a live Django project, then you need to create a Web application instead. You just need to select Web application as the Application type when you are creating a Client Id in the Credentials section:

In Authorized Javascript origins insert your website domain, and in authorized redirect uris, insert your domain plus /oauth2/oauth2callback. This endpoint will be used by django to catch the oatuh2 response.

After that, you should have the Oauth credentials in a json format.

Install oauth2 and Google Analytics

Next, we need to install the oaut2client python package. Activate your virtual environment (if you have one) and type:

$ pip install oauth2client

This will install oauth2client and the following packages: httplib2 (an http library), pyasn1 and pyasn1-modules (to use the ASN.1 protocol), and rsa to use the RSA cryptosystem).

Don’t forget to add these packages into the requirements file of your virtual environment.

And also, we will need the API Python Client library:

$ pip install python-gflags
$ pip install -U google-api-python-client

Create a Django App

Next, we will create a Django app to manage the Authorization process. Depending on where you want to create the app’s folder, you will run a slightly different command (this one will create the folder in the current directory):

$ python manage.py startapp oauth2_authentication

Take the client_secrets.json file and save it inside this app, and include this app into your settings.py file, in the INSTALLED_APPS.

Now that we have the app created, we are going to create two different models that will store the Flow and Credential objects.

Edit the models.py inside the oaut2_authentication app and write:

from django.db import models
from django.contrib.auth.models import User
from oauth2client.django_orm import FlowField, CredentialsField


class FlowModel(models.Model):
    id = models.ForeignKey(User, primary_key=True)
    flow = FlowField()


class CredentialsModel(models.Model):
    id = models.ForeignKey(User, primary_key=True)
    credential = CredentialsField()

Next, we’ll edit the urls of your app. First, open the main urls.py and add the following urlpattern:

url(r'^oauth2/', include('oauth2_authentication.urls', namespace="oauth2"))

And in the oauth2_authentication/urls.py write:

from django.conf.urls import patterns, url
from . import views

urlpatterns = patterns(
    '',
    url(r'^$', views.index, name='index'),
    url(r'oauth2callback', views.auth_return, name='return'),
)

This way, /oauth2 will start the authorization process, and /oauth2/oaut2callback will wait for the authorization response.

But we need to write our views.py first:

import os
import httplib2
from oauth2client import xsrfutil
from oauth2client.client import flow_from_clientsecrets
from oauth2client.django_orm import Storage

from apiclient.discovery import build

from django.contrib.auth.decorators import login_required
from django.http import HttpResponseBadRequest
from django.http import HttpResponseRedirect
from django.shortcuts import render
from django.conf import settings
from django.contrib.auth import get_user_model
from django.core.urlresolvers import reverse
from django.contrib.sites.models import get_current_site

from .models import CredentialsModel, FlowModel

CLIENT_SECRETS = os.path.join(
    os.path.dirname(__file__), 'client_secrets.json')


def get_accounts_ids(service):
    accounts = service.management().accounts().list().execute()
    ids = []
    if accounts.get('items'):
        for account in accounts['items']:
            ids.append(account['id'])
    return ids


@login_required
def index(request):
    # use the first REDIRECT_URI if you are developing your app
    # locally, and the second in production
    # REDIRECT_URI = 'http://localhost:8000/oauth2/oauth2callback'
    REDIRECT_URI = "https://%s%s" % (
        get_current_site(request).domain, reverse("oauth2:return"))
    FLOW = flow_from_clientsecrets(
        CLIENT_SECRETS,
        scope='https://www.googleapis.com/auth/analytics.readonly',
        redirect_uri=REDIRECT_URI
    )
    user = request.user
    storage = Storage(CredentialsModel, 'id', user, 'credential')
    credential = storage.get()
    if credential is None or credential.invalid is True:
        FLOW.params['state'] = xsrfutil.generate_token(
            settings.SECRET_KEY, user)
        authorize_url = FLOW.step1_get_authorize_url()
        f = FlowModel(id=user, flow=FLOW)
        f.save()
        return HttpResponseRedirect(authorize_url)
    else:
        http = httplib2.Http()
        http = credential.authorize(http)
        service = build('analytics', 'v3', http=http)
        ids = get_account_ids(service)
        return render(
            request, 'oauth2_authentication/main.html', {'ids':ids})


@login_required
def auth_return(request):
    user = request.user
    if not xsrfutil.validate_token(
            settings.SECRET_KEY, request.REQUEST['state'], user):
        return HttpResponseBadRequest()
    FLOW = FlowModel.objects.get(id=user).flow
    credential = FLOW.step2_exchange(request.REQUEST)
    storage = Storage(CredentialsModel, 'id', user, 'credential')
    storage.put(credential)
    return HttpResponseRedirect("/oauth2")

Note that you have to choose which REDIRECT_URI to use, depending on if you are developing your Django app locally or your app is live on a server.

Inside the FLOW object, we have used the readonly scope, but you could also choose:

https://www.googleapis.com/auth/analytics for writing permissions, and https://www.googleapis.com/auth/analytics.manage.users to view and manage permission of users in the Analytics accounts (more info).

And finally, write something in the oauth2_authentication/main.html template, like:

Your credentials are up to date! :-)

{% if ids %}
The List of Ids of your Google Analytics accounts are: {{ids}}
{% endif %}

Now you just need to start the server and visit /oauth2 to start the authentication process. If everything worked correctly, you should see the list of ids of your Google Analytics account.

Note that once you have the Google Analytics service object you can retrieve all the data you want from your accounts. To see more examples, you can check this post: Google Analytics API with Python.

The post Use the Google Analytics API with Django appeared first on Marina Mele's site.

How to correctly remove a Model or an App in Django

When you remove a Django Model or a full app in Django, you have to be careful in how you do it, and how the database will be affected.

Thus, in this short post I’ll explain what I do when deleting a model or an app.

Remove the Models from models.py

Weather you’re removing a single model or a full app, you should first remove the desired models from the models.py file.

Moreover, you have to make sure that no other file imports these models or uses them (admin.py, views.py, etc).

Next, run South or migrate your database to properly delete these models from your database.

If you are using South, you should run something like

$ python manage.py schemamigration --auto your-app
$ python manage.py migrate your-app

changing your-app by the name of the app to migrate (where your deleted models were).

Remove an App from your Django project

Next, if you are removing an entire app, you can now remove all the files in that app (models.py, views.py, etc). Again, make sure that the other apps don’t make an import from these files.

And finally, remove this app from your Installed Apps in the settings.py file.

Remove the remnant ContentTypes

But after doing all this, you will still have remnants of your models in the ContentType table, like permissions related to the deleted models.

One way to see this is when creating a Group in the Auth app using the admin. When you have to choose the permissions of the group, the box will show permissions that refer to the deleted models.

To clean these tables, we will create the file clean_model_app_remove.py and write the following:

from django.contrib.contenttypes.models import ContentType

# List of deleted apps
DEL_APPS = ["app-you-deleted", "second-app-deleted"]
# List of deleted models (that are not in the app deleted) In lowercase!
DEL_MODELS = ["model-you-deleted", "second-model-deleted"]

ct = ContentType.objects.all().order_by("app_label", "model")

for c in ct:
    if (c.app_label in DEL_APPS) or (c.model in DEL_MODELS):
        print "Deleting Content Type %s %s" % (c.app_label, c.model)
        c.delete()

where you should write in DEL_APPS the apps that you deleted, and in DEL_MODELS the models that you delete and that do not belong to any of the deleted apps. Moreover, note that the models have to be written in lowercase.

Finally, we just need to run our script with

$ python manage.py shell
>>>execfile("clean_model_app_remove.py")

Great! Now we cleanly removed our apps/models!

The post How to correctly remove a Model or an App in Django appeared first on Marina Mele's site.

How to Use Chart.js with Django

Chart.js is a cool open source JavaScript library that helps you render HTML5 charts. It is responsive and counts with 8 different chart types.

In this tutorial we are going to explore a little bit of how to make Django talk with Chart.js and render some simple charts based on data extracted from our models.

Installation

For this tutorial all you are going to do is add the Chart.js lib to your HTML page:

<script src="https://cdn.jsdelivr.net/npm/chart.js@2.9.3/dist/Chart.min.js"></script>

You can download it from Chart.js official website and use it locally, or you can use it from a CDN using the URL above.

Example Scenario

I’m going to use the same example I used for the tutorial How to Create Group By Queries With Django ORM which is a good complement to this tutorial because actually the tricky part of working with charts is to transform the data so it can fit in a bar chart / line chart / etc.

We are going to use the two models below, Country and City:

class Country(models.Model):
    name = models.CharField(max_length=30)

class City(models.Model):
    name = models.CharField(max_length=30)
    country = models.ForeignKey(Country, on_delete=models.CASCADE)
    population = models.PositiveIntegerField()

And the raw data stored in the database:

Example 1: Pie Chart

For the first example we are only going to retrieve the top 5 most populous cities and render it as a pie chart. In this strategy we are going to return the chart data as part of the view context and inject the results in the JavaScript code using the Django Template language.

views.py

from django.shortcuts import render
from mysite.core.models import City

def pie_chart(request):
    labels = []
    data = []

    queryset = City.objects.order_by('-population')[:5]
    for city in queryset:
        labels.append(city.name)
        data.append(city.population)

    return render(request, 'pie_chart.html', {
        'labels': labels,
        'data': data,
    })

Basically in the view above we are iterating through the City queryset and building a list of labels and a list of data. Here in this case the data is the population count saved in the City model.

For the urls.py just a simple routing:

urls.py

from django.urls import path
from mysite.core import views

urlpatterns = [
    path('pie-chart/', views.pie_chart, name='pie-chart'),
]

Now the template. I got a basic snippet from the Chart.js Pie Chart Documentation.

pie_chart.html

{% extends 'base.html' %}

{% block content %}
  <div id="container" style="width: 75%;">
    <canvas id="pie-chart"></canvas>
  </div>

  <script src="https://cdn.jsdelivr.net/npm/chart.js@2.9.3/dist/Chart.min.js"></script>
  <script>

    var config = {
      type: 'pie',
      data: {
        datasets: [{
          data: {{ data|safe }},
          backgroundColor: [
            '#696969', '#808080', '#A9A9A9', '#C0C0C0', '#D3D3D3'
          ],
          label: 'Population'
        }],
        labels: {{ labels|safe }}
      },
      options: {
        responsive: true
      }
    };

    window.onload = function() {
      var ctx = document.getElementById('pie-chart').getContext('2d');
      window.myPie = new Chart(ctx, config);
    };

  </script>

{% endblock %}

In the example above the base.html template is not important but you can see it in the code example I shared in the end of this post.

This strategy is not ideal but works fine. The bad thing is that we are using the Django Template Language to interfere with the JavaScript logic. When we put {{ data|safe}} we are injecting a variable that came from the server directly in the JavaScript code.

The code above looks like this:

Example 2: Bar Chart with Ajax

As the title says, we are now going to render a bar chart using an async call.

views.py

from django.shortcuts import render
from django.db.models import Sum
from django.http import JsonResponse
from mysite.core.models import City

def home(request):
    return render(request, 'home.html')

def population_chart(request):
    labels = []
    data = []

    queryset = City.objects.values('country__name').annotate(country_population=Sum('population')).order_by('-country_population')
    for entry in queryset:
        labels.append(entry['country__name'])
        data.append(entry['country_population'])
    
    return JsonResponse(data={
        'labels': labels,
        'data': data,
    })

So here we are using two views. The home view would be the main page where the chart would be loaded at. The other view population_chart would be the one with the sole responsibility to aggregate the data the return a JSON response with the labels and data.

If you are wondering about what this queryset is doing, it is grouping the cities by the country and aggregating the total population of each country. The result is going to be a list of country + total population. To learn more about this kind of query have a look on this post: How to Create Group By Queries With Django ORM

urls.py

from django.urls import path
from mysite.core import views

urlpatterns = [
    path('', views.home, name='home'),
    path('population-chart/', views.population_chart, name='population-chart'),
]

home.html

{% extends 'base.html' %}

{% block content %}

  <div id="container" style="width: 75%;">
    <canvas id="population-chart" data-url="{% url 'population-chart' %}"></canvas>
  </div>

  <script src="https://code.jquery.com/jquery-3.4.1.min.js"></script>
  <script src="https://cdn.jsdelivr.net/npm/chart.js@2.9.3/dist/Chart.min.js"></script>
  <script>

    $(function () {

      var $populationChart = $("#population-chart");
      $.ajax({
        url: $populationChart.data("url"),
        success: function (data) {

          var ctx = $populationChart[0].getContext("2d");

          new Chart(ctx, {
            type: 'bar',
            data: {
              labels: data.labels,
              datasets: [{
                label: 'Population',
                backgroundColor: 'blue',
                data: data.data
              }]          
            },
            options: {
              responsive: true,
              legend: {
                position: 'top',
              },
              title: {
                display: true,
                text: 'Population Bar Chart'
              }
            }
          });

        }
      });

    });

  </script>

{% endblock %}

Now we have a better separation of concerns. Looking at the chart container:

<canvas id="population-chart" data-url="{% url 'population-chart' %}"></canvas>

We added a reference to the URL that holds the chart rendering logic. Later on we are using it to execute the Ajax call.

var $populationChart = $("#population-chart");
$.ajax({
  url: $populationChart.data("url"),
  success: function (data) {
    // ...
  }
});

Inside the success callback we then finally execute the Chart.js related code using the JsonResponse data.

Conclusions

I hope this tutorial helped you to get started with working with charts using Chart.js. I published another tutorial on the same subject a while ago but using the Highcharts library. The approach is pretty much the same: How to Integrate Highcharts.js with Django.

If you want to grab the code I used in this tutorial you can find it here: github.com/sibtc/django-chartjs-example.

My 5 Favorite Talks at DjangoCon Europe 2019

This year DjangoCon Europe happened in Copenhagen, Denmark, at a very creative and special place AFUK - Academy for Untamed Creativity. Surrounded by artistic souls, we learned more about web technologies, got to know each other at ecologic reusable disposable cups of coffee, enjoyed delicious authentic food, and socialized with Django-branded beverages. As always, there was also a party with people drinking IPA (not to be confused with API). And at the weekend Django developers were solving bugs for Django and related open-source software at the coding sprints.

Here I would like to present you with the top 5 talks that I liked most of all.

Django and Web Security Headers

Adam Johnson (@AdamChainz) was talking about special response headers that tell browsers to treat data of the website more securely and which Django settings are responsible for those headers.

Summary by rixx

Docs or it didn't Happen!

Mikey Ariel (@ThatDocsLady) was talking about the necessity of documentation and what to write there.

Summary by rixx

Pushing the ORM to its Limits

Sigurd Ljødal (@sigurdlj) was talking about advanced Django ORM use cases.

Summary by rixx

Logging Rethought 2: The Actions of Frank Taylor Jr.

Markus Holtermann (@m_holtermann) was talking about logging structured data to log files instead of the traditional plain text messages.

Summary by rixx

Maintaining a Django Codebase after 10k Commits

Joachim Jablon (@Ewjoachim) and Stéphane Angel (@twidi) were talking about the best practices developing large-scale Django projects.

Summary by rixx

Honorable Mentions

• "Feeding the Pony: Contributing back to Django & How to make that work for you" by Carlton Gibson.
• "Sketching out a Django redesign" by Tom Christie.
• "Reduce, Reuse, Recycle - Persisting WebSocket connections with SharedWorkers" by Aaronn Bassett.
• "Maps with GeoDjango, PostGIS and Leaflet" by Paolo Melchiorre.
• "Here Come The Robots - Django and Machine Learning" by Tom Dyson.

More Information

You can see all talk descriptions, video records and some slides at the official conference website.

Amazing conference photos were taken by Bartek Pawlik.

I was also really astonished how effective were the talk summaries written by rixx almost in real time.

Cover photo by Bartek Pawlik.

QuerySet Filters on Many-to-many Relations

Django ORM (Object-relational mapping) makes querying the database so intuitive, that at some point you might forget that SQL is being used in the background.

This year at the DjangoCon Europe Katie McLaughlin was giving a talk and mentioned one thing that affects the SQL query generated by Django ORM, depending on how you call the QuerySet or manager methods. This particularity is especially relevant when you are creating your QuerySets dynamically. Here it is. When you have a many-to-many relationship, and you try to filter objects by the fields of the related model, every new filter() method of a QuerySet creates a new INNER JOIN clause. I won't discuss whether that's a Django bug or a feature, but these are my observations about it.

The Books and Authors Example

Let's create an app with books and authors, where each book can be written by multiple authors.

# -*- coding: UTF-8 -*-
from __future__ import unicode_literals

from django.db import models
from django.utils.translation import ugettext_lazy as _
from django.utils.encoding import python_2_unicode_compatible


@python_2_unicode_compatible
class Author(models.Model):
    first_name = models.CharField(_("First name"), max_length=200)
    last_name = models.CharField(_("Last name"), max_length=200)
    author_name = models.CharField(_("Author name"), max_length=200)

    class Meta:
        verbose_name = _("Author")
        verbose_name_plural = _("Authors")
        ordering = ("author_name",)

    def __str__(self):
        return self.author_name


@python_2_unicode_compatible
class Book(models.Model):
    title = models.CharField(_("Title"), max_length=200)
    authors = models.ManyToManyField(Author, verbose_name=_("Authors"))
    publishing_date = models.DateField(_("Publishing date"), blank=True, null=True)

    class Meta:
        verbose_name = _("Book")
        verbose_name_plural = _("Books")
        ordering = ("title",)

    def __str__(self):
        return self.title

The similar app with sample data can be found in this repository.

Inefficient Filter

With the above models, you could define the following QuerySet to select books which author is me, Aidas Bendoraitis.

queryset = Book.objects.filter(
    authors__first_name='Aidas',
).filter(
    authors__last_name='Bendoraitis',
)

We can check what SQL query it would generate with str(queryset.query) (or queryset.query.__str__()).

The output would be something like this:

SELECT `libraryapp_book`.`id`, `libraryapp_book`.`title`, `libraryapp_book`.`publishing_date`
FROM `libraryapp_book`
INNER JOIN `libraryapp_book_authors` ON ( `libraryapp_book`.`id` = `libraryapp_book_authors`.`book_id` )
INNER JOIN `libraryapp_author` ON ( `libraryapp_book_authors`.`author_id` = `libraryapp_author`.`id` )
INNER JOIN `libraryapp_book_authors` T4 ON ( `libraryapp_book`.`id` = T4.`book_id` )
INNER JOIN `libraryapp_author` T5 ON ( T4.`author_id` = T5.`id` )
WHERE (`libraryapp_author`.`first_name` = 'Aidas' AND T5.`last_name` = 'Bendoraitis')
ORDER BY `libraryapp_book`.`title` ASC;

Did you notice, that the database table libraryapp_author was attached through the libraryapp_book_authors table to the libraryapp_book table TWICE where just ONCE would be enough?

Efficient Filter

On the other hand, if you are defining query expressions in the same filter() method like this:

queryset = Book.objects.filter(
    authors__first_name='Aidas',
    authors__last_name='Bendoraitis',
)

The generated SQL query will be much shorter and (theoretically) would perform faster:

SELECT `libraryapp_book`.`id`, `libraryapp_book`.`title`, `libraryapp_book`.`publishing_date`
FROM `libraryapp_book`
INNER JOIN `libraryapp_book_authors` ON ( `libraryapp_book`.`id` = `libraryapp_book_authors`.`book_id` )
INNER JOIN `libraryapp_author` ON ( `libraryapp_book_authors`.`author_id` = `libraryapp_author`.`id` )
WHERE (`libraryapp_author`.`first_name` = 'Aidas' AND `libraryapp_author`.`last_name` = 'Bendoraitis')
ORDER BY `libraryapp_book`.`title` ASC;

The same SQL query can be achieved using the Q() objects:

queryset = Book.objects.filter(
    models.Q(authors__first_name='Aidas') &
    models.Q(authors__last_name='Bendoraitis')
)

The Q() objects add a lot of flexibility to filters allowing to OR, AND, and negate query expressions.

Dynamic Filtering

So to have faster performance, when creating QuerySets dynamically, DON'T use filter() multiple times:

queryset = Book.objects.all()
if first_name:
    queryset = queryset.filter(
        authors__first_name=first_name,
    )
if last_name:
    queryset = queryset.filter(
        authors__last_name=last_name,
    )

DO this instead:

filters = models.Q()
if first_name:
    filters &= models.Q(
        authors__first_name=first_name,
    )
if last_name:
    filters &= models.Q(
        authors__last_name=last_name,
    )
queryset = Book.objects.filter(filters)

Here the empty Q() doesn't have any impact for the generated SQL query, so you don't need the complexity of creating a list of filters and then joining all of them with the bitwise AND operator, like this:

import operator
from django.utils.six.moves import reduce

filters = []
if first_name:
    filters.append(models.Q(
        authors__first_name=first_name,
    ))
if last_name:
    filters.append(models.Q(
        authors__last_name=last_name,
    ))
queryset = Book.objects.filter(reduce(operator.iand, filters))

Profiling

In DEBUG mode, you can check how long the previously executed SQL queries took by checking django.db.connection.queries:

>>> from django.db import connection
>>> connection.queries
[{'sql': 'SELECT …', 'time': '0.001'}, {'sql': 'SELECT …', 'time': '0.004'}]

The Takeaways

• When querying many-to-many relationships, avoid using multiple filter() methods, make use of Q() objects instead.
• You can check the SQL query of a QuerySet with str(queryset.query).
• Check the performance of recently executed SQL queries with django.db.connection.queries.
• With small datasets, the performance difference is not so obvious. For your specific cases you should do the benchmarks yourself.

Cover photo by Tobias Fischer.

Recap of DjangoCon Europe 2017

"DjangoCon, why is everybody wearing this t-shirt?" wondered the security guys in the airport of Florence, Italy, in the beginning of April. The reason for that was DjangoCon Europe 2017 happening there for a week, full of interesting talks in an exciting location.

What I liked, was that the conference was not only about technical novelties in Django world, but also about human issues that programmers deal with in everyday life.

Interesting Non-tech Topics

According to a manifest, the conference had a goal to strengthen the Django community and to shape responsible attitude towards the works done with Django.

Healthy and Successful Community

We have to build stronger communities including everyone who wants to participate without discrimination. Although, at first, it might be difficult as people have biases, i.e. prejudices for or against one person or group; by being emphatic we can accept and include everyone no matter what is their gender identity or expression, sexual orientation, ethnicity, race, neurodiversity, age, religion, disabilities, geographical location, food diversities, body size, or family status.

Valuing diversity and individual differences is the key for a healthy, positive and successful community, that empowers its members and helps them grow stronger and happier.

Responsibility for How We Use Technology

Information technology companies (Apple, Alphabet, Microsoft, Amazon, and Facebook) are among the most traded companies in the world. IT connects people and their things, automates processes, stores and treats historical data. Usually you don't need many physical resources to start an IT business. Software developers have a power to shape the future, but should use this power responsibly:

With this, great responsibility is upon us: to make the future a better place, to make the future more evenly distributed, across gender gaps and discriminations, breaking economical, political and geographical barriers.

Business

• When creating an online business, it is important to think about the business value that your product will give to people and the way you will make money with it. Don't make assumptions without talking to your customers.
• When choosing employees for your company, give them freedom how to prove their knowledge: by a quiz, or whiteboard interview, or a take-home programming task. Different people have different ways how to best represent their skills.
• Launch as early as possible. Track the usage statistics with Google Analytics or other analytics service. Collect emails for the mailing list. Write about your product in a blog and personalized emails.
• Django is an open-source project based on the hard work of many professionals, and if you gain any commercial value of it and appreciate the framework, you should donate to the Django Software Foundation.

Interesting Tech Topics

From the technical point of view, I liked several ideas mentioned in the conference:

Automate your processes

• For starting new projects, you can have boilerplates with the mostly used functionalities already prepared. Django management command startproject has a parameter --template for that where you can pass a URL to a zip file.
• Developers should have troubleshooting checklists for debugging, just like airplane pilots.
• There are several types of tests. Unit tests check the functionality of individual functions or methods. Integration tests check how different units work together. The functional tests check how the processes of business requirements work from start to end. Finally, there is manual testing requiring people to click through the website and fill in the forms. Some tests like the ones involving third-party authentication or mobile phones, are hardly possible to automate. Anyway, manual testing is the most expensive in time and resources (besides it being boring for the tester), functional tests go after them, then integration tests, and lastly unit tests. Although automatic testing adds up to the development time, in the long run it makes the systems more stable and error proof.

What about Django

• You can extend the Django ORM with custom lookups, transactions, and filtered prefetchings, to make your QuerySets more readable and more capable.
• Once again, PostgreSQL has more capabilities than MySQL and is more stable. Use EXPLAIN ANALYZE ... SQL command to find the bottlenecks of your database queries. You can usually fix them by adding indexes.
• You can have custom indexes for your database tables, to optimize the performance on PostgreSQL (or some other vendor) database.
• Django 1.11 is out and it's a long-term support version.

What about Third Party Django Packages

• After analyzing the 6 most popular model translation packages (parler, hvad, klingon, modeltranslation, nece, and i18nfield) from different angles (database support, integration in django admin and forms, performance, etc.), django-hvad seemed to be the winning approach.
• You can visually build static websites with very little coded configuration using django-cms and djangocms-cascade. The djangocms-cascade provides an alternative nested-plugins system for Django CMS.

What about Django Projects

• If you build web apps for developing countries, you have to keep these things in mind: people might be using cell phones instead of computers (you need responsive design with small or no images), Internet connectivity is slow and unstable (websites have to be fast and should preferably have offline versions), the users do not always understand English (the websites should be translated and adapted), and locations where people live do not always have street addresses.
• Some interesting use cases: tracking the health of the heart with Arduino and Django, providing weather data to the whole Europe using Django, managing a radio station in Birmingham using Django.

Thanks

Finally, thanks to the organizers for making this conference as great as it was. The city was beautiful, the food and coffee was delicious, the location for the talks was impressive. Looking forward to the next DjangoCon Europe!

Cover photo by Aidas Bendoraitis

Fresh Book for Django Developers

This week the post office delivered a package that made me very satisfied. It was a box with three paper versions of my "Web Development with Django Cookbook - Second Edition". The book was published at the end of January after months of hard, but fulfilling work in the late evenings and at weekends.

The first Django Cookbook was dealing with Django 1.6. Unfortunately, the support for that version is over. So it made sense to write an update for a newer Django version. The second edition was adapted for Django 1.8 which has a long-term support until April 2018 or later. This edition introduces new features added to Django 1.7 and Django 1.8, such as database migrations, QuerySet expressions, or System Check Framework. Most concepts in this new book should also be working with Django 1.9.

My top 5 favourite new recipes are these:

• Configuring settings for development, testing, staging, and production environments
• Using database query expressions
• Implementing a multilingual search with Haystack
• Testing pages with Selenium
• Releasing a reusable Django app

The book is worth reading for any Django developer, but will be best understood by those who already know the basics of web development with Django. You can learn more about the book and buy it at the Packt website or Amazon.

I thank the Packt Publishing very much for long cooperation in the development of this book. I am especially thankful to acquisition editor Nadeem N. Bagban, content development editors Arwa Manasawala and Sumeet Sawant, and technical editor Bharat Patil. Also I am grateful for insightful feedback from the reviewer Jake Kronika.

What 5 recipes do you find the most useful?

How to Save Extra Data to a Django REST Framework Serializer

In this tutorial you are going to learn how to pass extra data to your serializer, before saving it to the database.

• Introduction
• Example Using APIView
• Example Using ViewSet

Introduction

When using regular Django forms, there is this common pattern where we save the form with commit=False and then pass some extra data to the instance before saving it to the database, like this:

form = InvoiceForm(request.POST)
if form.is_valid():
    invoice = form.save(commit=False)
    invoice.user = request.user
    invoice.save()

This is very useful because we can save the required information using only one database query and it also make it possible to handle not nullable columns that was not defined in the form.

To simulate this pattern using a Django REST Framework serializer you can do something like this:

serializer = InvoiceSerializer(data=request.data)
if serializer.is_valid():
    serializer.save(user=request.user)

You can also pass several parameters at once:

serializer = InvoiceSerializer(data=request.data)
if serializer.is_valid():
    serializer.save(user=request.user, date=timezone.now(), status='sent')

Example Using APIView

In this example I created an app named core.

models.py

from django.contrib.auth.models import User
from django.db import models

class Invoice(models.Model):
    SENT = 1
    PAID = 2
    VOID = 3
    STATUS_CHOICES = (
        (SENT, 'sent'),
        (PAID, 'paid'),
        (VOID, 'void'),
    )

    user = models.ForeignKey(User, on_delete=models.CASCADE, related_name='invoices')
    number = models.CharField(max_length=30)
    date = models.DateTimeField(auto_now_add=True)
    status = models.PositiveSmallIntegerField(choices=STATUS_CHOICES)
    amount = models.DecimalField(max_digits=10, decimal_places=2)

serializers.py

from rest_framework import serializers
from core.models import Invoice

class InvoiceSerializer(serializers.ModelSerializer):
    class Meta:
        model = Invoice
        fields = ('number', 'amount')

views.py

from rest_framework import status
from rest_framework.response import Response
from rest_framework.views import APIView
from core.models import Invoice
from core.serializers import InvoiceSerializer

class InvoiceAPIView(APIView):
    def post(self, request):
        serializer = InvoiceSerializer(data=request.data)
        serializer.is_valid(raise_exception=True)
        serializer.save(user=request.user, status=Invoice.SENT)
        return Response(status=status.HTTP_201_CREATED)

Example Using ViewSet

Very similar example, using the same models.py and serializers.py as in the previous example.

views.py

from rest_framework.viewsets import ModelViewSet
from core.models import Invoice
from core.serializers import InvoiceSerializer

class InvoiceViewSet(ModelViewSet):
    queryset = Invoice.objects.all()
    serializer_class = InvoiceSerializer

    def perform_create(self, serializer):
        serializer.save(user=self.request.user, status=Invoice.SENT)

How to Use Date Picker with Django

In this tutorial we are going to explore three date/datetime pickers options that you can easily use in a Django project. We are going to explore how to do it manually first, then how to set up a custom widget and finally how to use a third-party Django app with support to datetime pickers.

• Introduction
• Tempus Dominus Bootstrap 4
  • Direct Usage
  • Custom Widget
• XDSoft DateTimePicker
  • Direct Usage
  • Custom Widget
• Fengyuan Chen’s Datepicker
  • Direct Usage
  • Custom Widget
• Conclusions

Introduction

The implementation of a date picker is mostly done on the front-end.

The key part of the implementation is to assure Django will receive the date input value in the correct format, and also that Django will be able to reproduce the format when rendering a form with initial data.

We can also use custom widgets to provide a deeper integration between the front-end and back-end and also to promote better reuse throughout a project.

In the next sections we are going to explore following date pickers:

Tempus Dominus Bootstrap 4 Docs Source

XDSoft DateTimePicker Docs Source

Fengyuan Chen’s Datepicker Docs Source

Tempus Dominus Bootstrap 4

Docs Source

This is a great JavaScript library and it integrate well with Bootstrap 4. The downside is that it requires moment.js and sort of need Font-Awesome for the icons.

It only make sense to use this library with you are already using Bootstrap 4 + jQuery, otherwise the list of CSS and JS may look a little bit overwhelming.

To install it you can use their CDN or download the latest release from their GitHub Releases page.

If you downloaded the code from the releases page, grab the processed code from the build/ folder.

Below, a static HTML example of the datepicker:

<!doctype html>
<html lang="en">
  <head>
    <meta charset="utf-8">
    <meta name="viewport" content="width=device-width, initial-scale=1, shrink-to-fit=no">
    <title>Static Example</title>

    <!-- Bootstrap 4 -->
    <link rel="stylesheet" href="https://stackpath.bootstrapcdn.com/bootstrap/4.2.1/css/bootstrap.min.css" integrity="sha384-GJzZqFGwb1QTTN6wy59ffF1BuGJpLSa9DkKMp0DgiMDm4iYMj70gZWKYbI706tWS" crossorigin="anonymous">
    <script src="https://code.jquery.com/jquery-3.3.1.slim.min.js" integrity="sha384-q8i/X+965DzO0rT7abK41JStQIAqVgRVzpbzo5smXKp4YfRvH+8abtTE1Pi6jizo" crossorigin="anonymous"></script>
    <script src="https://cdnjs.cloudflare.com/ajax/libs/popper.js/1.14.6/umd/popper.min.js" integrity="sha384-wHAiFfRlMFy6i5SRaxvfOCifBUQy1xHdJ/yoi7FRNXMRBu5WHdZYu1hA6ZOblgut" crossorigin="anonymous"></script>
    <script src="https://stackpath.bootstrapcdn.com/bootstrap/4.2.1/js/bootstrap.min.js" integrity="sha384-B0UglyR+jN6CkvvICOB2joaf5I4l3gm9GU6Hc1og6Ls7i6U/mkkaduKaBhlAXv9k" crossorigin="anonymous"></script>

    <!-- Font Awesome -->
    <link href="https://stackpath.bootstrapcdn.com/font-awesome/4.7.0/css/font-awesome.min.css" rel="stylesheet" integrity="sha384-wvfXpqpZZVQGK6TAh5PVlGOfQNHSoD2xbE+QkPxCAFlNEevoEH3Sl0sibVcOQVnN" crossorigin="anonymous">

    <!-- Moment.js -->
    <script src="https://cdnjs.cloudflare.com/ajax/libs/moment.js/2.23.0/moment.min.js" integrity="sha256-VBLiveTKyUZMEzJd6z2mhfxIqz3ZATCuVMawPZGzIfA=" crossorigin="anonymous"></script>

    <!-- Tempus Dominus Bootstrap 4 -->
    <link rel="stylesheet" href="https://cdnjs.cloudflare.com/ajax/libs/tempusdominus-bootstrap-4/5.1.2/css/tempusdominus-bootstrap-4.min.css" integrity="sha256-XPTBwC3SBoWHSmKasAk01c08M6sIA5gF5+sRxqak2Qs=" crossorigin="anonymous" />
    <script src="https://cdnjs.cloudflare.com/ajax/libs/tempusdominus-bootstrap-4/5.1.2/js/tempusdominus-bootstrap-4.min.js" integrity="sha256-z0oKYg6xiLq3yJGsp/LsY9XykbweQlHl42jHv2XTBz4=" crossorigin="anonymous"></script>

  </head>
  <body>

    <div class="input-group date" id="datetimepicker1" data-target-input="nearest">
      <input type="text" class="form-control datetimepicker-input" data-target="#datetimepicker1"/>
      <div class="input-group-append" data-target="#datetimepicker1" data-toggle="datetimepicker">
        <div class="input-group-text"><i class="fa fa-calendar"></i></div>
      </div>
    </div>

    <script>
      $(function () {
        $("#datetimepicker1").datetimepicker();
      });
    </script>

  </body>
</html>

Direct Usage

The challenge now is to have this input snippet integrated with a Django form.

forms.py

from django import forms

class DateForm(forms.Form):
    date = forms.DateTimeField(
        input_formats=['%d/%m/%Y %H:%M'],
        widget=forms.DateTimeInput(attrs={
            'class': 'form-control datetimepicker-input',
            'data-target': '#datetimepicker1'
        })
    )

template

<div class="input-group date" id="datetimepicker1" data-target-input="nearest">
  {{ form.date }}
  <div class="input-group-append" data-target="#datetimepicker1" data-toggle="datetimepicker">
    <div class="input-group-text"><i class="fa fa-calendar"></i></div>
  </div>
</div>

<script>
  $(function () {
    $("#datetimepicker1").datetimepicker({
      format: 'DD/MM/YYYY HH:mm',
    });
  });
</script>

The script tag can be placed anywhere because the snippet $(function () { ... }); will run the datetimepicker initialization when the page is ready. The only requirement is that this script tag is placed after the jQuery script tag.

Custom Widget

You can create the widget in any app you want, here I’m going to consider we have a Django app named core.

core/widgets.py

from django.forms import DateTimeInput

class BootstrapDateTimePickerInput(DateTimeInput):
    template_name = 'widgets/bootstrap_datetimepicker.html'

    def get_context(self, name, value, attrs):
        datetimepicker_id = 'datetimepicker_{name}'.format(name=name)
        if attrs is None:
            attrs = dict()
        attrs['data-target'] = '#{id}'.format(id=datetimepicker_id)
        attrs['class'] = 'form-control datetimepicker-input'
        context = super().get_context(name, value, attrs)
        context['widget']['datetimepicker_id'] = datetimepicker_id
        return context

In the implementation above we generate a unique ID datetimepicker_id and also include it in the widget context.

Then the front-end implementation is done inside the widget HTML snippet.

widgets/bootstrap_datetimepicker.html

<div class="input-group date" id="{{ widget.datetimepicker_id }}" data-target-input="nearest">
  {% include "django/forms/widgets/input.html" %}
  <div class="input-group-append" data-target="#{{ widget.datetimepicker_id }}" data-toggle="datetimepicker">
    <div class="input-group-text"><i class="fa fa-calendar"></i></div>
  </div>
</div>

<script>
  $(function () {
    $("#{{ widget.datetimepicker_id }}").datetimepicker({
      format: 'DD/MM/YYYY HH:mm',
    });
  });
</script>

Note how we make use of the built-in django/forms/widgets/input.html template.

Now the usage:

core/forms.py

from .widgets import BootstrapDateTimePickerInput

class DateForm(forms.Form):
    date = forms.DateTimeField(
        input_formats=['%d/%m/%Y %H:%M'], 
        widget=BootstrapDateTimePickerInput()
    )

Now simply render the field:

template

{{ form.date }}

The good thing about having the widget is that your form could have several date fields using the widget and you could simply render the whole form like:

<form method="post">
  {% csrf_token %}
  {{ form.as_p }}
  <input type="submit" value="Submit">
</form>

XDSoft DateTimePicker

Docs Source

The XDSoft DateTimePicker is a very versatile date picker and doesn’t rely on moment.js or Bootstrap, although it looks good in a Bootstrap website.

It is easy to use and it is very straightforward.

You can download the source from GitHub releases page.

Below, a static example so you can see the minimum requirements and how all the pieces come together:

<!doctype html>
<html lang="en">
<head>
  <meta charset="utf-8">
  <meta name="viewport" content="width=device-width, initial-scale=1, shrink-to-fit=no">
  <title>Static Example</title>

  <!-- jQuery -->
  <script src="https://code.jquery.com/jquery-3.3.1.slim.min.js" integrity="sha384-q8i/X+965DzO0rT7abK41JStQIAqVgRVzpbzo5smXKp4YfRvH+8abtTE1Pi6jizo" crossorigin="anonymous"></script>

  <!-- XDSoft DateTimePicker -->
  <link rel="stylesheet" href="https://cdnjs.cloudflare.com/ajax/libs/jquery-datetimepicker/2.5.20/jquery.datetimepicker.min.css" integrity="sha256-DOS9W6NR+NFe1fUhEE0PGKY/fubbUCnOfTje2JMDw3Y=" crossorigin="anonymous" />
  <script src="https://cdnjs.cloudflare.com/ajax/libs/jquery-datetimepicker/2.5.20/jquery.datetimepicker.full.min.js" integrity="sha256-FEqEelWI3WouFOo2VWP/uJfs1y8KJ++FLh2Lbqc8SJk=" crossorigin="anonymous"></script>
</head>
<body>

  <input id="datetimepicker" type="text">

  <script>
    $(function () {
      $("#datetimepicker").datetimepicker();
    });
  </script>

</body>
</html>

Direct Usage

A basic integration with Django would look like this:

forms.py

from django import forms

class DateForm(forms.Form):
    date = forms.DateTimeField(input_formats=['%d/%m/%Y %H:%M'])

Simple form, default widget, nothing special.

Now using it on the template:

template

{{ form.date }}

<script>
  $(function () {
    $("#id_date").datetimepicker({
      format: 'd/m/Y H:i',
    });
  });
</script>

The id_date is the default ID Django generates for the form fields (id_ + name).

Custom Widget

core/widgets.py

from django.forms import DateTimeInput

class XDSoftDateTimePickerInput(DateTimeInput):
    template_name = 'widgets/xdsoft_datetimepicker.html'

widgets/xdsoft_datetimepicker.html

{% include "django/forms/widgets/input.html" %}

<script>
  $(function () {
    $("input[name='{{ widget.name }}']").datetimepicker({
      format: 'd/m/Y H:i',
    });
  });
</script>

To have a more generic implementation, this time we are selecting the field to initialize the component using its name instead of its id, should the user change the id prefix.

Now the usage:

core/forms.py

from django import forms
from .widgets import XDSoftDateTimePickerInput

class DateForm(forms.Form):
    date = forms.DateTimeField(
        input_formats=['%d/%m/%Y %H:%M'], 
        widget=XDSoftDateTimePickerInput()
    )

template

{{ form.date }}

Fengyuan Chen’s Datepicker

Docs Source

This is a very beautiful and minimalist date picker. Unfortunately there is no time support. But if you only need dates this is a great choice.

To install this datepicker you can either use their CDN or download the sources from their GitHub releases page. Please note that they do not provide a compiled/processed JavaScript files. But you can download those to your local machine using the CDN.

<!doctype html>
<html lang="en">
<head>
  <meta charset="utf-8">
  <meta name="viewport" content="width=device-width, initial-scale=1, shrink-to-fit=no">
  <title>Static Example</title>
  <style>body {font-family: Arial, sans-serif;}</style>
  
  <!-- jQuery -->
  <script src="https://code.jquery.com/jquery-3.3.1.slim.min.js" integrity="sha384-q8i/X+965DzO0rT7abK41JStQIAqVgRVzpbzo5smXKp4YfRvH+8abtTE1Pi6jizo" crossorigin="anonymous"></script>

  <!-- Fengyuan Chen's Datepicker -->
  <link rel="stylesheet" href="https://cdnjs.cloudflare.com/ajax/libs/datepicker/0.6.5/datepicker.min.css" integrity="sha256-b88RdwbRJEzRx95nCuuva+hO5ExvXXnpX+78h8DjyOE=" crossorigin="anonymous" />
  <script src="https://cdnjs.cloudflare.com/ajax/libs/datepicker/0.6.5/datepicker.min.js" integrity="sha256-/7FLTdzP6CfC1VBAj/rsp3Rinuuu9leMRGd354hvk0k=" crossorigin="anonymous"></script>
</head>
<body>

  <input id="datepicker">

  <script>
    $(function () {
      $("#datepicker").datepicker();
    });
  </script>

</body>
</html>

Direct Usage

A basic integration with Django (note that we are now using DateField instead of DateTimeField):

forms.py

from django import forms

class DateForm(forms.Form):
    date = forms.DateTimeField(input_formats=['%d/%m/%Y %H:%M'])

template

{{ form.date }}

<script>
  $(function () {
    $("#id_date").datepicker({
      format:'dd/mm/yyyy',
    });
  });
</script>

Custom Widget

core/widgets.py

from django.forms import DateInput

class FengyuanChenDatePickerInput(DateInput):
    template_name = 'widgets/fengyuanchen_datepicker.html'