Lateo.net - Flux RSS en pagaille (pour en ajouter : @ moi)

🔒
❌ À propos de FreshRSS
Il y a de nouveaux articles disponibles, cliquez pour rafraîchir la page.
À partir d’avant-hierDjangoTricks

Django Project on NGINX Unit

Django Project on NGINX Unit

Recently, I learned about the NGINX Unit and decided to try it on my DjangoTricks website. Unit is a web server developed by people from NGINX, with pluggable support for Python (WSGI and ASGI), Ruby, Node.js, PHP, and a few other languages. I wanted to see whether it's really easy to set it up, have it locally on my Mac and the remote Ubuntu server, and try out the ASGI features of Django, allowing real-time communication. Also, I wanted to see whether Django is faster with Unit than with NGINX and Gunicorn. This article is about my findings.

My observations

Unit service uses HTTP requests to read and update its configuration. The configuration is a single JSON file that you can upload to the Unit service via a command line from the same computer or modify its values by keys in the JSON structure.

Normally, the docs suggest using the curl command to update the configuration. However, as I am using Ansible to deploy my Django websites, I wanted to create a script I could later copy to other projects. I used Google Gemini to convert bash commands from the documentation to Ansible directives and corrected its mistakes.

The trickiest part for me was to figure out how to use Let's Encrypt certificates in the simplest way possible. The docs are extensive and comprehensible, but sometimes, they dig into technical details that are unnecessary for a common Django developer.

Also, it's worth mentioning that the Unit plugin version must match your Python version in the virtual environment. It was unexpected for me when Brew installed Python 3.12 with unit-python3 and then required my project to use Python 3.12 instead of Python 3.10 (which I used for the DjangoTricks website). So I had to recreate my virtual environment and probably will have problems later with pip-compile-multi when I prepare packages for the production server, still running Python 3.10.

Below are the instructions I used to set up the NGINX Unit with my existing DjangoTricks website on Ubuntu 22.04. For simplicity, I am writing plain Terminal commands instead of analogous Ansible directives.

1. Install Unit service to your server

Follow the installation instructions from documentation to install unit, unit-dev, unit-python3.10, and whatever other plugins you want. Make sure the service is running.

2. Prepare Let's Encrypt certificates

Create a temporary JSON configuration file /var/webapps/djangotricks/unit-config/unit-config-pre.json, which will allow Let's Encrypt certbot to access the .well-known directory for domain confirmation:

{
  "listeners": {
    "*:80": {
      "pass": "routes/acme"
    }
  },
  "routes": {
    "acme": [
      {
        "match": {
          "uri": "/.well-known/acme-challenge/*"
        },
        "action": {
          "share": "/var/www/letsencrypt"
        }
      }
    ]
  }
}

Install it to Unit:

$ curl -X PUT --data-binary @/var/webapps/djangotricks/unit-config/unit-config-pre.json \
--unix-socket /var/run/control.unit.sock http://localhost/config

If you make any mistakes in the configuration, it will be rejected with an error message and not executed.

Create Let's Encrypt certificates:

$ certbot certonly -n -w /var/www/letsencrypt -m hello@djangotricks.com \
--agree-tos --no-verify-ssl -d djangotricks.com -d www.djangotricks.com

Create a bundle that is required by the NGINX Unit:

cat /etc/letsencrypt/live/djangotricks.com/fullchain.pem \
/etc/letsencrypt/live/djangotricks.com/privkey.pem > \
/var/webapps/djangotricks/unit-config/bundle1.pem

Install certificate to NGINX Unit as certbot1:

curl -X PUT --data-binary @/var/webapps/djangotricks/unit-config/bundle1.pem \
--unix-socket /var/run/control.unit.sock http://localhost/certificates/certbot1

3. Install Django project configuration

Create a JSON configuration file /var/webapps/djangotricks/unit-config/unit-config.json which will use your SSL certificate and will serve your Django project:

{
  "listeners": {
    "*:80": {
      "pass": "routes/acme"
    },
    "*:443": {
      "pass": "routes/main",
      "tls": {
        "certificate": "certbot1"
      }
    }
  },
  "routes": {
    "main": [
      {
        "match": {
          "host": [
            "djangotricks.com",
            "www.djangotricks.com"
          ],
          "scheme": "https"
        },
        "action": {
          "pass": "applications/django"
        }
      },
      {
        "action": {
          "return": 444
        }
      }
    ],
    "acme": [
      {
        "match": {
          "host": [
            "djangotricks.com",
            "www.djangotricks.com"
          ],
          "uri": "/.well-known/acme-challenge/*"
        },
        "action": {
          "share": "/var/www/letsencrypt"
        }
      },
      {
        "action": {
          "return": 444
        }
      }
    ]
  },
  "applications": {
    "django": {
      "type": "python",
      "path": "/var/webapps/djangotricks/project/djangotricks",
      "home": "/var/webapps/djangotricks/venv/",
      "module": "djangotricks.wsgi",
      "environment": {
        "DJANGO_SETTINGS_MODULE": "djangotricks.settings.production"
      },
      "user": "djangotricks",
      "group": "users"
    }
  }
}

In this configuration, HTTP requests can only be used for certification validation, and HTTPS requests point to the Django project if the domain used is correct. In other cases, the status "444 - No Response" is returned. (It's for preventing access for hackers who point their domains to your IP address).

In the NGINX Unit, switching between WSGI and ASGI is literally a matter of changing one letter from "w" to "a" in the line about the Django application module, from:

"module": "djangotricks.wsgi",

to:

"module": "djangotricks.asgi",

I could have easily served the static files in this configuration here, too, but my STATIC_URL contains a dynamic part to force retrieval of new files from the server instead of the browser cache. So, I used WhiteNoise to serve the static files.

For redirection from djangotricks.com to www.djangotricks.com, I also chose to use PREPEND_WWW = True setting instead of Unit directives.

And here, finally, installing it to Unit (it will overwrite the previous configuration):

$ curl -X PUT --data-binary @/var/webapps/djangotricks/unit-config/unit-config.json \
--unix-socket /var/run/control.unit.sock http://localhost/config

How it performed

DjangoTricks is a pretty small website; therefore, I couldn't do extensive benchmarks, but I checked two cases: how a filtered list view performs with NGINX and Gunicorn vs. NGINX Unit, and how you can replace NGINX, Gunicorn, and Huey background tasks with ASGI requests using NGINX Unit.

First of all, the https://www.djangotricks.com/tricks/?categories=development&technologies=django-4-2 returned the HTML result on average in 139 ms on NGINX with Gunicorn, whereas it was on average 140 ms with NGINX Unit using WSGI and 149 ms with NGINX Unit using ASGI. So, the NGINX Unit with WSGI is 0.72% slower than NGINX with Gunicorn, and the NGINX Unit with ASGI is 7.19% slower than NGINX with Gunicorn.

However, when I checked https://www.djangotricks.com/detect-django-version/ how it performs with background tasks and continuous Ajax requests until the result is retrieved vs. asynchronous checking using ASGI, I went on average from 6.62 s to 0.75 s. Of course, it depends on the timeout of the continuous Ajax request, but generally, a real-time ASGI setup can improve the user experience significantly.

Final words

Although NGINX Unit with Python is slightly (unnoticeably) slower than NGINX with Gunicorn, it allows Django developers to use asynchronous requests and implement real-time user experience. Also, you could probably have a Django website and Matomo analytics or WordPress blog on the same server. The NGINX Unit configuration is relatively easy to understand, and you can script the process for reusability.


Cover Image by Volker Meyer.

Things I want to remember about SSH

SSH, short for Secure Shell, is a protocol for secure network communications. It is widely used for executing commands on remote servers, and for file uploads or downloads. If you are working with Django, use Git version control, or administrate servers, you surely are using SSH. In this post, I want to share some technical details about it.

Secure Shell is using private and public key pairs. You can either use automatically generated private and public keys combined with a password, or manually generated private and public keys. In the latter case, you need to keep your private key on your computer and upload the public key to the remote server.

Creating a pair of SSH keys manually

If you are using GitHub, Bitbucket, DigitalOcean, or some other service, you might have seen the possibility to upload public SSH keys for direct access to remote servers.

Here is how you usually create the SSH keys on the computer from which you want to establish a secure connection (your local machine or one of your servers that has access to other servers or services). In the Terminal you would execute these commands:

$ ssh-keygen
$ ssh-agent /usr/local/bin/bash
$ ssh-add ~/.ssh/id_rsa

The id_rsa is the name of the default SSH private key. The public key would be id_rsa.pub. And by default they both will be located under ~/.ssh/.

When running ssh-keygen you can choose different key names and even add a passphrase. For instance, you could have github_id_rsa and github_id_rsa.pub keys for communication with GitHub. My recommendation would be for each new service to create a new private-public key pair so that in case you need to transfer your computer's data to a different machine, you could selectively transfer the access to the remote servers.

Also, if you are not using the passphrase for the SSH key pair, I would recommend having your disk encrypted and a secure user password for your computer. If your laptop gets stolen, the thief wouldn't be able to get to your remote servers without knowing your computer's password.

Creating an access to a remote server by SSH key

In the case of GitHub, Bitbucket, and other online services with SSH communication, you usually have to copy the contents of the public key into a text field in a web form.

If you want to create a secure communication by manually generated private-public keys with a server where your Django project is deployed, you should append the contents of the public key to the ~/.ssh/authorized_keys file on the remote server.

To get the content of the public key in the Terminal, you can use:

$ cat ~/.ssh/id_rsa.pub

Then copy the output to the clipboard.

Or on macOS you can run pbcopy as follows:

$ pbcopy < ~/.ssh/id_rsa.pub 

To append the contents of the public key to the remote server, you can do this:

$  echo "...pasted public key...">>~/.ssh/authorized_keys

Creating authorization at a remote server by password

If you want to establish an SSH connection with a password and automatically generated private-public keys, you would need to edit /etc/ssh/sshd_config and ensure these two settings:

PasswordAuthentication yes
PermitEmptyPasswords no

After the change, you would restart the ssh server with the following command:

$ sudo service ssh restart

Also, make sure that the user you are connecting with has a password:

$ sudo passwd the_user

Connecting to a remote server

The default way to connect via SSH to a remote server with a password is executing the following in the Terminal:

$ ssh the_user@example.com

To connect with a private key, you would execute this:

$ ssh -i ~/.ssh/examplecom_id_rsa the_user@example.com

Next, let's see how we can simplify this using some local SSH configuration.

Configuring local SSH client

Edit ~/.ssh/config and add the following lines for each SSH connection that you want to define:

Host examplecom
    HostName example.com
    User the_user
    IdentityFile ~/.ssh/examplecom_id_rsa

If the domain of the website is not yet pointing to the IP address of the server, you can also connect by IP address:

Host examplecom
    HostName 1.2.3.4
    User the_user
    IdentityFile ~/.ssh/examplecom_id_rsa

The following allows you to login to your remote servers by manually generated private-public key with just these lines:

$ ssh examplecom

To request for password instead of using the manually generated keys, you would need to modify the snippet as follows:

Host examplecom
    HostName example.com
    User the_user
    PubkeyAuthentication=no

When you connect via SSH and wait don't type anything for 30 minutes or so, the connection gets lost. But you can require your client to connect to the server every 4 minutes or so by adding the following lines to the beginning of the ~/.ssh/config on your local computer:

Host *
    ServerAliveInterval 240

Uploading and downloading files using SSH connection

Typically, Secure Shell allows you to execute terminal commands on the remote server using bash, zsh, sh, or another shell. But very often, you also need to transfer files securely to and from the server. For that, you have these options: scp command, rsync command, or FTP client with SFTP support.

scp

The scp stands for Secure Copy.

This is how you would copy the secrets.json file from the remote server to your local development environment:

$ scp the_user@example.com:~/src/myproject/myproject/settings/secrets.json ./myproject/settings/secrets.json

Here is an example of the same, but with custom ~/.ssh/config configuration:

$ scp examplecom:~/src/myproject/myproject/settings/secrets.json ./myproject/settings/secrets.json

To copy the file from the local computer to the remote server, you would switch the places of source and target:

$ scp ./myproject/settings/secrets.json examplecom:~/src/myproject/myproject/settings/secrets.json

rsync

To synchronize directories on the server and locally, you can use the rsync command. This is how to do it for downloading the media/ directory (note that the trailing slashes matter):

$ rsync --archive --compress --partial --progress the_user@example.com:~/src/myproject/myproject/media/ ./myproject/media/

Here is an example of the same with a custom ~/.ssh/config configuration:

$ rsync --archive --compress --partial --progress examplecom:~/src/myproject/myproject/media/ ./myproject/media/

To upload the media/ directory to the remote server, you would again switch places for the source and target:

$ rsync --archive --compress --partial --progress ./myproject/media/ examplecom:~/src/myproject/myproject/media/

sftp

FTP clients like Transmit allow you to have SFTP connections either by username and password or by username and private key. You can even generate the private-public keys directly in the app there.

SFTP works like FTP, but all communication is encrypted there.

The final words

Use only encrypted connections for your network communications, encrypt your hard disk if you use manually generated private-public keys, and use strong passwords.

Be safe!


Cover photo by Jason D.

How to Export Data to XLSX Files

A while ago I wrote an article about exporting data to different spreadsheet formats. As recently I was reimplementing export to Excel for the 1st things 1st project, I noticed that the API changed a little, so it's time to blog about that again.

For Excel export I am using the XLSX file format which is a zipped XML-based format for spreadsheets with formatting support. XLSX files can be opened with Microsoft Excel, Apache OpenOffice, Apple Numbers, LibreOffice, Google Drive, and a handful of other applications. For building the XLSX file I am using openpyxl library.

Installing openpyxl

You can install openpyxl to your virtual environment the usual way with pip:

(venv) pip install openpyxl==2.6.0

Simplest Export View

To create a function exporting data from a QuerySet to XLSX file, you would need to create a view that returns a response with a special content type and file content as an attachment. Plug that view to URL rules and then link it from an export button in a template.

Probably the simplest view that generates XLSX file out of Django QuerySet would be this:

# movies/views.py
from datetime import datetime
from datetime import timedelta
from openpyxl import Workbook
from django.http import HttpResponse

from .models import MovieCategory, Movie

def export_movies_to_xlsx(request):
    """
    Downloads all movies as Excel file with a single worksheet
    """
    movie_queryset = Movie.objects.all()
    
    response = HttpResponse(
        content_type='application/vnd.openxmlformats-officedocument.spreadsheetml.sheet',
    )
    response['Content-Disposition'] = 'attachment; filename={date}-movies.xlsx'.format(
        date=datetime.now().strftime('%Y-%m-%d'),
    )
    workbook = Workbook()
    
    # Get active worksheet/tab
    worksheet = workbook.active
    worksheet.title = 'Movies'

    # Define the titles for columns
    columns = [
        'ID',
        'Title',
        'Description',
        'Length',
        'Rating',
        'Price',
    ]
    row_num = 1

    # Assign the titles for each cell of the header
    for col_num, column_title in enumerate(columns, 1):
        cell = worksheet.cell(row=row_num, column=col_num)
        cell.value = column_title

    # Iterate through all movies
    for movie in movie_queryset:
        row_num += 1
        
        # Define the data for each cell in the row 
        row = [
            movie.pk,
            movie.title,
            movie.description,
            movie.length_in_minutes,
            movie.rating,
            movie.price,
        ]
        
        # Assign the data for each cell of the row 
        for col_num, cell_value in enumerate(row, 1):
            cell = worksheet.cell(row=row_num, column=col_num)
            cell.value = cell_value

    workbook.save(response)

    return response

If you try this, you will notice, that there is no special formatting in it, all columns are of the same width, the value types are barely recognized, the header is displayed the same as the content. This is enough for further data export to CSV or manipulation with pandas. But if you want to present the data for the user in a friendly way, you need to add some magic.

Creating More Worksheets

By default, each Excel file has one worksheet represented as a tab. You can access it with:

worksheet = workbook.active
worksheet.title = 'The New Tab Title'

If you want to create tabs dynamically with data from the database of Python structures, you can at first delete the current tab and add the others with:

workbook.remove(workbook.active)

for index, category in enumerate(category_queryset):
    worksheet = workbook.create_sheet(
        title=category.title,
        index=index,
    )

Although not all spreadsheet applications support this, you can set the background color of the worksheet tab with:

worksheet.sheet_properties.tabColor = 'f7f7f9'

Working with Cells

Each cell can be accessed by its 1-based indexes for the rows and for the columns:

top_left_cell = worksheet.cell(row=1, column=1)
top_left_cell.value = "This is good!"

Styles and formatting are applied to individual cells instead of rows or columns. There are several styling categories with multiple configurations for each of them. You can find some available options from the documentation, but even more by exploring the source code.

from openpyxl.styles import Font, Alignment, Border, Side, PatternFill

top_left_cell.font = Font(name='Calibri', bold=True)
top_left_cell.alignment = Alignment(horizontal='center')
top_left_cell.border = Border(
    bottom=Side(border_style='medium', color='FF000000'),
)
top_left_cell.fill = PatternFill(
    start_color='f7f7f9',
    end_color='f7f7f9',
    fill_type='solid',
)

If you are planning to have multiple styled elements, instantiate the font, alignment, border, fill options upfront and then assign the instances to the cell attributes. Otherwise, you can get into memory issues when you have a lot of data entries.

Setting Column Widths

If you want to have some wider or narrower width for some of your columns, you can do this by modifying column dimensions. They are accessed by column letter which can be retrieved using a utility function:

from openpyxl.utils import get_column_letter

column_letter = get_column_letter(col_num)
column_dimensions = worksheet.column_dimensions[column_letter]
column_dimensions.width = 40

The units here are some relative points depending on the width of the letters in the specified font. I would suggest playing around with the width value until you find what works for you.

When defining column width is not enough, you might want to wrap text into multiple lines so that everything can be read by people without problems. This can be done with the alignment setting for the cell as follows:

from openpyxl.styles import Alignment

wrapped_alignment = Alignment(vertical='top', wrap_text=True)
cell.alignment = wrapped_alignment

Data Formatting

Excel automatically detects text or number types and aligns text to the left and numbers to the right. If necessary that can be overwritten.

There are some gotchas on how to format cells when you need a percentage, prices, or time durations.

Percentage

For percentage, you have to pass the number in float format from 0.0 till 1.0 and style should be 'Percent' as follows:

cell.value = 0.75
cell.style = 'Percent'

Currency

For currency, you need values of Decimal format, the style should be 'Currency', and you will need a special number format for currency other than American dollars, for example:

from decimal import Decimal
cell.value = Decimal('14.99')
cell.style = 'Currency'
cell.number_format = '#,##0.00 €'

Durations

For time duration, you have to pass timedelta as the value and define special number format:

from datetime import timedelta

cell.value = timedelta(minutes=90)
cell.number_format = '[h]:mm;@'

This number format ensures that your duration can be greater than '23:59', for example, '140:00'.

Freezing Rows and Columns

In Excel, you can freeze rows and columns so that they stay fixed when you scroll the content vertically or horizontally. That's similar to position: fixed in CSS.

To freeze the rows and columns, locate the top-left cell that is below the row that you want to freeze and is on the right from the column that you want to freeze. For example, if you want to freeze one row and one column, the cell would be 'B2'. Then run this:

worksheet.freeze_panes = worksheet['B2']

Fully Customized Export View

So having the knowledge of this article now we can build a view that creates separate sheets for each movie category. Each sheet would list movies of the category with titles, descriptions, length in hours and minutes, rating in percent, and price in Euros. The tabs, as well as the headers, can have different background colors for each movie category. Cells would be well formatted. Titles and descriptions would use multiple lines to fully fit into the cells.

# movies/views.py
from datetime import datetime
from datetime import timedelta
from openpyxl import Workbook
from openpyxl.styles import Font, Alignment, Border, Side, PatternFill
from openpyxl.utils import get_column_letter
from django.http import HttpResponse

from .models import MovieCategory, Movie

def export_movies_to_xlsx(request):
    """
    Downloads all movies as Excel file with a worksheet for each movie category
    """
    category_queryset = MovieCategory.objects.all()
    
    response = HttpResponse(
        content_type='application/vnd.openxmlformats-officedocument.spreadsheetml.sheet',
    )
    response['Content-Disposition'] = 'attachment; filename={date}-movies.xlsx'.format(
        date=datetime.now().strftime('%Y-%m-%d'),
    )
    workbook = Workbook()
    
    # Delete the default worksheet
    workbook.remove(workbook.active)

    # Define some styles and formatting that will be later used for cells
    header_font = Font(name='Calibri', bold=True)
    centered_alignment = Alignment(horizontal='center')
    border_bottom = Border(
        bottom=Side(border_style='medium', color='FF000000'),
    )
    wrapped_alignment = Alignment(
        vertical='top',
        wrap_text=True
    )

    # Define the column titles and widths
    columns = [
        ('ID', 8),
        ('Title', 40),
        ('Description', 80),
        ('Length', 15),
        ('Rating', 15),
        ('Price', 15),
    ]
    
    # Iterate through movie categories
    for category_index, category in enumerate(category_queryset):
        # Create a worksheet/tab with the title of the category
        worksheet = workbook.create_sheet(
            title=category.title,
            index=category_index,
        )
        # Define the background color of the header cells
        fill = PatternFill(
            start_color=category.html_color,
            end_color=category.html_color,
            fill_type='solid',
        )
        row_num = 1
        
        # Assign values, styles, and formatting for each cell in the header
        for col_num, (column_title, column_width) in enumerate(columns, 1):
            cell = worksheet.cell(row=row_num, column=col_num)
            cell.value = column_title
            cell.font = header_font
            cell.border = border_bottom
            cell.alignment = centered_alignment
            cell.fill = fill
            # set column width
            column_letter = get_column_letter(col_num)
            column_dimensions = worksheet.column_dimensions[column_letter]
            column_dimensions.width = column_width

        # Iterate through all movies of a category
        for movie in category.movie_set.all():
            row_num += 1
            
            # Define data and formats for each cell in the row
            row = [
                (movie.pk, 'Normal'),
                (movie.title, 'Normal'),
                (movie.description, 'Normal'),
                (timedelta(minutes=movie.length_in_minutes), 'Normal'),
                (movie.rating / 100, 'Percent'),
                (movie.price, 'Currency'),
            ]
            
            # Assign values, styles, and formatting for each cell in the row
            for col_num, (cell_value, cell_format) in enumerate(row, 1):
                cell = worksheet.cell(row=row_num, column=col_num)
                cell.value = cell_value
                cell.style = cell_format
                if cell_format == 'Currency':
                    cell.number_format = '#,##0.00 €'
                if col_num == 4:
                    cell.number_format = '[h]:mm;@'
                cell.alignment = wrapped_alignment

        # freeze the first row
        worksheet.freeze_panes = worksheet['A2']
        
        # set tab color
        worksheet.sheet_properties.tabColor = category.html_color

    workbook.save(response)

    return response

The Takeaways

  • Spreadsheet data can be used for further mathematical processing with pandas.
  • XLSX file format allows quite a bunch of formatting options that can make your spreadsheet data more presentable and user-friendly.
  • To see Excel export in action, go to 1st things 1st, log in as a demo user, and navigate to project results where you can export them as XLSX. Feedback is always welcome.

Cover photo by Tim Evans.

Equivalents in Python and JavaScript. Bonus

From time to time I google for the right syntax how to process lists and dictionaries in Python or arrays and objects in JavaScript. So I decided to extend my series of equivalents with those functions. After all, it's me too, who will be using the information I provide here.

All truthful elements

Sometimes we need to check from a list of conditions if all of them are true, or from a list of elements if all of them are not empty.

This can be checked with the following in Python:

items = [1, 2, 3]
all_truthy = all(items)
# True

And here is an equivalent in JavaScript:

items = [1, 2, 3];
all_truthy = items.every(Boolean);
// true

Any truthful elements

Similarly, we can check if at least one of the conditions is true, or there is at least one non-empty element in a list.

It Python we would do that with:

items = [0, 1, 2, 3]
some_truthy = any(items)
# True

And in JavaScript we would check it like this:

items = [0, 1, 2, 3];
some_truthy = items.some(Boolean);
// true

Iterate through each element and its index

Here is an example of how to iterate through a list of items and also check their indices in Python. It is useful for verbose console output when creating different command line tools that process data:

items = ['a', 'b', 'c', 'd']
for index, element in enumerate(items):
    print(f'{index}: {element};')

In JavaScript an analogous way to do the same would be using the forEach() method. The usual for loop is also an option, but I find the forEach() more elegant and clear.

items = ['a', 'b', 'c', 'd'];
items.forEach(function(element, index) {
    console.log(`${index}: ${element};`);
});

Map elements to the results of a function

To process all elements of a list, you can either iterate through them with the for loop and create a new list with modifications, or you can do that in one step by mapping the list items to a modification function. In Python this can be done with the map() function:

items = [0, 1, 2, 3]
all_doubled = list(map(lambda x: 2 * x, items))
# [0, 2, 4, 6]

In JavaScript the map() is a method of an array:

items = [0, 1, 2, 3];
all_doubled = items.map(x => 2 * x);
// [0, 2, 4, 6]

Filter elements by a function

When you need to search for some elements in a list or array and want to avoid for loops, you can use the filtering functionality. In Python that is doable with the filter() function that accepts the filtering function and the list and returns a new filtered list.

items = [0, 1, 2, 3]
only_even = list(filter(lambda x: x % 2 == 0, items))
# [0, 2]

In JavaScript there is a filter() method of the array for that.

items = [0, 1, 2, 3];
only_even = items.filter(x => x % 2 === 0);
// [0, 2]

In both cases, the filtering function checks each item if it is matching the filter criteria and returns true in that case.

Reduce elements by a function to a single value

When you want to apply some function to a list of items to get a single result in one go, you can use the reduce function. It works for summing, multiplying, ORing, ANDing, or checking maximums and minimums.

In Python there is a reduce() function for that.

from functools import reduce
items = [1, 2, 3, 4]
total = reduce(lambda total, current: total + current, items)
# 10

In JavaScript there is a reduce() method of the array.

items = [1, 2, 3, 4];
total = items.reduce((total, current) => total + current);
// 10

Merge dictionaries

There are multiple ways to merge dictionaries in Python or objects in JavaScript. But these are probably the simplest ones.

In Python it's decomposing dictionaries to tuples of keys and arrays, joining them, and creating a new dictionary.

d1 = {'a': 'A', 'b': 'B'}
d2 = {'a': 'AAA', 'c': 'CCC'}
merged = dict(list(d1.items()) + list(d2.items()))
# {'a': 'AAA', 'b': 'B', 'c': 'CCC'}

Analogously, in JavaScript it's spreading two objects into a new object:

d1 = {a: 'A', b: 'B'}
d2 = {a: 'AAA', c: 'CCC'}
merged = {...d1, ...d2};
// {a: 'AAA', b: 'B', c: 'CCC'}

The Takeaways

  • In both languages, you can traverse through lists of items without explicitly incrementing and referencing an index.
  • For processing list items, you don't necessarily need a loop. The dedicated methods or functions all() / every(), any() / some(), map(), filter(), and reduce() are there to help you.
  • In both languages, you can merge multiple dictionaries into one. If the same key appears in several dictionaries, the latest one will be used in the merged dictionary.

Of course, I also updated the cheat sheet with the full list of equivalents in Python and JavaScript that you saw here described. This cheat sheet helps me with a good overview next to my laptop, so I believe that it would be helpful to you too. The new revision 10 is with syntax highlighting, so it makes it even better to explore and understand.

Get the Ultimate Cheat Sheet of Equivalents in Python and JavaScript

Use it for good!


Cover photo by Darren Chan.

How to Create PDF Documents with Django in 2019

If you've read my Web Development with Django Cookbook, you might remember a recipe for creating PDF documents using Pisa xhtml2pdf. Well, this library does its job, but it supports only a subset of HTML and CSS features. For example, for multi-column layouts, you have to use tables, like it's 1994.

I needed some fresh and flexible option to generate donation receipts for the donation platform www.make-impact.org and reports for the strategic planner 1st things 1st I have been building. After a quick research I found another much more suitable library. It's called WeasyPrint. In this article, I will tell you how to use it with Django and what's valuable in it.

Features

WeasyPrint uses HTML and CSS 2.1 to create pixel-perfect, or let's rather say point-perfect, PDF documents. WeasyPrint doesn't use WebKit or Gecko but has its own rendering engine. As a proof that it works correctly, it passes the famous among web developers Acid2 test which was created back in the days before HTML5 to check how compatible browsers are with CSS 2 standards.

All supported features (and unsupported exceptions) are listed in the documentation. But my absolute favorites are these:

  • Layouts with floated elements. You don't have to use tables anymore if you want to have the recipient address on the left side and the sender information on the right side in a letter, or if you want to have the main content and the side notes in an exercise book. Just use floated elements.
  • Working links. The generated document can have clickable links to external URLs and internal anchors. You can straightforwardly create a clickable table of contents or a banner that leads back to your website.
  • Support for web fonts. With the wide variety of embeddable web fonts, your documents don't need to look boring anymore. Why not write titles in elegant cursive or in bold western letters?
  • Background images. By default, when you print an HTML page, all foreground images get printed, but the backgrounds are skipped. When you generate a PDF document for printing, you can show background images anywhere, even in the margins of the printed page.
  • SVG kept as vector images. When you have diagrams and graphics in a PDF document, you usually want to preserve the quality of the lines. Even if they look good on the screen, raster images might be not what you want, because on a printed page the resolution will differ and the quality can be lost. WeasyPrint keeps SVG images as vector images, so you have the highest possible quality in the prints.

Important Notes

WeasyPrint needs Python 3.4 or newer. That's great for new Django projects, but might be an obstacle if you want to integrate it into an existing website running on Python 2.7. Can it be the main argumentation for you to upgrade your old Django projects to the new Python version?

WeasyPrint is dependent on several OS libraries: Pango, GdkPixbuf, Cairo, and Libffi. In the documentation, there are understandable one-line instructions how to install them on different operating systems. You can have a problem only if you don't have full control of the server where you are going to deploy your project.

If you need some basic headers and footers for all pages, you can use @page CSS selector for that. If you need extended headers and footers for each page, it's best to combine the PDF document out of separate HTML documents for each page. Examples follow below.

The fun fact, Emojis are drawn using some weird raster single-color font. I don't recommend using them in your PDFs unless you replace them with SVG images.

Show Me the Code

A technical article is always more valuable when it has some quick code snippets to copy and paste. Here you go!

Simple PDF View

This snippet generates a donation receipt and shows it directly in the browser. Should the PDF be downloadable immediately, change content disposition from inline to attachment.

# -*- coding: UTF-8 -*-
from __future__ import unicode_literals

from django.http import HttpResponse
from django.template.loader import render_to_string
from django.utils.text import slugify
from django.contrib.auth.decorators import login_required

from weasyprint import HTML
from weasyprint.fonts import FontConfiguration

from .models import Donation

@login_required
def donation_receipt(request, donation_id):
    donation = get_object_or_404(Donation, pk=donation_id, user=request.user)
    response = HttpResponse(content_type="application/pdf")
    response['Content-Disposition'] = "inline; filename={date}-{name}-donation-receipt.pdf".format(
        date=donation.created.strftime('%Y-%m-%d'),
        name=slugify(donation.donor_name),
    )
    html = render_to_string("donations/receipt_pdf.html", {
        'donation': donation,
    })

    font_config = FontConfiguration()
    HTML(string=html).write_pdf(response, font_config=font_config)
    return response

Page Configuration Using CSS

Your PDF document can have a footer with an image and text on every page, using background-image and content properties:

{% load staticfiles i18n %}
<link href="https://fonts.googleapis.com/css?family=Playfair+Display:400,400i,700,700i,900" rel="stylesheet" />
<style>
@page {
    size: "A4";
    margin: 2.5cm 1.5cm 3.5cm 1.5cm;
    @bottom-center {
        background: url({% static 'site/img/logo-pdf.svg' %}) no-repeat center top;
        background-size: auto 1.5cm;
        padding-top: 1.8cm;
        content: "{% trans "Donation made via www.make-impact.org" %}";
        font: 10pt "Playfair Display";
        text-align: center;
        vertical-align: top;
    }
}
</style>

Pagination

You can show page numbers in the footer using CSS as follows.

@page {
    margin: 3cm 2cm;
    @top-center {
        content: "Documentation";
    }
    @bottom-right {
        content: "Page " counter(page) " of " counter(pages); 
    }
}

Horizontal Page Layout

You can rotate the page to horizontal layout with size: landscape.

@page {
    size: landscape;
}

HTML-based Footer

Another option to show an image and text in the header or footer on every page is to use an HTML element with position: fixed. This way you have more flexibility about formatting, but the element on all your pages will have the same content.

<style>
footer {
    position: fixed;
    bottom: -2.5cm;
    width: 100%;
    text-align: center;
    font-size: 10pt;
}
footer img {
    height: 1.5cm;
}
</style>
<footer>
    {% with website_url="https://www.make-impact.org" %}
        <a href="{{ website_url }}">
            <img alt="" src="{% static 'site/img/logo-contoured.svg' %}" />
        </a><br />
        {% blocktrans %}Donation made via <a href="{{ website_url }}">www.make-impact.org</a>{% endblocktrans %}
    {% endwith %}
</footer>

Document Rendering from Page to Page

When you need to have a document with complex unique headers and footers, it is best to render each page as a separate HTML document and then to combine them into one. This is how to do that:

def letter_pdf(request, letter_id):
    letter = get_object_or_404(Letter, pk=letter_id)
    response = HttpResponse(content_type='application/pdf')
    response['Content-Disposition'] = (
        'inline; '
        f'filename={letter.created:%Y-%m-%d}-letter.pdf'
    )
    COMPONENTS = [
        'letters/pdf/cover.html',
        'letters/pdf/page01.html',
        'letters/pdf/page02.html',
        'letters/pdf/page03.html',
    ]
    documents = []
    font_config = FontConfiguration()
    for template_name in COMPONENTS:
        html = render_to_string(template_name, {
            'letter': letter,
        })
        document = HTML(string=html).render(font_config=font_config)
        documents.append(document)

    all_pages = [page for document in documents for page in document.pages]
    documents[0].copy(all_pages).write_pdf(response)

    return response

Final Thoughts

I believe that WeasyPrint could be used not only for invoices, tickets, or booking confirmations but also for online magazines and small booklets. If you want to see PDF rendering with WeasyPrint in action, make a donation to your chosen organization at www.make-impact.org (when it's ready) and download the donation receipt. Or check the demo account at my.1st-things-1st.com and find the button to download the results of a prioritization project as PDF document.


Cover photo by Daniel Korpai.

What's New in the Third Edition of Web Development with Django Cookbook?

A couple of months ago the third release of Django Cookbook was published under the title Django 2 Web Development Cookbook - Third Edition. This edition was thoroughly and progressively re-written by Jake Kronika, the guy who had reviewed my second edition and had added a lot of value to it. I was sure that he wouldn't disappoint the readers, so I invited him to write the update. In this article, I will guide you through the main new highlights of over 500 pages of this new book.

Up to Date

Just like William S. Vincent's books, Django 2 Web Development Cookbook - Third Edition is adapted to Django 2.1 and Python 3.6. So you will be dealing with the state of the art technologies building your Django projects. Unicode strings, f-strings, super() without parameters, HTML5 tags, and object-oriented JavaScript to mention a few are used all over the book. The code is carefully generalized and even more adapted to the Don't-Repeat-Yourself (DRY) principle.

Working with Docker

Docker is one of the most popular deployment technologies and Jake gives a good compact introduction how to use it with Django.

Using Environment Variables for Configuration

12-factor app guidelines suggest saving app configuration in environment variables. In the book, there is a practical example of how to use it.

Multilingual Fields even with Region-specific Language Codes

I introduced multilingual fields in previous editions of the book, but there they had a limitation, that region-specific languages like Australian English or Swiss German were not supported. Now they are!

Using Precisely Semantic Markup with schema.org Microdata

Schema.org Microdata allows you to define the context of the content more specifically so that the content is more machine-readable. This was new to me and I still don't know the exact practical value of it, but I guess it is related to accessibility, new ways of presenting data via plugins, and Artificial Intelligence.

Defining Custom Templates for the Default Django Form Fields

Since Django 1.11 form fields are rendered using templates instead of Python code and those templates can be customized. There is a recipe that shows you how to do that.

Providing Responsive Images

HTML5 has the <picture> tag with <source> children that can be used in combination with the sorl-thumbnail Python package to generate different versions of the image based on your viewport size: load small image on the mobile, middle image on the tablet, and big image on the desktop or smart TV.

Uploading Images and Deleting them by Ajax

In my previous editions, I only showed how to upload a file by Ajax and attach it to a Django model. In Jake's update, it is shown how you can also delete the image.

Validating Passwords with Special Requirements

Since Django 1.11 you can define special requirements for the passwords of your users, for example, have a mix of small and big letters or include at least 1 number and 3 special characters, etc. There is a practical recipe how to do that.

Adding Watermarks to Images

When it comes to branding or copyright protection, it is common to add special watermarks, semitransparent images on top of your normal pictures. Jake added an example, how to do that and it was very interesting to me.

Authenticating with Auth0

In one recipe it is shown how to login to a Django website using Auth0, which seems to be a passwordless authentication system with integrations of popular connection services like OpenID Connect, Facebook, Google, Github, LinkedIn, PayPal, Yahoo!, and others. I haven't tried that myself yet, but it can be an interesting option for a social website.

Using Redis for Caching

It is common to cache websites using Memcached service, but a good alternative is caching with Redis and django-redis. Moreover, you can easily save user sessions there.

Creating Hierarchies with django-treebeard

In the previous editions, I introduced django-mptt for creating hierarchical structures. However, recently many projects are moving towards its alternative - django-treebeard which has more stability and writing speed. In the book, Jake shows you how to work with it.

Conclusion

There was a lot of new things to learn. For example, for me personally Docker usage was new, and I haven't heard of schema.org microdata and Auth0 which were introduced in this book. All in all, I think, Jake Kronika did an enormous job with this update and it's really worth purchasing this book, especially as there is a winter-holidays sale where you can get the EPUB, MOBI, and PDF with the code examples just for ~ 5 €.

Have a nice Christmas time and come back to this blog next year!


Cover photo by chuttersnap.

Equivalents in Python and JavaScript. Part 3

This is the 3rd part of 4-article series about analogies in Python and JavaScript. In the previous parts we covered a large part of the traditional Python 2.7 and JavaScript based on the ECMAScript 5 standard. This time we will start looking into Python 3.6 and JavaScript based on the ECMAScript 6 standard. ECMAScript 6 standard is pretty new and supported only the newest versions of browsers. For older browsers you will need Babel to compile your next-generation JavaScript code to the cross-browser-compatible equivalents. It opens the door to so many interesting things to explore. We will start from string interpolation, unpacking lists, lambda functions, iterations without indexes, generators, and sets!

Variables in strings

The old and inefficient way to build strings with values from variables is this concatenation:

name = 'World'
value = 'Hello, ' + name + '!\nWelcome!'

This can get very sparse and difficult to read. Also it is very easy to miss whitespaces in the sentence around variables.

Since Python version 3.6 and JavaScript based on the ECMAScript 6 standard, you can use so called string interpolation. These are string templates which are filled in with values from variables.

In Python they are also called f-string, because their notation starts with letter "f":

name = 'World'
value = f"""Hello, {name}!
Welcome!"""

price = 14.9
value = f'Price: {price:.2f} €'  # 'Price: 14.90 €'

In JavaScript string templates start and end with backticks:

name = 'World';
value = `Hello, ${name}!
Welcome!`;

price = 14.9;
value = `Price ${price.toFixed(2)} €`;  // 'Price: 14.90 €'

Note that string templates can be of a single line as well as of multiple lines. For f-strings in Python you can pass the format for variables, but you can't call methods of a variable unless they are properties and call getter methods.

Unpacking lists

Python and now JavaScript has an interesting feature to assign items of sequences into separate variables. For example, we can read the three values of a list into variables a, b, and c with the following syntax:

[a, b, c] = [1, 2, 3]

For tuples the parenthesis can be omitted. The following is a very popular way to swap values of two variables in Python:

a = 1
b = 2
a, b = b, a  # swap values

With the next generation JavaScript this can also be achieved:

a = 1;
b = 2;
[a, b] = [b, a];  // swap values

In Python 3.6 if we have an unknown number of items in a list or tuple, we can assign them to a tuple of several values while also unpacking the rest to a list:

first, second, *the_rest = [1, 2, 3, 4]
# first == 1
# second == 2
# the_rest == [3, 4]

This can also be done with JavaScript (ECMAScript 6):

[first, second, ...the_rest] = [1, 2, 3, 4];
// first === 1
// last === 2
// the_rest === [3, 4]

Lambda functions

Python and JavaScript have a very neat functionality to create functions in a single line. These functions are called lambdas. Lambdas are very simple functions that take one or more arguments and return some calculated value. Usually lambdas are used when you need to pass a function to another function as a callback or as a function to manipulate every separate elements in a sequence.

In Python, you would define a lambda using the lambda keyword, like this:

sum = lambda x, y: x + y
square = lambda x: x ** 2

In JavaScript lambdas use the => notation. If there are more than one arguments, they have to be in parenthesis:

sum = (x, y) => x + y;
square = x => Math.pow(x, 2);

Iteration without indexes

Many programming languages allow iterating through a sequence only by using indexes and incrementing their values. Then to get an item at some position, you would read it from an array, for example:

for (i=0; i<items.length; i++) {
    console.log(items[i]);
}

This is not a nice syntax and is very technical - it doesn't read naturally. What we really want is just to grab each value from the list. And Python has a very neat possibility just to iterate through the elements:

for item in ['A', 'B', 'C']:
    print(item)

In the modern JavaScript this is also possible with the for..of operator:

for (let item of ['A', 'B', 'C']) {
    console.log(item);
}

You can also iterate through a string characters in Python:

for character in 'ABC':
    print(character)

And in the modern JavaScript:

for (let character of 'ABC') {
    console.log(character);
}

Generators

Python and modern JavaScript has a possibility to define special functions through which you can iterate. With each iteration they return the next generated value in a sequence. These functions are called generators. With generators you can get numbers in a range, lines from a file, data from different paginated API calls, fibonacci numbers, and any other dynamicly generated sequences.

Technically generators are like normal functions, but instead of returning a value, they yield a value. This value will be returned for one iteration. And this generation happens as long as the end of the function is reached.

To illustrate that, the following Python code will create a generator countdown() which will return numbers from the given one back to 1, (like 10, 9, 8, ..., 1):

def countdown(counter):
    while counter > 0:
        yield counter
        counter -= 1
        
for counter in countdown(10):
    print(counter)

Exactly the same can be achieved in modern JavaScript, but notice the asterisk at the function statement. It defines that it's a generator:

function* countdown(counter) {
    while (counter > 0) {
        yield counter;
        counter--;
    }
}
for (let counter of countdown(10)) {
    console.log(counter);
}

Sets

We already had a look at lists, tuples and arrays. But here is another type of data - sets. Sets are groups of elements that ensure that each element there exists only once. Set theory also specifies set operations like union, intersection, and difference, but we won't cover them here today.

This is how to create a set, add elements to it, check if a value exists, check the total amount of elements in a set, and iterate through its values, and remove a value using Python:

s = set(['A'])
s.add('B'); s.add('C')
'A' in s
len(s) == 3
for elem in s:
    print(elem)
s.remove('C')

Here is how to achieve the same in modern JavaScript:

s = new Set(['A']);
s.add('B').add('C');
s.has('A') === true;
s.size === 3;
for (let elem of s.values()) {
    console.log(elem);
}
s.delete('C')

The Takeaways

  • String interpolation or literal templates allows you to have much cleaner and nicer code even with a possibility to have multiple lines of text.
  • You can iterate through elements in a sequence or group without using indexes.
  • Use generators when you have a sequence of nearly unlimited elements.
  • Use sets when you want to ensure fast check if data exists in a collection.
  • Use lambdas when you need short and clear single-line functions.

As you know from the previous parts, I am offering a cheat sheet with the whole list of equivalents in Python and JavaScript, both, time honored and future proof. To have something printed in front of your eyes is much more convenient than switching among windows or scrolling up and down until you find what you exactly were searching for. So I suggest you to get the cheat sheet and use it for good!

Get the Ultimate Cheat Sheet of Equivalents in Python and JavaScript

In the next and last part of the series, we will have a look at function arguments, classes, inheritance, and properties. Stay tuned!


Cover photo by Alex Knight

Equivalents in Python and JavaScript. Part 2

Last time we started a new series of articles about analogies in Python and JavaScript. We had a look at lists, arrays, dictionaries, objects, and strings, conditional assignments, and parsing integers. This time we will go through more interesting and more complex things like serializing dictionaries and lists to JSON, operations with regular expressions, as well as raising and catching errors.

JSON

When working with APIs it is very usual to serialize objects to JSON format and be able to parse JSON strings.

In Python it is done with the json module like this:

import json
json_data = json.dumps(dictionary, indent=4)
dictionary = json.loads(json_data)

Here we'll indent the nested elements in the JSON string by 4 spaces.

In JavaScript there is a JSON object that has methods to create and parse JSON strings:

json_data = JSON.stringify(dictionary, null, 4);
dictionary = JSON.parse(json_data);

Splitting strings by regular expressions

Regular expressions are multi-tool that once you master, you can accomplish lots of things.

In the last article, we saw how one can join lists of strings into a single string. But how can you split a long string into lists of strings? What if the delimiter can be not a single character as the comma, but a range of possible variations? This can be done with regular expressions and the split() method.

In Python, the split() method belongs to the regular expression pattern object. This is how you could split a text string into sentences by punctuation marks:

import re

# One or more characters of "!?." followed by whitespace
delimiter = re.compile(r'[!?\.]+\s*')

text = "Hello!!! What's new? Follow me."
sentences = delimiter.split(text)
# sentences == ['Hello', "What's new", 'Follow me', '']

In JavaScript the split() method belongs to the string:

// One or more characters of "!?." followed by whitespace
delimiter = /[!?\.]+\s*/;

text = "Hello!!! What's new? Follow me.";
sentences = text.split(delimiter)
// sentences === ["Hello", "What's new", "Follow me", ""]

Matching regular expression patterns in strings

Regular expressions are often used to validate data from the forms.

For example, to validate if the entered email address is correct, you would need to match it against a regular expression pattern. In Python that would look like this:

import re

# name, "@", and domain
pattern = re.compile(r'([\w.+\-]+)@([\w\-]+\.[\w\-.]+)')

match = pattern.match('hi@example.com')
# match.group(0) == 'hi@example.com'
# match.group(1) == 'hi'
# match.group(2) == 'example.com'

If the text matches the pattern, it returns a match object with the group() method to read the whole matched string, or separate captures of the pattern that were defined with the parenthesis. 0 means getting the whole string, 1 means getting the match in the first group, 2 means getting the match in the second group, and so on. If the text doesn't match the pattern, the None value will be returned.

In JavaScript the match() method belongs to the string and it returns either a match object, or null. Pretty similar:

// name, "@", and domain
pattern = /([\w.+\-]+)@([\w\-]+\.[\w\-.]+)/;

match = 'hi@example.com'.match(pattern);
// match[0] === 'hi@example.com'
// match[1] === 'hi'
// match[2] === 'example.com'

The match object in JavaScript acts as an array. Its value at the zeroth position is the whole matched string. The other indexes correspond to the captures of the pattern defined with the parenthesis.


Moreover, sometimes you need to search if a specific value exists in a string and at which letter position it will be found. That can be done with the search() method.

In Python this method belongs to the regular expression pattern and it returns the match object. The match object has the start() method telling at which letter position the match starts:

text = 'Say hi at hi@example.com'
first_match = pattern.search(text)
if first_match:
    start = first_match.start()  # start == 10

In JavaScript the search() method belongs to the string and it returns just an integer telling at which letter position the match starts. If nothing is found, -1 is returned:

text = 'Say hi at hi@example.com';
first_match = text.search(pattern);
if (first_match > -1) {
    start = first_match;  // start === 10
}

Replacing patterns in strings using regular expressions

Replacing with regular expressions usually happen when cleaning up data, or adding additional features. For example, we could take some text and make all email addresses clickable.

Python developers would use the sub() method of the regular expression pattern:

html = pattern.sub(
    r'<a href="mailto:\g<0>">\g<0></a>',
    'Say hi at hi@example.com',
)
# html == 'Say hi at <a href="mailto:hi@example.com">hi@example.com</a>'

JavaScript developers would use the replace() method of the string:

html = 'Say hi at hi@example.com'.replace(
    pattern, 
    '<a href="mailto:$&">$&</a>',
);
// html === 'Say hi at <a href="mailto:hi@example.com">hi@example.com</a>'

In Python the captures, also called as "backreferences", are accessible in the replacement string as \g<0>, \g<1>, \g<2>, etc. In JavaScript the same is accessible as $&, $1, $2, etc. Backreferences are usually used to wrap some strings or to switch places of different pieces of text.


It is also possible to replace a match with a function call. This can be used to do replacements within replacements or to count or collect some features of a text. For example, using replacements with function calls in JavaScript, I once wrote a fully functional HTML syntax highlighter.

Here let's change all email addresses in a text to UPPERCASE.

In Python, the replacement function receives the match object. We can use its group() method to do something with the matched text and return a text as a replacement:

text = pattern.sub(
    lambda match: match.group(0).upper(), 
    'Say hi at hi@example.com',
)
# text == 'Say hi at HI@EXAMPLE.COM'

In JavaScript the replacement function receives the whole match string, the first capture, the second capture, etc. We can do what we need with those values and then return some string as a replacement:

text = 'Say hi at hi@example.com'.replace(
    pattern,
    function(match, p1, p2) {
        return match.toUpperCase();
    }
);
// text === 'Say hi at HI@EXAMPLE.COM'

Error handling

Contrary to Python, client-side JavaScript normally isn't used for saving or reading files or connecting to remote databases. So try..catch blocks are quite rare in JavaScript compared to try..except analogy in Python.

Anyway, error handling can be used with custom user errors implemented and raised in JavaScript libraries and caught in the main code.

The following example in Python shows how to define a custom exception class MyException, how to raise it in a function, and how to catch it and handle in a try..except..finally block:

class MyException(Exception):
    def __init__(self, message):
        self.message = message
        
    def __str__(self):
        return self.message
        
def proceed():
    raise MyException('Error happened!')

try:
    proceed()
except MyException as err:
    print('Sorry! {}'.format(err))
finally:
    print('Finishing')    

The following example in JavaScript does exactly the same: here we define a MyException class, throw it in a function, and catch it and handle in the try..catch..finally block.

function MyException(message) {
   this.message = message;
   this.toString = function() {
       return this.message;
   }
}

function proceed() {
    throw new MyException('Error happened!');
}

try {
    proceed();
} catch (err) {
    if (err instanceof MyException) {
        console.log('Sorry! ' + err);
    }
} finally {
    console.log('Finishing');
}

The MyException class in both languages has a parameter message and a method to represent itself as a string using the value of the message.

Of course, exceptions should be raised/thrown just in the case of errors. And you define what is an error in your module design.

The Takeaways

  • Serialization to JSON is quite straightforward in both, Python and JavaScript.
  • Regular expressions can be used as multi-tools when working with textual data.
  • You can do replacements with function calls in both languages.
  • For more sophisticated software design you can use custom error classes.

As I mentioned last time, you can grab a side-by-side comparison of Python and JavaScript that I compiled for you (and my future self). Side by side you will see features from traditional list, array, dictionary, object, and string handling to modern string interpolation, lambdas, generators, sets, classes, and everything else. Use it for good.

Get the Ultimate Cheat Sheet of Equivalents in Python and JavaScript

In the next part of the series, we will have a look at textual templates, list unpacking, lambda functions, iteration without indexes, generators, and sets. Stay tuned!


Cover photo by Benjamin Hung.

Equivalents in Python and JavaScript. Part 1

Although Python and JavaScript are quite different languages, there are some analogies which full stack Python developers should know when developing web projects. In this series of 4 parts, I will explore what is similar in each of those languages and what are the common ways to solve common problems. This is not meant to be a reference and I will skip the basics like primitive variable types, conditions, and loops. But I will dig into more complex structures and data operations using both, Python and JavaScript. Also, I will try to focus on the practical use cases. This series should be interesting for the developers of Django, Flask, or another Python framework who want to get a grasp of traditional and modern vanilla JavaScript. On the other hand, it will be useful for the front-enders who want to better understand how the backend is working and maybe even start their own Django website.

Parsing integers

We'll begin with integer parsing.

In Python that's straightforward:

number = int(text)

But in JavaScript you have to explain what number system you expect: decimal, octal, hexadecimal, or binary:

number = parseInt(text, 10);

To use the "normal" decimal number system we are passing number 10 as the second parameter of the parseInt() function. 8 goes for octal, 16 for hexadecimal, or 2 – for binary. If the second parameter is missing, the number in text starts with zero, and you are using a slightly older browser, the number in the text will be interpreted as octal. For example,

parseInt('012') == 10  // in some older browsers
parseInt('012', 10) == 12

And that can really mess up your calculations.

Conditional assignment

For conditional assignment, Python and JavaScript have different syntaxes, but conditional assignments are quite popular in both languages. That's popular, because it's just a single statement to have a condition checking, the true-case value, and the false-case value.

Since Python 2.7 you can write conditional assignments like this:

value = 'ADULT' if age >= 18 else 'CHILD'

In JavaScript conditional assignments are done using ternary operator ?:, similar to the ones in C, C++, C#, Java, Ruby, PHP, Perl, Swift, and ActionScript:

value = age >= 18? 'ADULT': 'CHILD';

Object attribute value by attribute name

The normal way to access an object's attribute is by the dot notation in both, Python and JavaScript:

obj.color = 'YELLOW'

But what if you want to refer to an attribute by its name saved as a string? For example, the attribute name could be coming from a list of attributes or the attribute name is combined from two strings like 'title_' + lang_code.

For that reason, in Python, there are functions getattr() and setattr(). I use them a lot.

attribute = 'color'
value = getattr(obj, attribute, 'GREEN')
setattr(obj, attribute, value)

In JavaScript you can treat an object like a dictionary and pass the attribute name in square brackets:

attribute = 'color';
value = obj[attribute] || 'GREEN';
obj[attribute] = value;

To retrieve a default value when an object has no such attribute, in Python, getattr() has the third parameter. In JavaScript, if obj attribute doesn't exist, it will return the undefined value. Then it can be OR-ed with the default value that you want to assign. That's a common practice in JavaScript that you can find in many JavaScript libraries and frameworks.

Dictionary value by key

This is similar to the previous one. The normal way to assign a dictionary's value by key in both languages is using the square brackets:

dictionary = {}
dictionary['color'] = 'YELLOW'

To read a value in Python you can use the square-bracket notation, but it will fail on non-existing keys with KeyError. The more flexible way is to use the get() method which returns None for non-existing keys. Also you can pass an optional default value as the second parameter:

key = 'color'
value = dictionary.get(key, 'GREEN')

In JavaScript you would use the same trick as with object attributes, because dictionaries and objects are the same there:

key = 'color';
value = dictionary[key] || 'GREEN';

Slicing lists and strings

Python has the slice [:] operator to get parts of lists, tuples, and similar more complex structures, for example Django QuerySets:

items = [1, 2, 3, 4, 5]
first_two = items[:2]      # [1, 2]
last_two = items[-2:]      # [4, 5]
middle_three = items[1:4]  # [2, 3, 4]

In JavaScript arrays have the slice() method with the same effect and similar usage:

items = [1, 2, 3, 4, 5];
first_two = items.slice(0, 2);     // [1, 2] 
last_two = items.slice(-2);        // [4, 5]
middle_three = items.slice(1, 4);  // [2, 3, 4]

But don't mix it up with the splice() method which modifies the original array!


The [:] slice operator in Python also works for strings:

text = 'ABCDE'
first_two = text[:2]      # 'AB'
last_two = text[-2:]      # 'DE'
middle_three = text[1:4]  # 'BCD'

In JavaScript strings just like arrays have the slice() method:

text = 'ABCDE';
first_two = text.slice(0, 2);    // 'AB'
last_two = text.slice(-2);       // 'DE'
middle_three = text.slice(1, 4); // 'BCD'

Operations with list items

In programming it is very common to collect and analyze sequences of elements. In Python that is usually done with lists and in JavaScript with arrays. They have similar syntax and operations, but different method names to add and remove values.

This is how to concatenate two lists, add one value to the end, add one value to the beginning, get and remove a value from the beginning, get and remove a value from the end, and delete a certain value by index in Python:

items1 = ['A']
items2 = ['B']
items = items1 + items2  # items == ['A', 'B']
items.append('C')        # ['A', 'B', 'C']
items.insert(0, 'D')     # ['D', 'A', 'B', 'C']
first = items.pop(0)     # ['A', 'B', 'C']
last = items.pop()       # ['A', 'B']
items.delete(0)          # ['B']

This is how to do exactly the same with arrays in JavaScript:

items1 = ['A'];
items2 = ['B'];
items = items1.concat(items2);  // items === ['A', 'B']
items.push('C');                // ['A', 'B', 'C']
items.unshift('D');             // ['D', 'A', 'B', 'C']
first = items.shift();          // ['A', 'B', 'C']
last = items.pop();             // ['A', 'B']
items.splice(0, 1);             // ['B']

Joining lists of strings

It is very common after having a list or array of strings, to combine them into one string by a separator like comma or new line.

In Python that is done by the join() method of a string where you pass the list or tuple. Although it might feel unnatural, you start with the separator there. But I can assure that you get used to it after several times of usage.

items = ['A', 'B', 'C']
text = ', '.join(items)  # 'A, B, C'

In JavaScript the array has the join() method where you pass the separator:

items = ['A', 'B', 'C'];
text = items.join(', ');  // 'A, B, C'

The Takeaways

  • List and tuples in Python are similar to arrays in JavaScript.
  • Dictionaries in Python are similar to objects in JavaScript.
  • Strings in Python are similar to strings in JavaScript.
  • Numbers in JavaScript should be parsed with care.
  • Single-line conditional assignments exist in both languages.
  • Joining sequences of strings in Python is confusing, but you can quickly get used to it.

I compiled the whole list of equivalents of Python and JavaScript to a cheat sheet that you can print out and use for good. Side by side it compares traditional Python 2.7 and JavaScript based on ECMAScript 5 standard, as well as newer Python 3.6 and JavaScript based on ECMAScript 6 standard with such goodies as string interpolation, lambdas, generators, classes, etc.

Get the Ultimate Cheat Sheet of Equivalents in Python and JavaScript

In the next part of the series, we will have a look at JSON creation and parsing, operations with regular expressions, and error handling. Stay tuned!


Cover photo by Benjamin Hung.

Recap of DjangoCon Europe 2017

"DjangoCon, why is everybody wearing this t-shirt?" wondered the security guys in the airport of Florence, Italy, in the beginning of April. The reason for that was DjangoCon Europe 2017 happening there for a week, full of interesting talks in an exciting location.

What I liked, was that the conference was not only about technical novelties in Django world, but also about human issues that programmers deal with in everyday life.

Interesting Non-tech Topics

According to a manifest, the conference had a goal to strengthen the Django community and to shape responsible attitude towards the works done with Django.

Healthy and Successful Community

We have to build stronger communities including everyone who wants to participate without discrimination. Although, at first, it might be difficult as people have biases, i.e. prejudices for or against one person or group; by being emphatic we can accept and include everyone no matter what is their gender identity or expression, sexual orientation, ethnicity, race, neurodiversity, age, religion, disabilities, geographical location, food diversities, body size, or family status.

Valuing diversity and individual differences is the key for a healthy, positive and successful community, that empowers its members and helps them grow stronger and happier.

Responsibility for How We Use Technology

Information technology companies (Apple, Alphabet, Microsoft, Amazon, and Facebook) are among the most traded companies in the world. IT connects people and their things, automates processes, stores and treats historical data. Usually you don't need many physical resources to start an IT business. Software developers have a power to shape the future, but should use this power responsibly:

With this, great responsibility is upon us: to make the future a better place, to make the future more evenly distributed, across gender gaps and discriminations, breaking economical, political and geographical barriers.

Business

  • When creating an online business, it is important to think about the business value that your product will give to people and the way you will make money with it. Don't make assumptions without talking to your customers.
  • When choosing employees for your company, give them freedom how to prove their knowledge: by a quiz, or whiteboard interview, or a take-home programming task. Different people have different ways how to best represent their skills.
  • Launch as early as possible. Track the usage statistics with Google Analytics or other analytics service. Collect emails for the mailing list. Write about your product in a blog and personalized emails.
  • Django is an open-source project based on the hard work of many professionals, and if you gain any commercial value of it and appreciate the framework, you should donate to the Django Software Foundation.

Interesting Tech Topics

From the technical point of view, I liked several ideas mentioned in the conference:

Automate your processes

  • For starting new projects, you can have boilerplates with the mostly used functionalities already prepared. Django management command startproject has a parameter --template for that where you can pass a URL to a zip file.
  • Developers should have troubleshooting checklists for debugging, just like airplane pilots.
  • There are several types of tests. Unit tests check the functionality of individual functions or methods. Integration tests check how different units work together. The functional tests check how the processes of business requirements work from start to end. Finally, there is manual testing requiring people to click through the website and fill in the forms. Some tests like the ones involving third-party authentication or mobile phones, are hardly possible to automate. Anyway, manual testing is the most expensive in time and resources (besides it being boring for the tester), functional tests go after them, then integration tests, and lastly unit tests. Although automatic testing adds up to the development time, in the long run it makes the systems more stable and error proof.

What about Django

  • You can extend the Django ORM with custom lookups, transactions, and filtered prefetchings, to make your QuerySets more readable and more capable.
  • Once again, PostgreSQL has more capabilities than MySQL and is more stable. Use EXPLAIN ANALYZE ... SQL command to find the bottlenecks of your database queries. You can usually fix them by adding indexes.
  • You can have custom indexes for your database tables, to optimize the performance on PostgreSQL (or some other vendor) database.
  • Django 1.11 is out and it's a long-term support version.

What about Third Party Django Packages

  • After analyzing the 6 most popular model translation packages (parler, hvad, klingon, modeltranslation, nece, and i18nfield) from different angles (database support, integration in django admin and forms, performance, etc.), django-hvad seemed to be the winning approach.
  • You can visually build static websites with very little coded configuration using django-cms and djangocms-cascade. The djangocms-cascade provides an alternative nested-plugins system for Django CMS.

What about Django Projects

  • If you build web apps for developing countries, you have to keep these things in mind: people might be using cell phones instead of computers (you need responsive design with small or no images), Internet connectivity is slow and unstable (websites have to be fast and should preferably have offline versions), the users do not always understand English (the websites should be translated and adapted), and locations where people live do not always have street addresses.
  • Some interesting use cases: tracking the health of the heart with Arduino and Django, providing weather data to the whole Europe using Django, managing a radio station in Birmingham using Django.

Thanks

Finally, thanks to the organizers for making this conference as great as it was. The city was beautiful, the food and coffee was delicious, the location for the talks was impressive. Looking forward to the next DjangoCon Europe!


Cover photo by Aidas Bendoraitis

Tracking the Results of Cron Jobs

Every Django website needs some automatic background tasks to execute regularly. The outdated sessions need to be cleaned up, search index needs to be updated, some data needs to be imported from RSS feeds or APIs, backups need to be created, you name it. Usually, if not all the time, those regular tasks are being set as cron jobs. However, when some task is run in the background, by default, you don't get any feedback whether it was successfully completed, or whether it crashed on the way. In this post I will show you how I handle the results of cron jobs.

In a Django project, all those tasks are usually implemented as management commands. For each such command I write a short bash script, that will call the management command with specific parameters and will print the verbose output to a log file.

Let's say my project structure is like this on a remote server:

/home/myproject
├── bin
├── include
├── lib
├── public_html
├── backups
├── project
│ └── myproject
├── scripts
└── logs

A virtual environment is created in the home directory of myproject linux user. The Django project itself is kept under project directory. The scripts directory is for my bash scripts. And the logs directory is for the verbose output of the bash scripts.

For example, for the default clearsessions command that removes outdated sessions, I would create scripts/cleanup.sh bash script as follows:

#!/usr/bin/env bash
SECONDS=0
PROJECT_PATH=/home/myproject
CRON_LOG_FILE=${PROJECT_PATH}/logs/cleanup.log

echo "Cleaning up the database" > ${CRON_LOG_FILE}
date >> ${CRON_LOG_FILE}

cd ${PROJECT_PATH}
source bin/activate
cd project/myproject
python manage.py clearsessions --verbosity=2 --traceback >> ${CRON_LOG_FILE} 2>&1

echo "Finished." >> ${CRON_LOG_FILE}
duration=$SECONDS
echo "$(($duration / 60)) minutes and $(($duration % 60)) seconds elapsed." >> ${CRON_LOG_FILE}

To run this command every night at 1 AM, you could create a file myproject_crontab with the following content:

MAILTO=""
00 01 * * * /home/myproject/scripts/cleanup.sh

Then register the cron jobs with:

$ crontab myproject_crontab

By such a bash script, I can track:

  • At what time the script was last executed.
  • What is the verbose output of the management command.
  • If the management command broke, what was in the traceback.
  • Whether the command finished executing or hung up.
  • How long it took to run the command.

In addition, this gives me information whether the crontab was registered and whether the cron service was running at all. As I get the total time of execution in minutes and seconds, I can decide how often I can call the cron job regularly so that it doesn't clash with another cron job.

When you have multiple Django management commands, you can group them thematically into single bash script, or you can wrap them into individual bash scripts. After putting them into the crontab, the only thing left is manually checking the logs from time to time.

If you have any suggestions how I could even improve this setup, I would be glad to hear your opinion in the comments.

Here is the Gist of the scripts in this post. To see some examples of custom Django management commands, you can check Chapter 9, Data Import and Export in my book Web Development with Django Cookbook - Second Edition.


Cover photo by Redd Angelo

Django Administration: Inlines for Inlines

The default Django model administration comes with a concept of inlines. If you have a one-to-many relationship, you can edit the parent and its children in the same form. However, you are limited in a way that you cannot have inlines under inlines at nested one-to-many relations. For example, you can't show models Painter, Picture, and Review in the same form if one painter may have drawn multiple pictures and each picture may have several reviews.

In this article I would like to share a workaround allowing you to quickly access the inlines of an inline model. The idea is that for every inline you can provide a HTML link leading to the separate form where you can edit the related model and its own relations. It's as simple as that.

For example, in the form of Painter model, you have the instances of Picture listed with specific links "Edit this Picture separately":

When such a link is clicked, the administrator goes to the form of the Picture model which shows the instances of Review model listed underneath:

Let's have a look, how to implement this.

First of all, I will create a gallery app and define the three models there. Nothing fancy here. The important part there is just that the Picture model has a foreign key to the Painter model and the Review model has a foreign key to the Picture model.

# gallery/models.py
# -*- coding: UTF-8 -*-
from __future__ import unicode_literals

import os

from django.db import models
from django.utils.encoding import python_2_unicode_compatible
from django.utils.translation import ugettext_lazy as _
from django.utils.text import slugify


@python_2_unicode_compatible
class Painter(models.Model):
name = models.CharField(_("Name"), max_length=255)

class Meta:
verbose_name = _("Painter")
verbose_name_plural = _("Painters")

def __str__(self):
return self.name


def upload_to(instance, filename):
filename_base, filename_ext = os.path.splitext(filename)
return "painters/{painter}/{filename}{extension}".format(
painter=slugify(instance.painter.name),
filename=slugify(filename_base),
extension=filename_ext.lower(),
)

@python_2_unicode_compatible
class Picture(models.Model):
painter = models.ForeignKey(Painter, verbose_name=_("Painter"), on_delete=models.CASCADE)
title = models.CharField(_("Title"), max_length=255)
picture = models.ImageField(_("Picture"), upload_to=upload_to)

class Meta:
verbose_name = _("Picture")
verbose_name_plural = _("Pictures")

def __str__(self):
return self.title


@python_2_unicode_compatible
class Review(models.Model):
picture = models.ForeignKey(Picture, verbose_name=_("Picture"), on_delete=models.CASCADE)
reviewer = models.CharField(_("Reviewer name"), max_length=255)
comment = models.TextField(_("Comment"))

class Meta:
verbose_name = _("Review")
verbose_name_plural = _("Reviews")

def __str__(self):
return self.reviewer

Then I will create the administration definition for the models of the gallery app. Here I will set two types of administration for the Picture model:

  • By extending admin.StackedInline I will create administration stacked as inline.
  • By extending admin.ModelAdmin I will create administration in a separate form.

In Django model administration besides usual form fields, you can also include some computed values. This can be done by your fields (or fieldsets) and readonly_fields attributes referring to a callable or a method name.

You can set a translatable label for those computed values by defining short_description attribute for the callable or method. If you want to render some HTML, you can also set the allow_tags attribute to True (otherwise your HTML string will be escaped).

# gallery/admin.py
# -*- coding: UTF-8 -*-
from django.contrib import admin
from django.core.urlresolvers import reverse
from django.utils.translation import ugettext_lazy as _
from django.utils.text import force_text

from .models import Painter, Picture, Review

def get_picture_preview(obj):
if obj.pk: # if object has already been saved and has a primary key, show picture preview
return """<a href="{src}" target="_blank"><img src="{src}" alt="{title}" style="max-width: 200px; max-height: 200px;" /></a>""".format(
src=obj.picture.url,
title=obj.title,
)
return _("(choose a picture and save and continue editing to see the preview)")
get_picture_preview.allow_tags = True
get_picture_preview.short_description = _("Picture Preview")


class PictureInline(admin.StackedInline):
model = Picture
extra = 0
fields = ["get_edit_link", "title", "picture", get_picture_preview]
readonly_fields = ["get_edit_link", get_picture_preview]

def get_edit_link(self, obj=None):
if obj.pk: # if object has already been saved and has a primary key, show link to it
url = reverse('admin:%s_%s_change' % (obj._meta.app_label, obj._meta.model_name), args=[force_text(obj.pk)])
return """<a href="{url}">{text}</a>""".format(
url=url,
text=_("Edit this %s separately") % obj._meta.verbose_name,
)
return _("(save and continue editing to create a link)")
get_edit_link.short_description = _("Edit link")
get_edit_link.allow_tags = True


@admin.register(Painter)
class PainterAdmin(admin.ModelAdmin):
save_on_top = True
fields = ["name"]
inlines = [PictureInline]


class ReviewInline(admin.StackedInline):
model = Review
extra = 0
fields = ["reviewer", "comment"]


@admin.register(Picture)
class PictureAdmin(admin.ModelAdmin):
save_on_top = True
fields = ["painter", "title", "picture", get_picture_preview]
readonly_fields = [get_picture_preview]
inlines = [ReviewInline]

UPDATE! Since Django 2.0, the get_picture_preview() function should use mark_safe() instead of allow_tags=True:

from django.utils.safestring import mark_safe
# ...
def get_edit_link(self, obj=None):
if obj.pk: # if object has already been saved and has a primary key, show link to it
url = reverse(
'admin:%s_%s_change' % (obj._meta.app_label, obj._meta.model_name),
args=[force_text(obj.pk)]
)
return mark_safe("""<a href="{url}">{text}</a>""".format(
url=url,
text=_("Edit this %s separately") % obj._meta.verbose_name,
))
return _("(save and continue editing to create a link)")
get_edit_link.short_description = _("Edit link")

In this administration setup, the get_edit_link() method creates a HTML link between the inline and the separate administration form for the Picture model. As you can see, I also added the get_picture_preview() function as a bonus. It is included in both administration definitions for the Picture model and its purpose is to show a preview of the uploaded picture after saving it.

To recap, nested inlines are not supported by Django out of the box. However, you can have your inlines edited in a separate page with the forms linked to each other. For the linking you would use some magic of the readonly_fields attribute.

What if you really need to have inlines under inlines in your project? In that case you might check django-nested-admin and don't hesitate to share your experience with it in the comments.


Cover photo by Denys Nevozhai

Deploying a Django Website on Heroku

Once you have a working project, you have to host it somewhere. One of the most popular deployment platforms nowadays is Heroku. Heroku belongs to a Platform as a Service (PaaS) category of cloud computing services. Every Django project you host on Heroku is running inside a smart container in a fully managed runtime environment. Your project can scale horizontally (adding more computing machines) and you pay for what you use starting with a free tier. Moreover, you won't need much of system administrator's skills to do the deployment - once you do the initial setup, the further deployment is as simple as pushing Git repository to a special heroku remote.

However, there are some gotchas to know before choosing Heroku for your Django project:

  • One uses PostgreSQL database with your project. MySQL is not an option.
  • You cannot store your static and media files on Heroku. One should use Amazon S3 or some other storage for that.
  • There is no mailing server associated with Heroku. One can use third-party SendGrid plugin with additional costs, GMail SMTP server with sent email amount limitations, or some other SMTP server.
  • The Django project must be version-controlled under Git.
  • Heroku works with Python 2.7. Python 3 is not yet supported.

Recently I deployed a small Django project on Heroku. To have a quick reference for the future, I summed up the process here providing instructions how to do that for future reference.

1. Install Heroku Toolbelt

Sign up for a Heroku account. Then install Heroku tools for doing all the deployment work in the shell.

To connect your shell with Heroku, type:

$ heroku login

When asked, enter your Heroku account's email and password.

2. Prepare Pip Requirements

Activate your project's virtual environment and install Python packages required for Heroku:

(myproject_env)$ pip install django-toolbelt

This will install django, psycopg2, gunicorn, dj-database-url, static3, and dj-static to your virtual environment.

Install boto and Django Storages to be able to store static and media files on an S3 bucket:

(myproject_env)$ pip install boto
(myproject_env)$ pip install django-storages

Go to your project's directory and create the pip requirements that Heroku will use in the cloud for your project:

(myproject_env)$ pip freeze -l > requirements.txt

3. Create Heroku-specific Files

You will need two files to tell Heroku what Python version to use and how to start a webserver.

In your project's root directory create a file named runtime.txt with the following content:

python-2.7.11

Then at the same location create a file named Procfile with the following content:

web: gunicorn myproject.wsgi --log-file -

4. Configure the Settings

As mentioned in the "Web Development with Django Cookbook - Second Edition", we keep the developmnent and production settings in separate files both importing the common settings from a base file.

Basically we have myproject/conf/base.py with the settings common for all environments.

Then myproject/conf/dev.py contains the local database and dummy email configuration as follows:

# -*- coding: UTF-8 -*-
from __future__ import unicode_literals
from .base import *

DATABASES = {
"default": {
"CONN_MAX_AGE": 0,
"ENGINE": "django.db.backends.postgresql",
"HOST": "localhost",
"NAME": "myproject",
"PASSWORD": "",
"PORT": "",
"USER": "postgres"
}
}

EMAIL_BACKEND = "django.core.mail.backends.console.EmailBackend"

Lastly for the production settings we need myproject/conf/prod.py with special database configuration, non-debug mode, and unrestrictive allowed hosts as follows:

# -*- coding: UTF-8 -*-
from __future__ import unicode_literals
from .base import *
import dj_database_url

DATABASES = {
"default": dj_database_url.config()
}

ALLOWED_HOSTS = ["*"]

DEBUG = False

Now let's open myproject/settings.py and add the following content:

# -*- coding: UTF-8 -*-
from __future__ import unicode_literals
from .conf.dev import *

Finally, open the myproject/wsgi.py and change the location of the DJANGO_SETTINGS_MODULE there:

os.environ.setdefault("DJANGO_SETTINGS_MODULE", "myproject.conf.prod")

5. Set Up Amazon S3 for Static and Media Files

Create an Amazon S3 bucket myproject.media at the AWS Console (web interface for Amazon Web Services). Go to the properties of the bucket, expand "Permissions" section, click on the "add bucket policy" button and enter the following:

{
"Version": "2008-10-17",
"Statement": [
{
"Sid": "AllowPublicRead",
"Effect": "Allow",
"Principal": {
"AWS": "*"
},
"Action": "s3:GetObject",
"Resource": "arn:aws:s3:::myproject.media/*"
}
]
}

This ensures that files on the S3 bucket will be accessible publicly without any API keys.

Go back to your Django project and add storages to the INSTALLED_APPS in myproject/conf/base.py:

INSTALLED_APPS = [
# ...
"storages",
]

Media files and static files will be stored on different paths under S3 bucket. To implement that, we need to create two Python classes under a new file myproject/s3utils.py as follows:

# -*- coding: UTF-8 -*-
from __future__ import unicode_literals
from storages.backends.s3boto import S3BotoStorage

class StaticS3BotoStorage(S3BotoStorage):
"""
Storage for static files.
"""

def __init__(self, *args, **kwargs):
kwargs['location'] = 'static'
super(StaticS3BotoStorage, self).__init__(*args, **kwargs)


class MediaS3BotoStorage(S3BotoStorage):
"""
Storage for uploaded media files.
"""

def __init__(self, *args, **kwargs):
kwargs['location'] = 'media'
super(MediaS3BotoStorage, self).__init__(*args, **kwargs)

Finally, let's edit the myproject/conf/base.py and add AWS settings:

AWS_S3_SECURE_URLS = False       # use http instead of https
AWS_QUERYSTRING_AUTH = False # don't add complex authentication-related query parameters for requests
AWS_S3_ACCESS_KEY_ID = "..." # Your S3 Access Key
AWS_S3_SECRET_ACCESS_KEY = "..." # Your S3 Secret
AWS_STORAGE_BUCKET_NAME = "myproject.media"
AWS_S3_HOST = "s3-eu-west-1.amazonaws.com" # Change to the media center you chose when creating the bucket

STATICFILES_STORAGE = "myproject.s3utils.StaticS3BotoStorage"
DEFAULT_FILE_STORAGE = "myproject.s3utils.MediaS3BotoStorage"

# the next monkey patch is necessary to allow dots in the bucket names
import ssl
if hasattr(ssl, '_create_unverified_context'):
ssl._create_default_https_context = ssl._create_unverified_context

Collect static files to the S3 bucket:

(myproject_env)$ python manage.py collectstatic --noinput

6. Set Up Gmail to Send Emails

Open myproject/conf/prod.py and add the following settings:

EMAIL_USE_TLS = True
EMAIL_HOST = "smtp.gmail.com"
EMAIL_HOST_USER = "myproject@gmail.com"
EMAIL_HOST_PASSWORD = "mygmailpassword"
EMAIL_PORT = 587

7. Push to Heroku

Commit and push all the changes to your Git origin remote. Personally I prefer using SourceTree to do that, but you can also do that in the command line, PyCharm, or another software.

In your project directory type the following:

(myproject_env)$ heroku create my-unique-project

This will create a Git remote called "heroku", and a new Heroku project "my-unique-project" which can be later accessed at http://my-unique-project.herokuapp.com.

Push the changes to heroku remote:

(myproject_env)$ git push heroku master

8. Transfer Your Local Postgres Database To Heroku

Create local database dump:

(myproject_env)$ PGPASSWORD=mypassword pg_dump -Fc --no-acl --no-owner -h localhost -U myuser mydb > mydb.dump

Upload the database dump temporarily to some server, for example, S3 bucket: http://myproject.media.s3-eu-west-1.amazonaws.com/mydb.dump. Then import that dump into the Heroku database:

(myproject_env)$ heroku pg:backups restore 'http://myproject.media.s3-eu-west-1.amazonaws.com/mydb.dump' DATABASE_URL

Remove the database dump from S3 server.

9. Set Environment Variables

If your Git repository is not private, put your secret values in environment variables rather than in the Git repository directly.

(myproject_env)$ heroku config:set AWS_S3_ACCESS_KEY_ID=ABCDEFG123
$ heroku config:set AWS_S3_SECRET_ACCESS_KEY=aBcDeFg123

To read out the environment variables you can type:

(myproject_env)$ heroku config

To read out the environment variables in the Python code open myproject/conf/base.py and type:

import os
AWS_S3_ACCESS_KEY_ID = os.environ.get("AWS_S3_ACCESS_KEY_ID", "")
AWS_S3_SECRET_ACCESS_KEY = os.environ.get("AWS_S3_SECRET_ACCESS_KEY", "")

10. Set DNS Settings

Open your domain settings and set CNAME to "my-unique-project.herokuapp.com".

At last, you are done! Drop in the comments if I missed some part. For the new updates, see the next section.

*. Update Production

Push the changes to heroku remote:

(myproject_env)$ git push heroku master

If you have changed something in the static files, collect them again:

(myproject_env)$ python manage.py collectstatic --noinput

Collecting static files to S3 bucket takes quite a long time, so I do not recommend to do that automatically every time when you want to deploy to Heroku.

Further Reading

You can read more about Django on Heroku in the following resources:


Cover photo by Frances Gunn

How to Find the Performance Bottlenecks in Your Django Views?

Once you have your Django projects running, you come to situations, when you need to optimize for performance. The rule of thumb is to find the bottlenecks and then to take action to eliminate them by more idiomatic Python code, database denormalization, caching, or other techniques.

What is a bottleneck? Literally it refers to the top narrow part of a bottle. In engineering, bottleneck is a case where the performance or capacity of an entire system is limited by a single or small number of components or resources.

How to find these parts of your code? The most trivial way is to check the current time before specific code execution and after that code execution, and then count the time difference:

from datetime import datetime
start = datetime.now()
# heavy execution ...
end = datetime.now()
d = end - start # datetime.timedelta object
print d.total_seconds() # prints something like 7.861985

However, measuring code performance for Django projects like this is inefficient, because you need a lot of such wrappers for your code until you find which part is the most critical. Also you need a lot of manual computation to find the critical parts.

Recently I found line_profiler module that can inspect the performance of the code line by line. By default, to use line_profiler for your functions, you should decorate them with @profile decorator and then to execute the script:

$ kernprof -l some_script_to_profile.py

This script will execute your script, analize the decorated function, and will save results to a binary file that can later be inspected with:

$ python -m line_profiler some_script_to_profile.py.lprof

That's quite complicated, but to use line_profiler for Django views, you can install django-devserver which replaces the original development server of Django and will output the performance calculations immediately in the shell like this:

[30/Jan/2015 02:26:40] "GET /quotes/json/ HTTP/1.1" 200 137
[sql] 1 queries with 0 duplicates
[profile] Total time to render was 0.01s
[profile] Timer unit: 1e-06 s

Total time: 0.001965 s
File: /Users/archatas/Projects/quotes_env/project/inspirational/quotes/views.py
Function: quote_list_json at line 27

Line # Hits Time Per Hit % Time Line Contents
==============================================================
27 def quote_list_json(request):
28 1 2 2.0 0.1 quote_dict_list = []
29 2 1184 592.0 60.3 for quote in InspirationQuote.objects.all():
30 1 1 1.0 0.1 quote_dict = {
31 1 1 1.0 0.1 'author': quote.author,
32 1 1 1.0 0.1 'quote': quote.quote,
33 1 363 363.0 18.5 'picture': quote.get_medium_picture_url(),
34 }
35 1 1 1.0 0.1 quote_dict_list.append(quote_dict)
36
37 1 42 42.0 2.1 json_data = json.dumps(quote_dict_list)
38 1 370 370.0 18.8 return HttpResponse(json_data, content_type="application/json")

The most interesting data in this table is the "% Time" column, giving an overview in percentage which lines of the Django view function are the most time-consuming. For example, here it says that I should pay the most attention to the QuerySet, the method get_medium_picture_url() and the HttpResponse object.

To setup line profiling, install line_profiler and django-devserver to you virtual environment:

(myproject_env)$ pip install line_profiler
(myproject_env)$ pip install django-devserver

Then make sure that you have the following settings in your settings.py or local_settings.py:

# settings.py
INSTALLED_APPS = (
# ...
'devserver',
)

MIDDLEWARE_CLASSES = (
# ...
'devserver.middleware.DevServerMiddleware',
)

DEVSERVER_MODULES = (
'devserver.modules.sql.SQLRealTimeModule',
'devserver.modules.sql.SQLSummaryModule',
'devserver.modules.profile.ProfileSummaryModule',

# Modules not enabled by default
'devserver.modules.profile.LineProfilerModule',
)

DEVSERVER_AUTO_PROFILE = True # profiles all views without the need of function decorator

When you execute

(myproject_env)$ python manage.py runserver

it will run the development server from django-devserver and for each visited view, it will show the analysis of code performance. I have tested this setup with Django 1.7, but it should work since Django 1.3.

Do you know any more useful tools to check for performance bottlenecks?

❌