Connexion
At DigitalOcean, when you need large amounts of static data (images, documents, or videos), you have two options: Volumes Block Storage and Spaces Object Storage. In case of Volumes, you mount an extra hard drive to your server and use the file system to manage your files. Whereas with Spaces, you store files in the cloud and use a special API to create, read and delete files there.
Spaces Object Storage is comparable with AWS S3 Cloud Object Storage. It even supports the same API for dealing with data there.
Here are some benefits of using Spaces:
I will walk you through setting up Spaces for your media files.
When creating Spaces Object Storage at DigitalOcean, you will be asked for these values:
For the created Spaces instance, I keep the settings unchanged:
This will create an instance, which resources, aka media files, can be accessed under
https://pybazaar.fra1.digitaloceanspaces.com
and
https://pybazaar.fra1.cdn.digitaloceanspaces.com
Now go to API ➔ Spaces Keys and choose Generate New Key.
There you'll generate an Access Key and Secret Key. You'll need those in the Django settings and for example Transit app for easy mass file management.
Install django-storages and boto3 to your Django project.
Lateo.net - Flux RSS en pagaille (pour en ajouter : @ moi) (venv)$ pip install boto3
(venv)$ pip install django-storages[s3]Add the STORAGES setting:
STORAGES = {
"default": {
"BACKEND": "storages.backends.s3.S3Storage",
"OPTIONS": {
"bucket_name": "pybazaar",
"access_key": get_secret("SPACES_ACCESS_KEY"),
"secret_key": get_secret("SPACES_SECRET_KEY"),
"region_name": "fra1",
"endpoint_url": "https://pybazaar.fra1.digitaloceanspaces.com",
"default_acl": "public-read",
"location": "media",
# required for the correct storage.exists() functioning
"file_overwrite": False,
# don't append any authentication parameters to the files.
"querystring_auth": False,
},
},
"staticfiles": {
# For static files, use file-system storage
"BACKEND": "django.contrib.staticfiles.storage.StaticFilesStorage",
# Or Whitenoise storage
# "BACKEND": "whitenoise.storage.CompressedStaticFilesStorage",
},
}
MEDIA_URL = "https://pybazaar.fra1.cdn.digitaloceanspaces.com/pybazaar/media/"The implementation of get_secrets() depends on your needs. You can import values from environment variables or from a JSON or INI file. Here's a version I am using in my projects. Just make sure not to include your secrets in the Git repository.
MEDIA_ROOT is not participating in the game of django-storages.
For static files, I recommend using File System or Whitenoise, althought theoretically you could also use django-storages for them too.
Try connecting your local environment to spaces at first, then check it remotely.
If you use a rich-text editor supporting images or links to documents, you should ensure that the media paths don't change when you dump your pages from production to development. Otherwise, the images and links will be broken. To ensure that, set MEDIA_URL to "/pybazaar/media/" locally when you use the file system storage locally again.
Use the Django Storage operations instead of file-system operations for all your media file operations: listing directory contents, creating a file, reading a file, updating file content, deleting a file, or checking the existence of a file.
For example, instead of this code:
import os
from django.conf import settings
with open(os.path.join(settings.MEDIA_ROOT, "README.txt"), "w") as f:
f.write("Hello, World!")use this one:
from django.core.files.storage import default_storage
from django.core.files.base import ContentFile
default_storage.save("README.txt", ContentFile("Hello, World!"))If you are using django-imagekit or another image manipulation library, make sure that you use the latest version that supports the STORAGES setting.
For older Django versions (< 4.2), you might need to set AWS_* settings instead of the STORAGES dictionary. See the django-packages docs here.
You can upload files for Spaces one-by-one at DigitalOcean console. Or better, use Transit v5 on MacOS.
Add new connection and choose Amazon S3
Upload some files there. Check if they are accessible from the CDN endpoint URL.
To have a dedicated subdomain of yours pointing to the spaces, e.g. https://media.pybazaar.com, you can follow the instructions in this article.
If you manage your DNS settings on DigitalOcean, it's not a big deal: you just click a few buttons to set the CNAME record and enable Let's encrypt certificates.
But if you manage your DNS elsewhere, you will have to manually set and regularly update your SSL settings, and set a CNAME record to point your subdomain (e.g. media.pybazaar.com) to their CDN endpoint (e.g. pybazaar.fra1.cdn.digitaloceanspaces.com).
Using django-storages you can relatively simply replace your file system with cloud-based storage such as Spaces Object Storage at DigitalOcean.
DigitalOcean gives $200 in credit over 60 days for everyone who will use my affiliate link to signup. If you are planning a website with lots of media files, it's worthy to have a try.
Cover photo by gdtography
When you are building websites with Django, there is no need to track their software versions. You can just have a stable branch in Git repository and update the production environment whenever it makes sense. However, when you create a Django app or package shared among various websites and maybe with multiple people, it is vital to keep track of the package versions.
Versioning allows you to identify a specific state of your package and has these benefits:
The most popular versioning scheme is called semantic versioning. The version consists of three numbers separated by dots. Django framework is using it too. For example, the latest stable version at the time of writing is 3.2.9. In semantic versioning, the first number is the major version, the second is the minor version, and the third is a patch version: MAJOR.MINOR.PATCH.
MAJOR version introduces backward incompatible features.MINOR version adds functionality that is backward compatible.PATCH version is for backward-compatible bug fixes.There can also be some versioning outliers for alpha, beta, release candidates rc1, rc2, etc., but we will not cover them in this post.
By convention, the package's version is saved in its myapp/__init__.py file as __version__ variable, for example:
__version__ = "0.2.4"In addition, it is saved in setup.py, README.md, CHANGELOG.md, and probably some more files.
When you develop a Django app or another Python package that you will share with other developers, it is also recommended to have a changelog. Usually, it's a markdown-based document with the list of features, bugs, and deprecations that were worked on between specific versions of your package.
A good changelog starts with the "Unreleased" section and is followed by sections for each version in reverse order. In each of those sections, there are lists of changes grouped into categories:
This could be the starting template for CHANGELOG.md:
Changelog
=========
All notable changes to this project will be documented in this file.
The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
[Unreleased]
------------
<!--
### Added
### Changed
### Deprecated
### Removed
### Fixed
### Security
-->As the new version is released, it will replace the Unreleased section while creating a new Unreleased section above it. For example, this:
[Unreleased]
------------
### Added
- Logging the cookie consent choices in the database because of the legal requirements.
### Fixed
- Disabling the buttons while saving the Cookie Choices so that they are not triggered more than once with slow Internet connections.
would become this:
[Unreleased]
------------
[v0.2.0] - 2021-10-27
------------------
### Added
- Logging the cookie consent choices in the database because of the legal requirements.
### Fixed
- Disabling the buttons while saving the Cookie Choices so that they are not triggered more than once with slow Internet connections.
To keep track of the versions manually would be pretty tedious work, with the likelihood of forgetting one or more files or mismatching the version numbers. Gladly, there is a utility tool to do that for you, which is called bump2version.
Install the bump2version utility to your virtual environment the standard way with pip:
(env)$ pip install bump2versionIn your package's __init__.py file, set the version to 0.0.0:
__version__ = "0.0.0"Set the version to 0.0.0 in all other files, where the version needs to be mentioned, for example, README.md and setup.py.
Then create setup.cfg with the following content:
[bumpversion]
current_version = 0.0.0
commit = True
tag = True
[bumpversion:file:setup.py]
search = version="{current_version}"
replace = version="{new_version}"
[bumpversion:file:myapp/__init__.py]
search = __version__ = "{current_version}"
replace = __version__ = "{new_version}"
[bumpversion:file:README.md]
search = django_myapp-{current_version}-py2.py3-none-any.whl
replace = django_myapp-{new_version}-py2.py3-none-any.whl
[bumpversion:file:CHANGELOG.md]
search =
[Unreleased]
------------
replace =
[Unreleased]
------------
[v{new_version}] - {utcnow:%%Y-%%m-%%d}
------------------
[bdist_wheel]
universal = 1Replace "django_myapp" and "myapp" with your Django app there. If you have more files, mentioning the package version, make sure to have analogous version replacements in the setup.cfg.
Commit and push your changes to the Git repository.
Every time you want to create a new version, type this in the shell:
$ bump2version patchor
$ bump2version minoror
$ bump2version majorfollowed by the command to build the package:
$ python3 setup.py sdist bdist_wheelAs mentioned before, patch is for the bug fixes, minor is for backward-compatible changes, and major is for backward-incompatible changes.
The bump2version command will use the configuration at setup.cfg and will do these things:
setup.py, myapp/__init__.py, and README.md.CHANGELOG.md.vMAJOR.MINOR.PATCH there.There are two things to note there.
First, in Markdown, there are two ways of setting the headings:
Changelog
=========
[Unreleased]
------------is identical to
# Changelog
## [Unreleased]Usually, I would use the second format for Markdown, but the replacement function in setup.cfg treats lines with the # symbol as comments, and escaping doesn't work either.
For example, this wouldn't work:
[bumpversion:file:CHANGELOG.md]
search =
## [Unreleased]
replace =
## [Unreleased]
## [v{new_version}] - {utcnow:%%Y-%%m-%%d}If anybody knows or finds a simple solution with the shorter Markdown variation, let me know.
Second, instead of using setup.cfg, you can use .bumpversion.cfg, but that file is hidden, so I recommend sticking with setup.cfg.
You can see this type of semantic versioning configuration in the paid Django GDPR Cookie Consent app I recently published on Gumroad.
Happy programming!
Cover photo by Jacob Campbell
Once you have a working project, you have to host it somewhere. One of the most popular deployment platforms nowadays is Heroku. Heroku belongs to a Platform as a Service (PaaS) category of cloud computing services. Every Django project you host on Heroku is running inside a smart container in a fully managed runtime environment. Your project can scale horizontally (adding more computing machines) and you pay for what you use starting with a free tier. Moreover, you won't need much of system administrator's skills to do the deployment - once you do the initial setup, the further deployment is as simple as pushing Git repository to a special heroku remote.
However, there are some gotchas to know before choosing Heroku for your Django project:
Recently I deployed a small Django project on Heroku. To have a quick reference for the future, I summed up the process here providing instructions how to do that for future reference.
Sign up for a Heroku account. Then install Heroku tools for doing all the deployment work in the shell.
To connect your shell with Heroku, type:
$ heroku loginWhen asked, enter your Heroku account's email and password.
Activate your project's virtual environment and install Python packages required for Heroku:
(myproject_env)$ pip install django-toolbeltThis will install django, psycopg2, gunicorn, dj-database-url, static3, and dj-static to your virtual environment.
Install boto and Django Storages to be able to store static and media files on an S3 bucket:
(myproject_env)$ pip install boto
(myproject_env)$ pip install django-storages
Go to your project's directory and create the pip requirements that Heroku will use in the cloud for your project:
(myproject_env)$ pip freeze -l > requirements.txtYou will need two files to tell Heroku what Python version to use and how to start a webserver.
In your project's root directory create a file named runtime.txt with the following content:
python-2.7.11Then at the same location create a file named Procfile with the following content:
web: gunicorn myproject.wsgi --log-file -As mentioned in the "Web Development with Django Cookbook - Second Edition", we keep the developmnent and production settings in separate files both importing the common settings from a base file.
Basically we have myproject/conf/base.py with the settings common for all environments.
Then myproject/conf/dev.py contains the local database and dummy email configuration as follows:
# -*- coding: UTF-8 -*-
from __future__ import unicode_literals
from .base import *
DATABASES = {
"default": {
"CONN_MAX_AGE": 0,
"ENGINE": "django.db.backends.postgresql",
"HOST": "localhost",
"NAME": "myproject",
"PASSWORD": "",
"PORT": "",
"USER": "postgres"
}
}
EMAIL_BACKEND = "django.core.mail.backends.console.EmailBackend"Lastly for the production settings we need myproject/conf/prod.py with special database configuration, non-debug mode, and unrestrictive allowed hosts as follows:
# -*- coding: UTF-8 -*-
from __future__ import unicode_literals
from .base import *
import dj_database_url
DATABASES = {
"default": dj_database_url.config()
}
ALLOWED_HOSTS = ["*"]
DEBUG = FalseNow let's open myproject/settings.py and add the following content:
# -*- coding: UTF-8 -*-
from __future__ import unicode_literals
from .conf.dev import *
Finally, open the myproject/wsgi.py and change the location of the DJANGO_SETTINGS_MODULE there:
os.environ.setdefault("DJANGO_SETTINGS_MODULE", "myproject.conf.prod")
Create an Amazon S3 bucket myproject.media at the AWS Console (web interface for Amazon Web Services). Go to the properties of the bucket, expand "Permissions" section, click on the "add bucket policy" button and enter the following:
{
"Version": "2008-10-17",
"Statement": [
{
"Sid": "AllowPublicRead",
"Effect": "Allow",
"Principal": {
"AWS": "*"
},
"Action": "s3:GetObject",
"Resource": "arn:aws:s3:::myproject.media/*"
}
]
}This ensures that files on the S3 bucket will be accessible publicly without any API keys.
Go back to your Django project and add storages to the INSTALLED_APPS in myproject/conf/base.py:
INSTALLED_APPS = [
# ...
"storages",
]
Media files and static files will be stored on different paths under S3 bucket. To implement that, we need to create two Python classes under a new file myproject/s3utils.py as follows:
# -*- coding: UTF-8 -*-
from __future__ import unicode_literals
from storages.backends.s3boto import S3BotoStorage
class StaticS3BotoStorage(S3BotoStorage):
"""
Storage for static files.
"""
def __init__(self, *args, **kwargs):
kwargs['location'] = 'static'
super(StaticS3BotoStorage, self).__init__(*args, **kwargs)
class MediaS3BotoStorage(S3BotoStorage):
"""
Storage for uploaded media files.
"""
def __init__(self, *args, **kwargs):
kwargs['location'] = 'media'
super(MediaS3BotoStorage, self).__init__(*args, **kwargs)
Finally, let's edit the myproject/conf/base.py and add AWS settings:
AWS_S3_SECURE_URLS = False # use http instead of https
AWS_QUERYSTRING_AUTH = False # don't add complex authentication-related query parameters for requests
AWS_S3_ACCESS_KEY_ID = "..." # Your S3 Access Key
AWS_S3_SECRET_ACCESS_KEY = "..." # Your S3 Secret
AWS_STORAGE_BUCKET_NAME = "myproject.media"
AWS_S3_HOST = "s3-eu-west-1.amazonaws.com" # Change to the media center you chose when creating the bucket
STATICFILES_STORAGE = "myproject.s3utils.StaticS3BotoStorage"
DEFAULT_FILE_STORAGE = "myproject.s3utils.MediaS3BotoStorage"
# the next monkey patch is necessary to allow dots in the bucket names
import ssl
if hasattr(ssl, '_create_unverified_context'):
ssl._create_default_https_context = ssl._create_unverified_context
Collect static files to the S3 bucket:
(myproject_env)$ python manage.py collectstatic --noinputOpen myproject/conf/prod.py and add the following settings:
EMAIL_USE_TLS = True
EMAIL_HOST = "smtp.gmail.com"
EMAIL_HOST_USER = "myproject@gmail.com"
EMAIL_HOST_PASSWORD = "mygmailpassword"
EMAIL_PORT = 587Commit and push all the changes to your Git origin remote. Personally I prefer using SourceTree to do that, but you can also do that in the command line, PyCharm, or another software.
In your project directory type the following:
(myproject_env)$ heroku create my-unique-projectThis will create a Git remote called "heroku", and a new Heroku project "my-unique-project" which can be later accessed at http://my-unique-project.herokuapp.com.
Push the changes to heroku remote:
(myproject_env)$ git push heroku masterCreate local database dump:
(myproject_env)$ PGPASSWORD=mypassword pg_dump -Fc --no-acl --no-owner -h localhost -U myuser mydb > mydb.dumpUpload the database dump temporarily to some server, for example, S3 bucket: http://myproject.media.s3-eu-west-1.amazonaws.com/mydb.dump. Then import that dump into the Heroku database:
(myproject_env)$ heroku pg:backups restore 'http://myproject.media.s3-eu-west-1.amazonaws.com/mydb.dump' DATABASE_URLRemove the database dump from S3 server.
If your Git repository is not private, put your secret values in environment variables rather than in the Git repository directly.
(myproject_env)$ heroku config:set AWS_S3_ACCESS_KEY_ID=ABCDEFG123
$ heroku config:set AWS_S3_SECRET_ACCESS_KEY=aBcDeFg123
To read out the environment variables you can type:
(myproject_env)$ heroku config
To read out the environment variables in the Python code open myproject/conf/base.py and type:
import os
AWS_S3_ACCESS_KEY_ID = os.environ.get("AWS_S3_ACCESS_KEY_ID", "")
AWS_S3_SECRET_ACCESS_KEY = os.environ.get("AWS_S3_SECRET_ACCESS_KEY", "")
Open your domain settings and set CNAME to "my-unique-project.herokuapp.com".
At last, you are done! Drop in the comments if I missed some part. For the new updates, see the next section.
Push the changes to heroku remote:
(myproject_env)$ git push heroku masterIf you have changed something in the static files, collect them again:
(myproject_env)$ python manage.py collectstatic --noinputCollecting static files to S3 bucket takes quite a long time, so I do not recommend to do that automatically every time when you want to deploy to Heroku.
You can read more about Django on Heroku in the following resources:
Cover photo by Frances Gunn
