As you might know, I have been developing, providing, and supporting the prioritization tool 1st things 1st. One of the essential features to implement was exporting calculated priorities to other productivity tools. Usually, building an export from one app to another takes 1-2 weeks for me. But this time, I decided to go a better route and use Zapier to export priorities to almost all possible apps in a similar amount of time. Whaaat!?? In this article, I will tell you how.
The no-code tool Zapier takes input from a wide variety of web apps and outputs it to many other apps. Optionally you can filter the input based on conditions. Or format the input differently (for example, convert HTML to Markdown). In addition, you can stack the output actions one after the other. Usually, people use 2-3 steps for their automation, but there are power users who create 50-step workflows.
The input is managed by Zapier's triggers. The output is controlled by Zapier's actions. These can be configured at the website UI or using a command-line tool. I used the UI as this was my first integration. Trigger events accept a JSON feed of objects with unique IDs. Each new item there is treated as a new input item. With a free tier, the triggers are checked every 15 minutes. Multiple triggers are handled in parallel, and the sorting order of execution is not guaranteed. As it is crucial to have the sorting order correct for 1st things 1st priorities, people from Zapier support suggested providing each priority with a 1-minute interval to make sure the priorities get listed in the target app sequentially.
The most challenging part of Zapier integration was setting up OAuth 2.0 provider. Even though I used a third-party Django app django-oauth-toolkit for that. Zapier accepts other authentication options too, but this one is the least demanding for the end-users.
OAuth 2.0 allows users of one application to use specific data of another application while keeping private information private. You might have used the OAuth 2.0 client directly or via a wrapper for connecting to Twitter apps. For Zapier, one has to set OAuth 2.0 provider.
The official tutorial for setting up OAuth 2.0 provider with django-oauth-toolkit
is a good start. However, one problem with it is that by default, any registered user can create OAuth 2.0 applications at your Django website, where in reality, you need just one global application.
First of all, I wanted to allow OAuth 2.0 application creation only for superusers.
For that, I created a new Django app oauth2_provider_adjustments
with modified views and URLs to use instead of the ones from django-oauth-toolkit
.
The views related to OAuth 2.0 app creation extended this SuperUserOnlyMixin
instead of LoginRequiredMixin
:
from django.contrib.auth.mixins import AccessMixin
class SuperUserOnlyMixin(AccessMixin):
def dispatch(self, request, *args, **kwargs):
if not request.user.is_superuser:
return self.handle_no_permission()
return super().dispatch(request, *args, **kwargs)
Then I replaced the default oauth2_provider
URLs:
urlpatterns = [
# …
path("o/", include("oauth2_provider.urls", namespace="oauth2_provider")),
]
with my custom ones:
urlpatterns = [
# …
path("o/", include("oauth2_provider_adjustments.urls", namespace="oauth2_provider")),
]
I set the new OAuth 2.0 application by going to /o/applications/register/
and filling in this info:
Name: Zapier
Client type: Confidential
Authorization grant type: Authorization code
Redirect uris: https://zapier.com/dashboard/auth/oauth/return/1stThings1stCLIAPI/
(copied from Zapier)
Algorithm: No OIDC support
If you have some expertise in the setup choices and see any flaws, let me know.
Zapier requires creating a test view that will return anything to check if there are no errors authenticating a user with OAuth 2.0. So I made a simple JSON view like this:
from django.http.response import JsonResponse
def user_info(request, *args, **kwargs):
if not request.user.is_authenticated:
return JsonResponse(
{
"error": "User not authenticated",
},
status=200,
)
return JsonResponse(
{
"first_name": request.user.first_name,
"last_name": request.user.last_name,
},
status=200,
)
Also, I had to have login and registration views for those cases when the user's session was not present.
Lastly, at Zapier, I had to set these values for OAuth 2.0:
Client ID: The Client ID from registered app
Client Secret: The Client Secret from registered app
Authorization URL: https://apps.1st-things-1st.com/o/authorize/
Scope: read write
Access Token Request: https://apps.1st-things-1st.com/o/token/
Refresh Token Request: https://apps.1st-things-1st.com/o/token/
I want to automatically refresh on unauthorized error:
Checked
Test: https://apps.1st-things-1st.com/user-info/
Connection Label: {{first_name}} {{last_name}}
There are two types of triggers in Zapier:
The feeds for triggers should (ideally) be paginated. But without meta information for the item count, page number, following page URL, etc., you would usually have with django-rest-framework
or other REST frameworks. Provide only an array of objects with unique IDs for each page. The only field name that matters is "id" – others can be anything. Here is an example:
[
{
"id": "39T7NsgQarYf",
"project": "5xPrQbPZNvJv",
"title": "01. Custom landing pages for several project types (83%)",
"plain_title": "Custom landing pages for several project types",
"description": "",
"score": 83,
"priority": 1,
"category": "Choose"
},
{
"id": "4wBSgq3spS49",
"project": "5xPrQbPZNvJv",
"title": "02. Zapier integration (79%)",
"plain_title": "Zapier integration",
"description": "",
"score": 79,
"priority": 2,
"category": "Choose"
},
{
"id": "6WvwwB7QAnVS",
"project": "5xPrQbPZNvJv",
"title": "03. Electron.js desktop app for several project types (42%)",
"plain_title": "Electron.js desktop app for several project types",
"description": "",
"score": 41,
"priority": 3,
"category": "Consider"
}
]
The feeds should list items in reverse order for the (A) type of triggers: the newest things go at the beginning. The pagination is only used to cut the number of items: the second and further pages of the paginated list are ignored by Zapier.
In my specific case of priorities, the order matters, and no items should be lost in the void. So I listed the priorities sequentially (not newest first) and set the number of items per page unrealistically high so that you basically get all the things on the first page of the feed.
The feeds for the triggers of (B) type are normally paginated from the first page until the page returns empty results. The order should be alphabetical, chronological, or by sorting order field, whatever makes sense. There you need just two fields, the ID and the title of the item (but more fields are allowed too), for example:
[
{
"id": "5xPrQbPZNvJv",
"title": "1st things 1st",
"owner": "Aidas Bendoraitis"
},
{
"id": "VEXGzThxL6Sr",
"title": "Make Impact",
"owner": "Aidas Bendoraitis"
},
{
"id": "WoqQbuhdUHGF",
"title": "DjangoTricks website",
"owner": "Aidas Bendoraitis"
},
]
I used django-rest-framework
to implement the API because of the batteries included, such as browsable API, permissions, serialization, pagination, etc.
For the specific Zapier requirements, I had to write a custom pagination class, SimplePagination
, to use with my API lists. It did two things: omitted the meta section and showed an empty list instead of a 404 error for pages that didn't have any results:
from django.core.paginator import InvalidPage
from rest_framework.pagination import PageNumberPagination
from rest_framework.response import Response
class SimplePagination(PageNumberPagination):
page_size = 20
def get_paginated_response(self, data):
return Response(data) # <-- Simple pagination without meta
def get_paginated_response_schema(self, schema):
return schema # <-- Simple pagination without meta
def paginate_queryset(self, queryset, request, view=None):
"""
Paginate a queryset if required, either returning a
page object, or `None` if pagination is not configured for this view.
"""
page_size = self.get_page_size(request)
if not page_size:
return None
paginator = self.django_paginator_class(queryset, page_size)
page_number = self.get_page_number(request, paginator)
try:
self.page = paginator.page(page_number)
except InvalidPage as exc:
msg = self.invalid_page_message.format(
page_number=page_number, message=str(exc)
)
return [] # <-- If no items found, don't raise NotFound error
if paginator.num_pages > 1 and self.template is not None:
# The browsable API should display pagination controls.
self.display_page_controls = True
self.request = request
return list(self.page)
To preserve the order of items, I had to make the priorities appear one by one at 1-minute intervals. I did that by having a Boolean field exported_to_zapier
at the priorities. The API showed priorities only if that field was set to True
, which wasn't the case by default. Then, background tasks were scheduled 1 minute after each other, triggered by a button click at 1st things 1st, which set the exported_to_zapier
to True
for each next priority. I was using huey
, but the same can be achieved with Celery
, cron jobs, or other background task manager:
# zapier_api/tasks.py
from django.conf import settings
from django.utils.translation import gettext
from huey.contrib.djhuey import db_task
@db_task()
def export_next_initiative_to_zapier(project_id):
from evaluations.models import Initiative
next_initiatives = Initiative.objects.filter(
project__pk=project_id,
exported_to_zapier=False,
).order_by("-total_weight", "order")
count = next_initiatives.count()
if count > 0:
next_initiative = next_initiatives.first()
next_initiative.exported_to_zapier = True
next_initiative.save(update_fields=["exported_to_zapier"])
if count > 1:
result = export_next_initiative_to_zapier.schedule(
kwargs={"project_id": project_id},
delay=settings.ZAPIER_EXPORT_DELAY,
)
result(blocking=False)
One gotcha: Zapier starts pagination from 0, whereas django-rest-framework
starts pagination from 1. To make them work together, I had to modify the API request (written in JavaScript) at Zapier trigger configuration:
const options = {
url: 'https://apps.1st-things-1st.com/api/v1/projects/',
method: 'GET',
headers: {
'Accept': 'application/json',
'Authorization': `Bearer ${bundle.authData.access_token}`
},
params: {
'page': bundle.meta.page + 1 // <-- The custom line for pagination
}
}
return z.request(options)
.then((response) => {
response.throwForStatus();
const results = response.json;
// You can do any parsing you need for results here before returning them
return results;
});
For the v1 of Zapier integration, I didn't need any Zapier actions, so they are yet something to explore, experiment with, and learn about. But the Zapier triggers seem already pretty helpful and a big win compared to individual exports without this tool.
If you want to try the result, do this:
Cover photo by Anna Nekrashevich
When you develop a web app or a mobile app with Django, it is common to use the Django REST Framework for communication with the server-side. The client-side makes GET, POST, PUT, and DELETE requests to the REST API to read, create, update, or delete data there. The communication by Ajax is pretty uncomplicated, but how would you upload an image or another file to the server? I will show you that in this article by creating user avatar upload via REST API. Find the full code for this feature on Github.
We will start by installing Pillow for image handling to the virtual environment using the standard pip command:
(venv)$ pip install Pillow
Create accounts
app with a custom User
model:
# myproject/apps/accounts/models.py
import os
import sys
from django.db import models
from django.contrib.auth.models import AbstractUser
from django.utils import timezone
from django.utils.translation import gettext_lazy as _
def upload_to(instance, filename):
now = timezone.now()
base, extension = os.path.splitext(filename.lower())
milliseconds = now.microsecond // 1000
return f"users/{instance.pk}/{now:%Y%m%d%H%M%S}{milliseconds}{extension}"
class User(AbstractUser):
# …
avatar = models.ImageField(_("Avatar"), upload_to=upload_to, blank=True)
You can add there as many fields as you need, but the noteworthy part there is the avatar
field.
Update the settings and add the accounts
app to INSTALLED_APPS
, set the AUTH_USER_MODEL
, and the configuration for the static and media directories:
# myproject/settings.py
INSTALLED_APPS = [
# …
"myproject.apps.accounts",
]
AUTH_USER_MODEL = "accounts.User"
STATICFILES_DIRS = [os.path.join(BASE_DIR, "myproject", "site_static")]
STATIC_ROOT = os.path.join(BASE_DIR, "myproject", "static")
STATIC_URL = "/static/"
MEDIA_ROOT = os.path.join(BASE_DIR, "myproject", "media")
MEDIA_URL = "/media/"
Next small steps:
makemigrations
and migrate
management commands.createsuperuser
management command.Install Django REST Framework for the REST APIs to your virtual environment, as always, using pip:
(venv)$ pip install djangorestframework
We'll be using authentication by tokens in this example. So add Django REST Framework to INSTALLED_APPS
in the settings and set TokenAuthentication
as the default authentication in the REST_FRAMEWORK
configuration:
# myproject/settings.py
INSTALLED_APPS = [
# …
"rest_framework",
"rest_framework.authtoken",
# …
]
REST_FRAMEWORK = {
'DEFAULT_AUTHENTICATION_CLASSES': [
'rest_framework.authentication.TokenAuthentication',
]
}
In Django REST Framework, serializers are used for data validation, rendering, and saving. They are similar to Django forms. Prepare UserAvatarSerializer
for avatar uploads:
# myproject/apps/accounts/serializers.py
from django.contrib.auth import get_user_model
from rest_framework.serializers import ModelSerializer
User = get_user_model()
class UserAvatarSerializer(ModelSerializer):
class Meta:
model = User
fields = ["avatar"]
def save(self, *args, **kwargs):
if self.instance.avatar:
self.instance.avatar.delete()
return super().save(*args, **kwargs)
Now create an API view UserAvatarUpload
for avatar uploads.
# myproject/apps/accounts/views.py
from rest_framework import status
from rest_framework.parsers import MultiPartParser, FormParser
from rest_framework.permissions import IsAuthenticated
from rest_framework.response import Response
from rest_framework.views import APIView
from .serializers import UserAvatarSerializer
class UserAvatarUpload(APIView):
parser_classes = [MultiPartParser, FormParser]
permission_classes = [IsAuthenticated]
def post(self, request, format=None):
serializer = UserAvatarSerializer(data=request.data, instance=request.user)
if serializer.is_valid():
serializer.save()
return Response(serializer.data, status=status.HTTP_200_OK)
else:
return Response(serializer.errors, status=status.HTTP_400_BAD_REQUEST)
Make sure that the view uses MultiPartParser
as one of the parser classes. That's necessary for the file transfers.
In the URL configuration, we will need those URL rules:
TemplateView
.# myroject/urls.py
from django.conf.urls.static import static
from django.contrib import admin
from django.urls import path
from django.views.generic import TemplateView
from django.conf import settings
from myproject.accounts.views import UserAvatarUpload
from rest_framework.authtoken.views import obtain_auth_token
urlpatterns = [
path("", TemplateView.as_view(template_name="index.html")),
path("api/auth-token/", obtain_auth_token, name="rest_auth_token"),
path("api/user-avatar/", UserAvatarUpload.as_view(), name="rest_user_avatar_upload"),
path("admin/", admin.site.urls),
]
urlpatterns += static(settings.STATIC_URL, document_root=settings.STATIC_ROOT)
urlpatterns += static(settings.MEDIA_URL, document_root=settings.MEDIA_ROOT)
I will illustrate the frontend using Bootstrap HTML and Vanilla JavaScript. Of course, you can implement the same using ReactJS, Vue, Angular, or other JavaScript framework and any other CSS framework.
The template for the index page has one login form with username and password or email and password fields (depending on your implementation), and one avatar upload form with a file selection field. Also, it includes a JavaScript file avatar.js
for Ajax communication.
{# myproject/templates/index.html #}
<!doctype html>
{% load static %}
<html lang="en">
<head>
<meta charset="utf-8">
<meta name="viewport" content="width=device-width, initial-scale=1, shrink-to-fit=no">
<link rel="stylesheet" href="https://stackpath.bootstrapcdn.com/bootstrap/4.4.1/css/bootstrap.min.css"
integrity="sha384-Vkoo8x4CGsO3+Hhxv8T/Q5PaXtkKtu6ug5TOeNV6gBiFeWPGFN9MuhOf23Q9Ifjh" crossorigin="anonymous">
<title>Hello, World!</title>
</head>
<body>
<div class="container">
<div class="row">
<div class="col-md-8">
<p class="text-muted my-3"><small>Open Developer Console for information about responses.</small></p>
<h1 class="my-3">1. Log in</h1>
<form id="login_form">
<div class="form-group">
<label for="id_email">Email address</label>
<input type="email" class="form-control" id="id_email" aria-describedby="emailHelp"
placeholder="Enter email"/>
</div>
<div class="form-group">
<label for="id_password">Password</label>
<input type="password" class="form-control" id="id_password" placeholder="Password"/>
</div>
<button type="submit" class="btn btn-primary">Log in</button>
</form>
<h1 class="my-3">2. Upload an avatar</h1>
<form id="avatar_form">
<div class="form-group">
<label for="id_avatar">Choose an image for your avatar</label>
<input type="file" class="form-control-file" id="id_avatar"/>
</div>
<button type="submit" class="btn btn-primary">Upload</button>
</form>
</div>
</div>
</div>
<script src="{% static 'site/js/avatar.js' %}"></script>
</body>
</html>
Last but not least, create the JavaScript file avatar.js
. It contains these things:
// myproject/site_static/site/js/avatar.js
let userToken;
document.getElementById('login_form').addEventListener('submit', function(event) {
event.preventDefault();
let email = document.getElementById('id_email').value;
let password = document.getElementById('id_password').value;
fetch('http://127.0.0.1:8000/api/auth-token/', {
method: 'POST',
headers: {
'Content-Type': 'application/json',
},
body: JSON.stringify({
"username": email,
"password": password,
})
}).then( response => {
return response.json();
}).then(data => {
console.log(data);
userToken = data.token;
console.log('Logged in. Got the token.');
}).catch((error) => {
console.error('Error:', error);
});
});
document.getElementById('avatar_form').addEventListener('submit', function(event) {
event.preventDefault();
let input = document.getElementById('id_avatar');
let data = new FormData();
data.append('avatar', input.files[0]);
fetch('http://127.0.0.1:8000/api/user-avatar/', {
method: 'POST',
headers: {
'Authorization': `Token ${userToken}`
},
body: data
}).then(response => {
return response.json();
}).then(data => {
console.log(data);
}).catch((error) => {
console.error('Error:', error);
});
});
In the JavaScript file, we are using fetch API for the REST API requests. The noteworthy part there is the FormData
class that we use to send the file to the server.
Now run the local development server and go to the http://127.0.0.1:8000
. There you will have something like this:
As more than a half Internet usage happens on mobile devices, there is a demand to switch from usual HTML websites and platforms to mobile apps. Whether you create a native mobile app, a hybrid app, or Progressive Web App, you will likely have to communicate with the server via REST API or GraphQL. It is pretty clear how to transfer textual data from and to a remote server. But after this exercise, we can also transfer binary files like images, PDF or Word documents, music, and videos.
Happy coding!
Cover Photo by Dan Silva