Connexion
I spend more time today than ever before interacting with terminal windows, which is something I don't think Past Me would have believed in the early '90s. Back then, poor MS-DOS was the staid whipping boy of the industry, and at least on the consumer side, graphical environments like Windows (and maybe even odder creatures like AmigaOS) seemed poised to stamp the command line into oblivion, leaving text interfaces behind as we all blasted into the ooey-GUI future.
As it turns out, though, the command line is still the best tool for some jobs—many jobs, in fact. I read a wise post some years ago (probably on Slashdot) arguing that a mouse-driven point-and-click interface essentially reduces the user to pointing at something on the screen and grunting, "DO! DO THAT!" at the computer. (The rise of right-click context menus adds the ability for the user to also grunt "MORE THINGS!" but doesn't otherwise add vocabulary.)
The command line, by contrast, gives the user the opportunity to precisely tell the computer what they want done, using words instead of one or two gestalts that the computer must interpret based on context.
Sed is an acronym for "stream editor." A stream refers to a source or destination for bytes. In other words, sed can read its input from standard input (stdin), apply the specified edits to the stream, and automatically output the results to standard output (stdout). Sed syntax allows an input file to be specified on the command line. However, the syntax does not directly support output file specification; this can be achieved through output redirection or editing files in place while making a backup of the original copy optionally. Sed is one of the most powerful tools on Linux and Unix-like systems. Learning it is worthwhile, so in this tutorial, we will start with the sed command syntax and examples.
Love this? sudo share_on: Twitter - Facebook - LinkedIn - Whatsapp - Reddit
The post Linux Sed Tutorial: Learn Text Editing with Syntax & Examples appeared first on nixCraft.
How do I change MySQL root password under Linux, FreeBSD, OpenBSD and UNIX-like like operating system over the ssh session?
Love this? sudo share_on: Twitter - Facebook - LinkedIn - Whatsapp - Reddit
The post MySQL Change root Password Command appeared first on nixCraft.
Most Linux and UNIX-like system configuration files are documented using comments, but sometimes I just need to see a line of configuration text in a config file. How can I view just the uncommented configuration file directives from squid.conf or httpd.conf file? How can I strip out comments and blank lines on a Linux or Unix-like system?
Love this? sudo share_on: Twitter - Facebook - LinkedIn - Whatsapp - Reddit
The post Linux / UNIX View Only Configuration File Directives ( Uncommented Lines of a Config File ) appeared first on nixCraft.
How can I copy one hard disk to another using the dd, ddrescue, or dcfldd commands?
Love this? sudo share_on: Twitter - Facebook - LinkedIn - Whatsapp - Reddit
The post Linux Copy One Hard Disk to Another Using dd Command appeared first on nixCraft.
LXD or Incus is a Linux operating system-level container system. You can build over 25+ Linux distros for testing, fun, and profit. You can even run GUI apps inside those containers and get output displayed back to your X display easily. For example, you can run Firefox in one Linux container for banking needs with custom add-ons, including specific firewall rules just for allowing outgoing banking and nothing else. You can have another container to run Chromium/Edge/FF for social media. Of course, it is not a replacement for something like Qubes OS, but you can do many things in an isolated environment. You can try a new Linux distro without the VM's overheads and test new features offered by that distro. All Linux containers will share the same Linux kernel and hardware devices as your GPU. Here is a quick shell script to build a quick lab to test various Linux distros.
Love this? sudo share_on: Twitter - Facebook - LinkedIn - Whatsapp - Reddit
The post Shell script to set up an LXD / Incus (Linux Containers) lab for testing purpose appeared first on nixCraft.
Here is a quick and dirty shell script I put to check LXD or Incus container size and how much space they are taking on the BTRFS subvolume. Naturally, you must run the script as a root user, and LXD or Icnus must be configured with BTRFS storage backend on Linux operating systems. See how to set up and install LXD on Ubuntu 20.04 LTS or installing Incus on Debian 12/11 using the apt command.
Love this? sudo share_on: Twitter - Facebook - LinkedIn - Whatsapp - Reddit
The post How To check LXD/Incus container BTRFS disk usage on Linux appeared first on nixCraft.
Here is a handy and useful Linux and Unix shell script that reduce PDF file size using Ghostscript. No need to upload your PDF file to the shady third-party website. Just do it from the terminal. I tested it with both CentOS and Ubuntu/Debian Linux. It should work with macOS, FreeBSD and other Unix-like systems as long as you have the Ghostscript installed.
Love this? sudo share_on: Twitter - Facebook - LinkedIn - Whatsapp - Reddit
The post Linux shell script to reduce PDF file size appeared first on nixCraft.
TTL is an acronym for Time-To-Live (TTL) in DNS. It sets the time in seconds that a DNS record is allowed to be cached by DNS resolvers (caching server) before it needs to be fetched again from the authoritative name server. In other words, longer TTL can reduce the load on authoritative DNS servers and improve response times by keeping records in the cache longer. The shorter TTL is useful for frequently changing DNS records, as it ensures that updates are propagated quickly across the Internet. Say you want to see the Time-To-Live (TTL) value for a given DNS record for A, AAAA, and MX. Here is a sample shell script that works on Linux, Unix, and macOS. You must have the bash and dig command installed.
Love this? sudo share_on: Twitter - Facebook - LinkedIn - Whatsapp - Reddit
The post Shell script to see Time-To-Live (TTL) for a DNS record on Linux, Unix, macOS and FreeBSD appeared first on nixCraft.
Here is a simple shell script to notify secondary read-only replica (slave) MariaDB server health status failure via email and push notification. Master-slave data replication allows you to copy databases to multiple MariaDB servers. It is useful for backup, data recovery, load balancing, and much more. See how to configure SSL-protected MariaDB replication between a master and slave server.
Love this? sudo share_on: Twitter - Facebook - LinkedIn - Whatsapp - Reddit
The post Shell script to monitor MariaDB replication and send email alert about server health status appeared first on nixCraft.
How do I capture a specific protocol or port, such as 80 (HTTP) or 443 (HTTPS), using the TCPDump tool under Linux/UNIX? How do I record traffic with TCPDump and find problems later on with my network or server issues? Let's dive into the nitty-gritty of capturing and analyzing your network traffic for trapshooting network and server issues.
Love this? sudo share_on: Twitter - Facebook - LinkedIn - Whatsapp - Reddit
The post TCPDump: Capture and Record Specific Protocols / Port Traffic appeared first on nixCraft.
I am a new macOS Unix user. I am writing a small bash shell script. How do I download file directly from the command-line interface using curl? How can I download file with cURL on a Linux or Unix-like systems?
Love this? sudo share_on: Twitter - Facebook - LinkedIn - Whatsapp - Reddit
The post How to download a file with curl on Linux/Unix command line appeared first on nixCraft.
I need to copy or clone file ownership and permissions from another file on Linux. Is there a bash command line option to clone the user, group ownership and permissions on a file from another file on Linux operating system?
Love this? sudo share_on: Twitter - Facebook - LinkedIn - Whatsapp - Reddit
The post How to copy permissions from one file to another on Linux appeared first on nixCraft.
Recently, I moved the DjangoTricks website and started PyBazaar on servers with Nginx Unit. One thing that was left undone was SSL certificate renewals. Let's Encrypt has special certbot parameters for renewing certificates for websites on Apache or Nginx servers, but they don't work out of the box with the Nginx Unit. In this blog post, I will tell you how to do that.
Nginx Unit doesn't use the fullchain.pem and privkey.pem generated by certbot directly from the location where they were generated. Instead, one has to create a bundle (like bundle1.pem) by concatenating them and then uploading it to the Nginx Unit configuration endpoint.
For that, I created a bash script:
Lateo.net - Flux RSS en pagaille (pour en ajouter : @ moi) 
#!/usr/bin/env bash
SECONDS=0
CRON_LOG_FILE=/var/webapps/pybazaar/logs/renew_certificate.log
echo "=== Renewing Letsencrypt Certificate ===" > ${CRON_LOG_FILE}
date >> ${CRON_LOG_FILE}
echo "Renewing certificate..." >> ${CRON_LOG_FILE}
certbot --renew-by-default certonly -n --webroot -w /var/www/letsencrypt/ -m hello@pybazaar.com --agree-tos --no-verify-ssl -d pybazaar.com -d www.pybazaar.com
echo "Creating bundle..." >> ${CRON_LOG_FILE}
cat /etc/letsencrypt/live/pybazaar.com/fullchain.pem /etc/letsencrypt/live/pybazaar.com/privkey.pem > /var/webapps/pybazaar/unit-config/bundle1.pem
echo "Temporarily switching the Unit configuration to a dummy one..." >> ${CRON_LOG_FILE}
curl -X PUT --data-binary @/var/webapps/pybazaar/unit-config/unit-config-pre.json --unix-socket /var/run/control.unit.sock http://localhost/config
echo "Deleting old certificate from Nginx Unit..." >> ${CRON_LOG_FILE}
curl -X DELETE --unix-socket /var/run/control.unit.sock http://localhost/certificates/certbot1
echo "Installing new certificate to Nginx Unit..." >> ${CRON_LOG_FILE}
curl -X PUT --data-binary @/var/webapps/pybazaar/unit-config/bundle1.pem --unix-socket /var/run/control.unit.sock http://localhost/certificates/certbot1
echo "Switching the Unit configuration to the correct one..." >> ${CRON_LOG_FILE}
curl -X PUT --data-binary @/var/webapps/pybazaar/unit-config/unit-config.json --unix-socket /var/run/control.unit.sock http://localhost/config
echo "Restarting Unit..." >> ${CRON_LOG_FILE}
service unit restart
echo "Finished." >> ${CRON_LOG_FILE}
duration=$SECONDS
echo "$(($duration / 60)) minutes and $(($duration % 60)) seconds elapsed." >> ${CRON_LOG_FILE}Once you have adapted the script, you can run it manually as a root user to test it:
$ chmod +x renew_certificate.sh
$ ./renew_certificate.shNote that the certbot command will try to validate your website's URL by attempting to reach a temporary file that it will create on http://example.com/.well-known/acme-challenge/, so make sure that this location is accessible and serving the static files.
For more details about the Nginx Unit, check my previous blog post.
If everything works as expected, you can add it to the root user's cron jobs to be executed weekly.
Export the current root cron jobs to a crontab.txt:
$ crontab -l > crontab.txtThen edit it and add the weekly script to update the SSL certificate:
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/snap/bin
SHELL=/bin/bash
MAILTO=""
@weekly /var/webapps/pybazaar/unit-config/renew_certificate.shThen run the following as the root user to apply it:
$ crontab crontab.txtThe good thing about not editing the cron job with crontab -e is that you can choose the editor and even put the crontab.txt under Git version control.
Happy web development with WSGI or ASGI!
Cover picture by Gotta Be Worth It
Recently, I learned about the NGINX Unit and decided to try it on my DjangoTricks website. Unit is a web server developed by people from NGINX, with pluggable support for Python (WSGI and ASGI), Ruby, Node.js, PHP, and a few other languages. I wanted to see whether it's really easy to set it up, have it locally on my Mac and the remote Ubuntu server, and try out the ASGI features of Django, allowing real-time communication. Also, I wanted to see whether Django is faster with Unit than with NGINX and Gunicorn. This article is about my findings.
Unit service uses HTTP requests to read and update its configuration. The configuration is a single JSON file that you can upload to the Unit service via a command line from the same computer or modify its values by keys in the JSON structure.
Normally, the docs suggest using the curl command to update the configuration. However, as I am using Ansible to deploy my Django websites, I wanted to create a script I could later copy to other projects. I used Google Gemini to convert bash commands from the documentation to Ansible directives and corrected its mistakes.
The trickiest part for me was to figure out how to use Let's Encrypt certificates in the simplest way possible. The docs are extensive and comprehensible, but sometimes, they dig into technical details that are unnecessary for a common Django developer.
Also, it's worth mentioning that the Unit plugin version must match your Python version in the virtual environment. It was unexpected for me when Brew installed Python 3.12 with unit-python3 and then required my project to use Python 3.12 instead of Python 3.10 (which I used for the DjangoTricks website). So I had to recreate my virtual environment and probably will have problems later with pip-compile-multi when I prepare packages for the production server, still running Python 3.10.
Below are the instructions I used to set up the NGINX Unit with my existing DjangoTricks website on Ubuntu 22.04. For simplicity, I am writing plain Terminal commands instead of analogous Ansible directives.
Follow the installation instructions from documentation to install unit, unit-dev, unit-python3.10, and whatever other plugins you want. Make sure the service is running.
Create a temporary JSON configuration file /var/webapps/djangotricks/unit-config/unit-config-pre.json, which will allow Let's Encrypt certbot to access the .well-known directory for domain confirmation:
{
"listeners": {
"*:80": {
"pass": "routes/acme"
}
},
"routes": {
"acme": [
{
"match": {
"uri": "/.well-known/acme-challenge/*"
},
"action": {
"share": "/var/www/letsencrypt/$uri"
}
}
]
}
}Install it to Unit:
$ curl -X PUT --data-binary @/var/webapps/djangotricks/unit-config/unit-config-pre.json \
--unix-socket /var/run/control.unit.sock http://localhost/configIf you make any mistakes in the configuration, it will be rejected with an error message and not executed.
Create Let's Encrypt certificates:
$ certbot certonly -n --webroot -w /var/www/letsencrypt/ -m hello@djangotricks.com \
--agree-tos --no-verify-ssl -d djangotricks.com -d www.djangotricks.comCreate a bundle that is required by the NGINX Unit:
cat /etc/letsencrypt/live/djangotricks.com/fullchain.pem \
/etc/letsencrypt/live/djangotricks.com/privkey.pem > \
/var/webapps/djangotricks/unit-config/bundle1.pemInstall certificate to NGINX Unit as certbot1:
curl -X PUT --data-binary @/var/webapps/djangotricks/unit-config/bundle1.pem \
--unix-socket /var/run/control.unit.sock http://localhost/certificates/certbot1Create a JSON configuration file /var/webapps/djangotricks/unit-config/unit-config.json which will use your SSL certificate and will serve your Django project:
{
"listeners": {
"*:80": {
"pass": "routes/main"
},
"*:443": {
"pass": "routes/main",
"tls": {
"certificate": "certbot1"
}
}
},
"routes": {
"main": [
{
"match": {
"host": [
"djangotricks.com",
"www.djangotricks.com"
],
"uri": "/.well-known/acme-challenge/*"
},
"action": {
"share": "/var/www/letsencrypt/$uri"
}
},
{
"match": {
"host": [
"djangotricks.com",
"www.djangotricks.com"
],
},
"action": {
"pass": "applications/django"
}
},
{
"action": {
"return": 444
}
}
]
},
"applications": {
"django": {
"type": "python",
"path": "/var/webapps/djangotricks/project/djangotricks",
"home": "/var/webapps/djangotricks/venv/",
"module": "djangotricks.wsgi",
"environment": {
"DJANGO_SETTINGS_MODULE": "djangotricks.settings.production"
},
"user": "djangotricks",
"group": "users"
}
}
}In this configuration, HTTP requests can only be used for certification validation, and HTTPS requests point to the Django project if the domain used is correct. In other cases, the status "444 - No Response" is returned. (It's for preventing access for hackers who point their domains to your IP address).
In the NGINX Unit, switching between WSGI and ASGI is literally a matter of changing one letter from "w" to "a" in the line about the Django application module, from:
"module": "djangotricks.wsgi",to:
"module": "djangotricks.asgi",I could have easily served the static files in this configuration here, too, but my STATIC_URL contains a dynamic part to force retrieval of new files from the server instead of the browser cache. So, I used WhiteNoise to serve the static files.
For redirection from djangotricks.com to www.djangotricks.com, I also chose to use PREPEND_WWW = True setting instead of Unit directives.
And here, finally, installing it to Unit (it will overwrite the previous configuration):
$ curl -X PUT --data-binary @/var/webapps/djangotricks/unit-config/unit-config.json \
--unix-socket /var/run/control.unit.sock http://localhost/configDjangoTricks is a pretty small website; therefore, I couldn't do extensive benchmarks, but I checked two cases: how a filtered list view performs with NGINX and Gunicorn vs. NGINX Unit, and how you can replace NGINX, Gunicorn, and Huey background tasks with ASGI requests using NGINX Unit.
First of all, the https://www.djangotricks.com/tricks/?categories=development&technologies=django-4-2 returned the HTML result on average in 139 ms on NGINX with Gunicorn, whereas it was on average 140 ms with NGINX Unit using WSGI and 149 ms with NGINX Unit using ASGI. So, the NGINX Unit with WSGI is 0.72% slower than NGINX with Gunicorn, and the NGINX Unit with ASGI is 7.19% slower than NGINX with Gunicorn.
However, when I checked https://www.djangotricks.com/detect-django-version/ how it performs with background tasks and continuous Ajax requests until the result is retrieved vs. asynchronous checking using ASGI, I went on average from 6.62 s to 0.75 s. Of course, it depends on the timeout of the continuous Ajax request, but generally, a real-time ASGI setup can improve the user experience significantly.
UPDATE on 2024-09-15: After using ASGI with the NGINX Unit for a while, I noticed that it crashed several times, and the server had to be restarted. It's still unclear whether the issue was due to NGINX Unit instability, Django's ASGI implementation, or simply heavy load. So use ASGI at your own risk.
Although NGINX Unit with Python is slightly (unnoticeably) slower than NGINX with Gunicorn, it allows Django developers to use asynchronous requests and implement real-time user experience. Also, you could probably have a Django website and Matomo analytics or WordPress blog on the same server. The NGINX Unit configuration is relatively easy to understand, and you can script the process for reusability.
Cover Image by Volker Meyer.
I have a valuable tip for you on extracting the filename and extension in Bash when using Linux, macOS, FreeBSD, or Unix-like systems. It's quick and easy to follow.
Love this? sudo share_on: Twitter - Facebook - LinkedIn - Whatsapp - Reddit
The post How to get and extract filename extension in Bash appeared first on nixCraft.
Let us see how to check for ssh connectivity in a shell script running under macOS, FreeBSD, or Linux/Unix desktop using BatchMode=yes and ConnectTimeout=$VALUE options. This simple trick helps write a wrapper script that does other things, such as making a backup or running external command agents such as Salt or Ansible to do other tasks.
Love this? sudo share_on: Twitter - Facebook - LinkedIn - Whatsapp - Reddit
The post How to check for ssh connectivity in a shell script appeared first on nixCraft.
While working on the bash shell script wrapper, I needed to ensure that I only passed an integer in the Bash script under Linux and Unix-like systems. Here is how to check if a number is an integer in Bash Linux or Unix script or at the command-line interface (CLI)
Love this? sudo share_on: Twitter - Facebook - LinkedIn - Whatsapp - Reddit
The post Bash see if an input number is an integer or not appeared first on nixCraft.
Today I will talk about a common problem faced by many Linux or Unix sysadmins and developers. Typically modern apps and security devices heavily depend upon APIs and lists that feed data in JSON format. For example, a list of spamming or VPN CIDRs or a good list of verified bots' IP CIDRs is in JSON format. If they accept JSON format, you can take this list and upload it to your router or WAF. Unfortunately, not all devices accept JSON format. Hence, we can take JSON format and convert it to CSV format per our needs.
Love this? sudo share_on: Twitter - Facebook - LinkedIn - Whatsapp - Reddit
The post How to convert JSON to CSV using Linux / Unix shell appeared first on nixCraft.
So I have this specific use case where I get data in a comma-separated values (CSV) file. I pick up the company name, address, telephone, email, and some other data from that file. Once data is collected, I create a final pdf using a simple script. Unfortunately, data comes with unwanted leading and trailing white spaces in each field. So my final pdf file gets messed up because of those white space. Let us see how to trim leading and trailing white space for a string in Bash running on Linux or Unix machines.
Love this? sudo share_on: Twitter - Facebook - LinkedIn - Whatsapp - Reddit
The post How to trim leading and trailing white space in bash appeared first on nixCraft.
At a certain point, I needed to change the VLAN IP address such as 172.0.0.2 with 172.0.0.1 for ease of use. As you know, Linux and Unix-like operating system stores all IP addresses and config in plain text files. This page explains my little adventure where I had to find and find and replace ALL IP addresses with a new one on Linux.
Love this? sudo share_on: Twitter - Facebook - LinkedIn - Whatsapp - Reddit
The post How to find and replace ALL IP addresses with a new one appeared first on nixCraft.
Let us say you want to repeat a character such as '-' OR '=' multiple times while writing bash script. Please note that I needed a quick script to work on Linux, macOS, and FreeBSD server to repeat a string/character n times. One simple way to to repeat a string or character n times is simple use the echo command/printf command as follows:
echo '------------------------------------'
printf '%s\n' '------------------------------------'
Love this? sudo share_on: Twitter - Facebook - LinkedIn - Whatsapp - Reddit
The post How to repeat a character ‘n’ times in Bash appeared first on nixCraft.
How can I check if a file does not exist in a Bash script?
The post How to check if file does not exist in Bash appeared first on nixCraft.
I need to set a bash environment variable called PURGEIMAGE and take specific actions in my script. For example, if PURGEIMAGE set, then purge images from CDN else just URL. I am going to call my script as PURGEIMAGE=yes ./purgecache.sh url1 url2. So how do I know if a bash variable set at the CLI and used in my bash script running on Linux or Unix-like systems?
The post How to check if bash variable defined in script appeared first on nixCraft.
I have two columns (fields) in my text file. I am trying to insert '|
FOO This is a description
BAR Another description for bar
TEXT Another description for TEXT
So I ran awk '{ print $1 "|"; $1=""; print}' input > output but print command is adding a new line:
FOO|
This is a description
BAR|
Another description for bar
TEXT|
Another description for TEXT
Is there any way to tell awk not to print a newline and create the file? Here is what I want so that I can create final HTML table based upon '|'
FOO|This is a description
BAR|Another description for bar
TEXT|Another description for TEXT
The post How to force awk not to print a newline appeared first on nixCraft.
ShellCheck is easy to use, free, and open-source static analysis tool that automatically finds bugs in your shell scripts. If you write shell scripts for automation or containers, you need this tool. Let us see how to install and use ShellCheck on the FreeBSD development Unix server.
The post How to install ShellCheck on FreeBSD – A shell script static analysis tool appeared first on nixCraft.
I need to define a bash variable called failed and set the value to False. When my script is called from a cron job, specific tasks might fail, and then I need to flip failed to True. Based upon $failed, I need to send an email alert that my cron job has failed. So how can I declare and use Boolean variables in a shell script running on a Linux server?
The post How to declare Boolean variables in bash and use them in a shell script appeared first on nixCraft.
Current GNU/Bash maintainer Chet Ramey announced the latest major release, Bash 5.1, on December 7th, 2020. Bash is the default shell on many Linux distros and popular among developers, especially those working with Linux containers. It is the fifth major release of bash. It came with a new enhanced random number generation and added a new variable called SRANDOM. Let us see how to install and use newly released GNU/BASH 5.1 on Linux.
The post GNU/Bash 5.1 released with the random ${SRANDOM} number engine appeared first on nixCraft.
I have many PDF files, and I need to convert them to a png file format, add a border to those images, and convert back all those images to pdf format. How can I convert pdf to image format on Linux and vice versa using the CLI?
The post How to convert pdf to image on Linux command line appeared first on nixCraft.
How do I check the TLS/SSL certificate expiration date from my Linux or Unix shell prompt? How can I find the TLS certificate expiry date from Linux and Unix shell scripts?
The post How to check TLS/SSL certificate expiration date from command-line appeared first on nixCraft.
How do I write the output into the file in Linux bash shell? Can you explain I/O redirection for both bash and POSIX shell to write data to the file under Unix or Linux?
The post How to write the output into the file in Linux appeared first on nixCraft.
I brought eggs from the market, and it says best before 20 days from the date of packing. Say my egg box was packed on 29/July/2020. How do I add days to date using Linux CLI? I want to find out date 29/July/2020 + 20days using the CLI, and how do I do it?
The post How to add days to date and get new date on Linux appeared first on nixCraft.
How can I find the correct ulimit values for a user account or process on Linux systems?
Love this? sudo share_on: Twitter - Facebook - LinkedIn - Whatsapp - Reddit
The post How to find ulimit for user on Linux appeared first on nixCraft.
How do I extract tar.xz files in Linux? Can you tell me command to unzip linux-5.6.13.tar.xz file? How can I uncompressing (or decompressing) all files in the current directory?
The post How to extract tar.xz files in Linux and unzip all files appeared first on nixCraft.
My Nginx web server is behind a reverse proxy server. How do I restore the original and real IP address of my client/visitors when behind a reverse proxy server such as AWS Cloudfront, Fastly, Cloudflare CDN/WAF?
The post Nginx restore real IP address when behind a reverse proxy appeared first on nixCraft.
I need to remove the last character from a line and shell variable such as "${_allowed_ips}". For example, each line is as follows in my file:
foo,
bar,
demo,
I need output as follows:
foo
bar
demo.
How can I use sed to delete the last character?
The post How to sed remove last character from each line appeared first on nixCraft.
The post How to save terminal output to a file under Linux/Unix appeared first on nixCraft.
Let us test your basic knowledge of Linux systems, including commands.
The post Linux Quiz appeared first on nixCraft.
Can you explain DEBIAN_FRONTEND apt or apt-get variable? How and why I need to use this in Docker or shell scripts under a Debian and Ubuntu Linux?
The post Explain DEBIAN_FRONTEND apt-get variable for Ubuntu / Debian appeared first on nixCraft.
How do I see how many processes are running in Linux? How can count all running processes in Linux?
The post How do I count how many processes are running in Linux? appeared first on nixCraft.
I set up a new cloud server, but my auto-completion of bash shell commands are not working. How do I enable autocomplete feature in CentOS 8 (CentOS Enterprise Linux 8)? How do I write simple bash completion code in CentOS 8?
The post How to add bash auto completion in CentOS 8 Linux appeared first on nixCraft.
I need to find and return the PID of the last command executed in Linux. How do I return the PID of command in Linux?
The post How to return pid of a last command in Linux / Unix appeared first on nixCraft.
How do I extract filename and extension in the Bash shell script from the given path? How can I get filename from the path under the bash shell?
The post Bash get filename from given path on Linux or Unix appeared first on nixCraft.
How can I find a website's DNS address? How do I check the current DNS servers of a domain or website using a Linux or macOS command line option? How do I find DNS address of a website using bash?
The post How to find a website’s DNS (name server) address appeared first on nixCraft.
Every Django website needs some automatic background tasks to execute regularly. The outdated sessions need to be cleaned up, search index needs to be updated, some data needs to be imported from RSS feeds or APIs, backups need to be created, you name it. Usually, if not all the time, those regular tasks are being set as cron jobs. However, when some task is run in the background, by default, you don't get any feedback whether it was successfully completed, or whether it crashed on the way. In this post I will show you how I handle the results of cron jobs.
In a Django project, all those tasks are usually implemented as management commands. For each such command I write a short bash script, that will call the management command with specific parameters and will print the verbose output to a log file.
Let's say my project structure is like this on a remote server:
/home/myproject
├── bin
├── include
├── lib
├── public_html
├── backups
├── project
│ └── myproject
├── scripts
└── logs
A virtual environment is created in the home directory of myproject linux user. The Django project itself is kept under project directory. The scripts directory is for my bash scripts. And the logs directory is for the verbose output of the bash scripts.
For example, for the default clearsessions command that removes outdated sessions, I would create scripts/cleanup.sh bash script as follows:
#!/usr/bin/env bash
SECONDS=0
PROJECT_PATH=/home/myproject
CRON_LOG_FILE=${PROJECT_PATH}/logs/cleanup.log
echo "Cleaning up the database" > ${CRON_LOG_FILE}
date >> ${CRON_LOG_FILE}
cd ${PROJECT_PATH}
source bin/activate
cd project/myproject
python manage.py clearsessions --verbosity=2 --traceback >> ${CRON_LOG_FILE} 2>&1
echo "Finished." >> ${CRON_LOG_FILE}
duration=$SECONDS
echo "$(($duration / 60)) minutes and $(($duration % 60)) seconds elapsed." >> ${CRON_LOG_FILE}
To run this command every night at 1 AM, you could create a file myproject_crontab with the following content:
MAILTO=""
00 01 * * * /home/myproject/scripts/cleanup.sh
Then register the cron jobs with:
$ crontab myproject_crontab
By such a bash script, I can track:
In addition, this gives me information whether the crontab was registered and whether the cron service was running at all. As I get the total time of execution in minutes and seconds, I can decide how often I can call the cron job regularly so that it doesn't clash with another cron job.
When you have multiple Django management commands, you can group them thematically into single bash script, or you can wrap them into individual bash scripts. After putting them into the crontab, the only thing left is manually checking the logs from time to time.
If you have any suggestions how I could even improve this setup, I would be glad to hear your opinion in the comments.
Here is the Gist of the scripts in this post. To see some examples of custom Django management commands, you can check Chapter 9, Data Import and Export in my book Web Development with Django Cookbook - Second Edition.
Cover photo by Redd Angelo
I need to get public IPv4 address on current EC2 or Lightsail VM Linux/Unix instance. How do I find Public IP address AWS EC2 or Lightsail VM?
The post How to find Public IP address AWS EC2 or Lightsail VM appeared first on nixCraft.
I noticed that my default shell set to zsh on Macos Catalina. How do I change my default shell to bash on Macos Catalina using the command line or GUI options?
The post Change default shell to bash on Macos Catalina appeared first on nixCraft.
I want to start command called foo, and kill it if still running after given DURATION in seconds. How do I run a command a time Limit on Linux? How do I run a Linux command, and have it timeout (abort) after N seconds?
The post Linux run a command with a time limit (timeout) appeared first on nixCraft.
