Thomas Weise (汤卫思)

A professor at Hefei University.

1 minute read

Sometimes, we want to download a file via the terminal . Downloading files may always fail due to connection issues. So we want a command that tries again and again until success. Here I post a small script does exactly that, which you may store as download.sh. The script takes one or multiple URLs as parameter and downloads the corresponding files. You could, for example, type download.sh "https://thomasweise.github.io/programmingWithPython/programmingWithPython.pdf" and it will download our book on the programming language. The script uses wget internally, but adds an infinite loop around it. It also first tries 10 times to download the file without rate limitation and afterwards limits the connect speed in follow-up attempts. If the URL is simply invalid or the file does not exist, the script loops forever. Here you can download this script.

#!/bin/bash -

# strict error handling
set -o pipefail  # trace ERR through pipes
set -o errtrace  # trace ERR through 'time command' and other functions
set -o nounset   # set -u : exit the script if you try to use an uninitialized variable
set -o errexit   # set -e : exit the script if any statement returns a non-true return value

echo "$(date +'%0Y-%0m-%0d %0R:%0S'): Welcome to using the download script for downloading '$@'."
for i in {1..10}; do
    set +o errexit
    wget --user-agent="Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:83.0) Gecko/20100101 Firefox/83.0" --retry-connrefused --waitretry=1 --read-timeout=20 --timeout=15 -t 0 --continue --random-wait --no-check-certificate "$@"
    retcode="$?"
    set -o errexit
    if [ "$retcode" -eq 0 ]; then exit; fi; # check return value, break if successful (0)
    echo "$(date +'%0Y-%0m-%0d %0R:%0S'): Error when trying to download '$@', got exit code $retcode, trying again after 1 second."
    sleep 1s;
done;

echo "$(date +'%0Y-%0m-%0d %0R:%0S'): Now trying with limited rate."
while [ 1 ]; do
    set +o errexit
    wget --user-agent="Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:83.0) Gecko/20100101 Firefox/83.0" --retry-connrefused --waitretry=1 --read-timeout=20 --timeout=15 -t 0 --continue --limit-rate=200k --random-wait --no-check-certificate "$@"
    retcode="$?"
    set -o errexit
    if [ "$retcode" -eq 0 ]; then exit; fi; # check return value, break if successful (0)
    echo "$(date +'%0Y-%0m-%0d %0R:%0S'): Error when trying to download '$@', got exit code $retcode, trying again after 1 second."
    sleep 1s;
done;

echo "$(date +'%0Y-%0m-%0d %0R:%0S'): Finished downloading '$@'."

You May Also Enjoy

Updating all Packages and Snaps

2 minute read

I use for my work and try to keep my system up-to-date. This means that, whenever I shut down my computer, I run a little Here I post a small script update.sh that updates all installed packages, both deb and snap packages. Sometimes, for whatever reason, there may be a problem with some inconsistent package state. My script basically applies all methods I know to fix such state. The script is very heavy-handed and loops over the update process four times. If for whatever reason one update would require another first and that could only be done by multiple updates, then that’s no problem. I usually run something like sudo update.sh && shutdown now, which then shuts down my computer. So I do not really need to care how long the script needs anyway. Here you can download this script.

Repeating a Command until it Succeeds under Linux

2 minute read

Assume that we have a command that we want to execute in a terminal. We know that the command may sometimes fail, but it should actually succeed. So we would try the command again and again until this happens. An example is, for instance, a download process. If we know that the URL is reachable, then the command should succeed. However, maybe there is a lost connection or other temporary disturbance. Another such command could be a build process which downloads and installs dependencies.

Recursively Deleting Empty Files and Directories under Linux

1 minute read

Sometimes, we have the need to delete empty files and empty directories under . This happens, for example, when we want to restart an experiment with . Here I post a small script that you may store as file deleteZeroSizeFiles.sh. It will do exactly that: It will search the current directory and all subdirectories for empty files, i.e., files of size zero. It will delete all of them. Then, it will recursively look for empty directories, i.e., directories that do not contain files or other directories. It will delete them as well. All files and directories that get deleted are also printed, so you can see what happened. Here you can download this script.