Download all images from URL into a single folder

There is plenty options, but easiest one is use command line. The wget is command line utility allows you to download whole web pages, files and images from the specific URL.

Follow command works just fine:

wget -nd -nc -np \
     -e robots=off \
     --recursive -p \
     --level=1 \
     --accept jpg,jpeg,png,gif \
     
     [example.website.com]

What's mean all that?

  • -nd, --no-directories: Do not create a hierarchy of directories when retrieving recursively.
  • -nc, --no-clobber: Do not overwrite existing files.
  • -np, --no-parent: Do not ever ascend to the parent directory when retrieving recursively.
  • -e robots=off: execute command robots=off as if it was part of .wgetrc file. This turns off the robot exclusion which means you ignore robots.txt and the robot meta tags (you should know the implications this comes with, take care).
  • -r, --recursive: Turn on recursive retrieving
  • -p, --page-requisites: Download all the files that are necessary.
  • -l depth, --level=depth: Specify recursion maximum depth level.
  • -A, --accept: Accepted file extensions.

Other useful download options:

  • -H: span hosts (wget doesn't download files from different domains or subdomains by default)
  • --random-wait: This option causes the time between requests to vary between 0.5 and 1.5
  • --wait 1.0: Wait the specified number of seconds between the retrievals.
  • --limit-rate=amount: Limit the download speed to amount bytes per second
  • -U "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/74.0.3729.169 Safari/537.36": Identify as agent-string to the HTTP server as Mozilla Firefox from Windows

Read more on wget manual page.

Real world example

Download all Homophones, Weakly images since 2011

wget -nd -nc -np \
     -e robots=off \
     --recursive -p \
     --level=1 \
     --accept jpg,jpeg \
     -H --random-wait \
     -U "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/74.0.3729.169 Safari/537.36" \
     http://homophonesweakly.blogspot.com/{2011..2019}

#bash#wget#2019

Polly.js - Record, replay, and stub HTTP interactions.

Record, replay, and stub HTTP interactions.
Record, replay, and stub HTTP interactions.

Polly.JS is a standalone, framework-agnostic JavaScript library that enables recording, replaying, and stubbing of HTTP interactions. By tapping into multiple request APIs across both Node & the browser, Polly.JS is able to mock requests and responses with little to no configuration while giving you the ability to take full control of each request with a simple, powerful, and intuitive API.

#javascript#testing#2019

Exiftool tips & tricks

Organize images

Organize images by years/months

exiftool -d "%Y/%m/%Y-%m-%d %H.%M.%S%%-c.%%le" "-filename<CreateDate" -r ./Photos

Download RAW files from SD card to current folder:

exiftool -d "%Y/%m/%Y-%m-%d %H.%M.%S%%-c.%%le" "-filename<CreateDate" --ext raf -r /Volumes/SD

Move all Olympus images to directory Olympus:

exiftool -r '-directory=Olympus' -if '$make eq "OLYMPUS CORPORATION"' .

Rename files to datestamp:

Filename looks like 2014-01-01 12:00:00.jpg and will append -NUM if DateTimeOriginal is the same for multiple files

exiftool '-FileName<DateTimeOriginal' -d "%Y-%m-%d %H.%M.%S%%-c.%%e" .  

Date & time

Find images in a directory that don't have a DateTimeOriginal:

exiftool -filename -filemodifydate -createdate -r -if '(not $datetimeoriginal) and $filetype eq "JPEG"' .

Update any photo that doesn't have DateTimeOriginal to have it based on file modify date:

exiftool '-datetimeoriginal<filemodifydate' -if '(not $datetimeoriginal or ($datetimeoriginal eq "0000:00:00 00:00:00")) and ($filetype eq "JPEG")' .

Set date by filename:

exiftool "-alldates<filename" $@

All metadata

Remove all metadata of a image file:

exiftool -all= -overwrite_original photo.jpg

Remove all metadata of all *.jpg files in current directory:

exiftool -all= -overwrite_original -ext *.jpg

GPS

Strip all metadata except for location (GPS):

exiftool -all= -tagsfromfile @ -gps:all *.jpg

Remove all GPS metadata of *.jpg files in current directory:

exiftool -gps:all= *.jpg

Create KML from geotagged photos:

DESKTOP=$HOME/Desktop
cat $DESKTOP/kml-start.fmt > out.kml
exiftool -n -r -q -p $DESKTOP/kml-placemark.fmt . >> out.kml
cat $DESKTOP/kml-end.fmt >> out.kml

Create CSV of Geo Information:

exiftool -csv -filename -imagesize -gps:GPSLatitude -gps:GPSLongitude ./ > long.csv

Extra

Check Shutter Count:

exiftool -ImageCount [filename]

JSON

Outputs a grouped collection of records as JSON in a directory:

exiftool -json -g /path > collectionprofile.json

#bash#exiftool#2019

Read .env file in bash

Let's have example .env file:

example=value
example2=value

then you can read this file like this:

#!/usr/bin/env bash

set -o allexport
[[ -f .env ]] && source .env
set +o allexport

Warning: All existing variables are overridden by .env content.

#bash#2019

Raspberry PI with Bonjour service and Time Machine

Raspbian Buster Lite (without desktop)

  1. Download Raspbian Buster Lite
  2. Install Raspbian with etcher to SD card
  3. Boot from SD card...

Basic configuration and Wifi connection

sudo raspi-config
  1. Change User Password :-) (default user pi with password raspberry)
  2. Change Localisation Options... select en_US.utf8 and whatever else
  3. Enable SSH in Interfacing Options...
  4. Configure WiFi in Network Options and change Hostname (in my case to pi)

If you have Wifi with hidden SSID, you will need change sudo nano /etc/wpa_supplicant/wpa_supplicant.conf file and add scan_ssid=1:

ctrl_interface=DIR=/var/run/wpa_supplicant GROUP=netdev
update_config=1
country=CZ

network={
    ssid="HiddenWifiSSID"
    scan_ssid=1
    psk="and password :)"
}

If your language characters are still wrong (square), you have to reconfigure locales again:

sudo dpkg-reconfigure locales
sudo dpkg-reconfigure console-setup
sudo dpkg-reconfigure keyboard-configuration

Reboot (sudo reboot) and then check with iwgetid if you are connected to your Wifi!

Install AFP support: netatalk

Netatalk provide AppleTalk Filing Protocol (AFP) interface:

sudo apt install netatalk -y

edit sudo nano /etc/netatalk/afp.conf

[Global]
; mimic model = RackMac

[Homes]
basedir regex = /home

;[My AFP Volume]
;path = /path/to/volume

;[My Time Machine Volume]
;path = /path/to/backup
;time machine = yes

Install Bonjour support: avahi )

sudo apt install avahi-daemon avahi-utils -y
sudo update-rc.d avahi-daemon defaults

Config AFPD with /etc/avahi/services/afpd.service

<?xml version="1.0" standalone='no'?><!--*-nxml-*-->
<!DOCTYPE service-group SYSTEM "avahi-service.dtd">
<service-group>
   <name replace-wildcards="yes">%h</name>
   <service>
        <type>_afpovertcp._tcp</type>
        <port>548</port>
   </service>
   <service>
        <type>_device-info._tcp</type>
        <port>0</port>
        <txt-record>model=RackMac</txt-record>
    </service>
</service-group>

PS: RackMac is define icon that will be shown in Finder (there is plenty other options e.g. Windows, Macintosh, TimeCapsule and so on)

Config SSH with /etc/avahi/services/ssh.service

<?xml version="1.0" standalone='no'?><!--*-nxml-*-->
<!DOCTYPE service-group SYSTEM "avahi-service.dtd">
<service-group>
     <name replace-wildcards="yes">%h SSH</name>
     <service>
          <type>_ssh._tcp</type>
          <port>22</port>
     </service>
</service-group>

For server less ssh you can copy your key to pi:

ssh-copy-id pi@pi.local

Configure avahi daemon

Then in /etc/avahi/avahi-daemon.conf change two parametters host-name and publish-workstation:

[server]
host-name=pi # this row
# ...

[publish]
publish-workstation=yes # this row

# ...

Enable Avahi and Nettalk

Enable afp and avahi

sudo systemctl enable netatalk
sudo systemctl enable avahi-daemon

and start them

sudo service avahi-daemon start
sudo service netatalk start

now check if everything works well with

avahi-browse -a | grep $(hostname)

#Raspberry#iot#2018

Install nodejs on Raspberry PI

First download latest version of nodejs:

wget -qO- http://nodejs.org/dist/latest/node-v12.4.0-linux-armv7l.tar.xz | tar xvz -C ./nodejs
cd nodejs

Compile from sources:

sudo cp -R bin/* /usr/bin/
sudo cp -R lib/* /usr/lib/
sudo apt update && sudo apt upgrade
sudo apt install build-essential

#iot#Raspberry#2018