mapillary-takeout-v4: details

Back to the README.

Contents

Size estimate

The estimated download size is calculated from the images you have already downloaded, in this order of preference:

  1. the average size of the images of the same camera model, from the same year if possible (the image size of a camera model changes over the years), or
  2. for a camera model you have not downloaded yet: the average bytes per pixel multiplied by the width and height of the image (e.g. for 360° images), or
  3. the average size of all downloaded images.

The camera model and image size are known from the Mapillary API before the download. Up to 200 downloaded images per camera model and year in the selected date range are used. If there are not enough, up to 100 images per camera model from the rest of the backup folder are read, newest first. For an empty backup folder 2.5 MB per image is used.

The download time assumes 90% of the network speed; the remaining 10% account for protocol overhead, API requests and waiting for the disk.

HTTP requests

The HTTP requests line counts the requests to the Mapillary API and the image downloads. Failures include network errors, HTTP errors (e.g. an expired image URL) and incomplete or broken images. Failed requests are retried; the line says whether all of them were retried successfully, or what was given up after all retries and is missing in this run, e.g. (17 failed, given up: 1 search time window, 12 images). Run the script again later to download the missing images.

Folder layout

Each sequence gets its own folder, below a folder for the day:

<output_folder>/<YYYY-MM-DD>/<captured_at>/<captured_at>_<sequence_id>_NNNN.jpg
<output_folder>/<YYYY-MM-DD>/<captured_at>/<captured_at>_<sequence_id>_NNNN.json   (with --metadata)

NNNN is the position of the image in the sequence, <captured_at> is the start time of the sequence. With --no-subfolder all images of a day are stored in the day folder:

<output_folder>/<YYYY-MM-DD>/<captured_at>_<sequence_id>_NNNN.jpg

Images which were downloaded with the other layout are recognized and not downloaded again, so you can switch between both layouts at any time.

With --fallback-2048, images which Mapillary doesn't have in original size are stored as <captured_at>_<sequence_id>_NNNN_2048.jpg (2048 px). They count as downloaded in the next runs, also without --fallback-2048.

During a download the output folder contains the lock file .mapillary-takeout.lock (with the process id), so a second run for the same folder, e.g. a cron job which starts while the last one is still running, exits with status 75. The lock is released when the script ends, also after a crash or kill -9; the empty file stays. This works on Linux, macOS and FreeBSD. On a drive without file locks (some SMB/NFS network drives, exFAT/FAT USB disks) the script prints a warning and runs without a lock.

Note: the file names differ from version 1.x, a backup made with version 1.x will not be recognized as already downloaded.

Full usage

./mapillary-takeout-v4.py --help
usage: mapillary-takeout-v4.py [-h] [--version] [--token TOKEN] [--start-date YYYY-MM-DD]
                               [--end-date YYYY-MM-DD] [--last-days N] [--bbox W,S,E,N]
                               [--make MAKE] [--model MODEL] [--list-cameras] [--debug 0..4]
                               [--timeout 1..300] [--timeout-meta 1..300] [--threads 1..128]
                               [--meta-threads 1..64] [--retries 0..512] [-D] [--subfolder]
                               [--no-subfolder] [--metadata] [--exif] [--no-exif]
                               [--exif-writer {auto,piexif,exiftool}] [--fallback-2048]
                               [--max-size SIZE] [--max-time TIME] [--disk-reserve SIZE]
                               [--no-disk-check] [--no-version-check] [--no-mtime]
                               username [output_folder]

Download your images from Mapillary (API v4), version: x.yy
Homepage: https://download.bbbike.org/src/mapillary-takeout-v4/

positional arguments:
  username              your Mapillary username
  output_folder         download destination (not needed for --list-cameras)

options:
  -h, --help            show this help message and exit
  --version             show the version and exit; with --debug 1 or higher (default) also whether
                        a newer version is available
  --token TOKEN         Mapillary access token (client token from
                        https://www.mapillary.com/dashboard/developers), default: $MAPILLARY_TOKEN
  --start-date YYYY-MM-DD
                        only images captured on or after this date, default: 2013-01-01
  --end-date YYYY-MM-DD
                        only images captured on or before this date (the end date is included)
  --last-days N         only images of the last N days: today and the N days before, e.g. --last-
                        days 14. Can't be combined with --start-date/--end-date
  --bbox W,S,E,N        limit the search to a bounding box (lon/lat)
  --make, --manufacturer MAKE
                        only images of this camera manufacturer (EXIF "Make"), e.g. Apple, GoPro
                        (case-insensitive substring)
  --model MODEL         only images of this camera model (EXIF "Model"), e.g. iPhone (case-
                        insensitive substring)
  --list-cameras        list the cameras (make/model) with the number of images and exit
  --debug 0..4          set the debug level, default: 1
  --timeout 1..300      connection/read timeout for image downloads in seconds, default: 10
  --timeout-meta 1..300
                        connection/read timeout for metadata requests in seconds, default: 60
  --threads 1..128      number of parallel image downloads, default: 16
  --meta-threads 1..64  number of parallel metadata requests (search, image URLs), default: 32
  --retries 0..512      max. retries of a sequence if no image could be downloaded, with
                        increasing wait time, default: 8
  -D, --dry-run         show what would be downloaded, with size and time estimates, and exit
  --subfolder           store images in a folder per day and a subfolder per sequence, default: on
  --no-subfolder        store images in a folder per day, without subfolders per sequence
  --metadata            store image metadata (GPS, compass, capture time) as .json, default: off
  --exif                write GPS position, capture time, compass angle and camera into the JPEG
                        (needs piexif or exiftool), default: on
  --no-exif             store the images as served by Mapillary, without EXIF
  --exif-writer {auto,piexif,exiftool}
                        tool to write EXIF, auto: piexif if installed, else exiftool, default:
                        auto
  --fallback-2048       images without original at Mapillary: download the 2048 px version as
                        <name>_2048.jpg, default: skip them with a warning
  --max-size SIZE       stop after downloading this much, e.g. 500M, 50G or 2T (default unit G);
                        the next run continues
  --max-time TIME       stop the run after this time, e.g. 90m, 12h or 2d (default unit h), also
                        the search; the next run continues
  --disk-reserve SIZE   free disk space to keep, on top of the estimated download size: a size
                        like 50G, 500M or 2T (default unit G), or a percentage of the download
                        size like 10%, default: 5%, at least 10.0 GB
  --no-disk-check       don't stop if the estimated download size is larger than the free disk
                        space
  --no-version-check    don't check at startup whether a newer version is available, also with
                        $MAPILLARY_TAKEOUT_NO_VERSION_CHECK=1
  --no-mtime            keep the download time as modification time of the images, default: set it
                        to the capture time

How it works

Checking the backup folder

Downloads and retries

There are two levels of retries: single HTTP requests, and complete download rounds of a sequence.

What happens with an image that can't be downloaded:

Situation Behavior
Temporary network or server error Retried in the next round
Expired download URL (HTTP 403) Fresh URL, retried right away
Not found on the server (HTTP 404, also with a fresh URL) Skipped, reported per sequence, exit status 1
No original size at Mapillary Skipped with a warning, exit status not changed: it would fail in every run, and a weekly cron job would always report an error
No original size, but a 2048 px version Downloaded as _2048.jpg with --fallback-2048, otherwise skipped with a warning

The download URLs (thumb_original_url) are signed and expire after some time; expired URLs are refreshed automatically.

At the end of a run a summary lists the images which were not downloaded, e.g.

No new images downloaded.
Not downloaded: 1,581 images not available at Mapillary in original size, 4 images not found on the Mapillary server (HTTP 404)

"You are up to date" is only shown if nothing was skipped.

Disk space

Limits

EXIF and modification time

Performance and memory

Tests

The tests need no network and no access token: a fake Mapillary API and a fake image CDN replace the HTTP requests. Run them from the top folder, one process per test file, in parallel on all CPUs:

python3 tests/run_tests.py

-v shows the name of every test, -j 4 limits the number of processes, and test files can be given as arguments, e.g. python3 tests/run_tests.py test_search.py. Without the runner the tests run one after another:

python3 -m unittest discover -s tests -v

They cover the image search (all images found, also with more than 2,000 images per time window), date ranges, camera filter, both folder layouts, the dry run and size estimate, retries and error handling, exit status, disk space check, lock file, 2048 px fallback, --max-size and --max-time, the version check, EXIF and the modification time. The EXIF tests need piexif or exiftool, otherwise they are skipped. The tests need Python 3.7 or later.

File Tests
takeout_helpers.py fake Mapillary API and image CDN, test images, base test cases
test_search.py image search, date ranges, camera filter, progress and month lines
test_download.py downloads, folder layouts, dry run
test_errors.py errors, retries, exit status
test_cli.py usage errors of the command line (runs the script)
test_cli_output.py output, --help, --version, option aliases (runs the script)
test_sequence_start.py sequences which started before the date range
test_unattended.py disk space check, lock file, 2048 px fallback
test_exif.py EXIF with piexif and exiftool
test_estimate.py size estimate of the dry run
test_storage.py compact storage of the search result
test_format.py number, size and time formatting
test_connections.py HTTP connection reuse with a local HTTP server
test_version_check.py check for a newer version, with a local HTTP server
test_limits.py --max-size and --max-time

History

The file names changed: version 1.x used the capture and upload time, version 2.x uses the capture time and the sequence ID. A backup made with version 1.x is therefore not recognized as already downloaded.