Skip to content

Add s3 protocol to `URIReader`

Eli Mallon requested to merge ak/s3_urireader into catalyst

Created by: AlexKordic

Add S3 resource download to URIReader class.

The S3 URLs look like:

s3+https://s3_key:secret@storage.googleapis.com/alexk-dms-upload-test/testvideo.ts s3+https://s3_key:secret@s3.amazonaws.com/alexk-dms-upload-test/testvideo.ts

Where s3_key and secret fallback to environment variables S3_ACCESS_KEY_ID and S3_SECRET_ACCESS_KEY respectively.

We employ AWS authorization method that should support google, amazon and other S3 compatible cloud providers.

This PR adds logic to calculate AWS signature for S3 GET request. New logic is placed just before HTTP/HTTPS protocols to create proper HTTP request for HTTP/HTTPS logic. This means existing code path is reused and range requests are supported (supportRangeRequest == true).

How to test

  • Compile
  • Run ./build/urireadertest "s3+https://your-key:your-secret@storage.googleapis.com/alexk-dms-upload-est/somedata.txt"
  • Observe file contents is produced on stdout

Merge request reports