Remote Objects (S3)
Before viewing remote objects, you must install the
boto3 package in the same environment as Harlequin. You can do this by installing Harlequin with the
$ pip install harlequin[s3]
Harlequin works with any object storage system that provides an S3-compatible API, including Amazon S3, Google Cloud Storage, and Minio; basically, if you can query the storage with
boto3 and manage credentials with the AWS CLI, Harlequin can display your files.
Harlequin relies on the AWS CLI credential toolchain for authentication. Specifically, it will attempt to connect to your storage using whatever credentials are currently active (or default) in your AWS CLI config, including the region. You can use the credentials from another profile by setting the
AWS_PROFILE environment variable.
For Google Cloud Storage please see the docs on the XML API and HMAC authentication. After generating HMAC Keys, you can use the AWS CLI (
aws configure) to store these keys in a profile accessible to Harlequin.
Your user must have
ListObjects or the equivalent permission to view a bucket’s objects in Harlequin.
Currently this is not configurable; if that doesn’t work for you, please open an issue.
Harlequin will display the remote tree if it is initialized with the
--show-s3 option (alias
--s3). This option takes a reference to remote object storage.
Displaying All Buckets
Use Harlequin with
--show-s3 all to display all Amazon S3 buckets that the authenticated user has access to. (This is not advised if you have access to millions of objects in S3):
$ harlequin --show-s3 all
For GCS or another endpoint that supports
ListBuckets, provide the endpoint url without a path:
$ harlequin --show-s3 "https://storage.googleapis.com"
Displaying a Single Bucket
--show-s3 with an argument that represents a bucket and (optionally) an endpoint url and key prefix:
$ harlequin --show-s3 my-bucket
$ harlequin --show-s3 my-bucket/my-prefix
A one-liner to set the AWS Profile and connect to a GCS bucket, filtering for a prefix:
$ AWS_PROFILE=gcs harlequin --s3 "https://storage.googleapis.com/my-gcs-bucket/my-prefix"
Harlequin takes any of the following formats:
# Amazon S3 Formats
# Google Cloud Storage Formats
# Minio, AWS PrivateLink, etc.