dmit.dmitio package

Module contents

Prerequisites

To use the S3 utility (part of the dmitio class) one needs to set the following environment variables:

AWS_ACCESS_KEY_ID
AWS_SECRET_ACCESS_KEY

If one is using AWS S3, the following environment variables are also required:

AWS_DEFAULT_REGION

If one is using a local S3 server, the following environment variables are also required:

AWS_S3_HOST

For bash an example could be (values are random and will not work!):

export AWS_ACCESS_KEY_ID=”AJIEPXNW32N3A23APW3C”
export AWS_SECRET_ACCESS_KEY=”XOnfa5HA+gwGEcC3ajklQ3kxlr3qXRR3s3k2xMaA”
export AWS_DEFAULT_REGION=”eu-central-1”
export AWS_S3_BUCKET=”my-bucket-name”

dmit.dmitio.read_json(infile: str)[source]

Reads data from a json file

Parameters: infile (str) – Full path to json input file
Returns: data – dict with data
Return type: dict

class dmit.dmitio.s3(cert: Union[str, bool] = '/etc/ipa/ca.crt')[source]

Bases: object

Example of how to use the s3 class

Examples

>>> #First call the s3 class:
>>> S3 = dmitio.s3()
>>> # (Optional) Get the s3 client (Low Level API)
>>> # Only useful if you want to do something else with the s3 client
>>> s3client = S3.s3_client
>>> # (Optional) Get the s3 resource (High Level API)
>>> # Only useful if you want to do something else with the s3 resource
>>> s3resource = S3.s3_resource
>>> # List the files in the bucket
>>> content = S3.list('my-bucket', 'path/to/files')
>>> # Upload a file to the bucket
>>> S3.upload('my-bucket', 'path/to/bucket-file', 'path/to/local-file')
>>> # Upload a file object to the bucket
>>> with open('/some/file', 'rb') as data:
>>>     S3.upload_object('my-bucket', 'path/to/bucket-file', data)
>>> # Download a file from the bucket
>>> S3.download('my-bucket', 'path/to/bucket-file', 'path/to/local-file')
>>> # Download a file object from the bucket to memory
>>> with open('/some/file', 'wb') as data:
>>>     S3.download_object('my-bucket', 'path/to/bucket-file', data)
>>> # Get bucket resource identifier
>>> bucket = s3resource.Bucket(name="kah")
>>> # Get S3 object identifier
>>> obj = s3resource.Object(bucket_name='my-bucket', key="/path/to/bucket-file")
>>> response = obj.get()
>>> file_content = response['Body'].read()
>>> with open('/some/file', 'wb') as f:
>>>     f.write(file_content)

burrito(description, job: callable) → bool[source]

Utility for s3 transfer status

Parameters

description (str) – job descrition
job (callable) – job to run

Returns

Returns True if the job was successful, False otherwise

Return type

bool

download(bucket: str, key: str, path: str) → bool[source]

Download a file from s3 bucket

Parameters

bucket (str) – Name of bucket
key (str) – Filename in bucket
path (str) – Local filename

Returns

Whether the transfer was successful (True) or failed (False)

Return type

bool

Examples

>>> s3.download('bucketname','key','path')

download_object(bucket: str, key: str, data) → bool[source]

Download a file object from s3 bucket

Parameters

bucket (str) – Name of bucket
key (str) – Filename in bucket
data (file-like object) – A file-like object to upload. At a minimum, it must implement the write method, and must return bytes.

Returns

Whether the transfer was successful (True) or failed (False)

Return type

bool

Examples

>>> with open('testfile', 'wb') as data:
>>>     s3.download_object('bucket', 'key', data)

get_s3_client() → boto3.client[source]

Get s3 client. Useful to use the client directly if you want to do something else with the s3 client

Returns: s3 client
Return type: boto3.client

get_s3_resource() → boto3.resource[source]

Get s3 resource. Useful to use the resource directly if you want to do something else with the s3 resource

Returns: s3 resource
Return type: boto3.resource

list(bucket: str, prefix: str) → list[source]

List files in s3 bucket

Parameters

bucket (str) – Name of bucket
prefix (str) – “path” in bucket

Returns

list of files in bucket

Return type

list

Examples

>>> s3.list('bucketname','prefix')

upload(bucket: str, key: str, path: str) → bool[source]

Upload a file to s3 bucket

Parameters

bucket (str) – Name of bucket
key (str) – Filename in bucket
path (str) – Local filename

Returns

Whether the transfer was successful (True) or failed (False)

Return type

bool

Examples

>>> s3.upload('bucketname','key','path')

upload_object(bucket: str, key: str, data) → bool[source]

Upload a file to s3 bucket

Parameters

bucket (str) – Name of bucket
key (str) – Filename in bucket
data (file-like object) – A file-like object to upload. At a minimum, it must implement the read method, and must return bytes.

Returns

Whether the transfer was successful (True) or failed (False)

Return type

bool

Examples

>>> with open('testfile', 'rb') as data:
>>>     s3.upload_object('bucket', 'key', data)

dmit.dmitio.save_json(datadict: dict, outfile: str) → None[source]

Saves dict to a json file

Parameters

datadict (dict) – json object (dict)
outfile (str) – Full path to json output file