R6 Class Wrapper for Metaflow S3 Client

R6 Class Wrapper for Metaflow S3 Client

R6 Class Wrapper for Metaflow S3 Client

Details

This class provides an R interface to the Metaflow S3 client functionality.

Public fields

py_s3

Python S3 object

Methods

Public methods

• S3$new()

• S3$get()

• S3$get_many()

• S3$get_recursive()

• S3$get_all()

• S3$list_paths()

• S3$list_recursive()

• S3$put()

• S3$put_many()

• S3$put_files()

• S3$info()

• S3$info_many()

• S3$close()

---

Method new()

Create a new S3Client object

Usage

S3$new(...)

Arguments

...

Additional arguments passed to the Python S3 constructor

---

Method get()

Get a single object from S3

Usage

S3$get(key, return_missing = FALSE, return_info = TRUE, ...)

Arguments

key

Object key

return_missing

If TRUE, return a missing object instead of raising an exception

return_info

If TRUE, fetch content-type and user metadata

...

Additional arguments passed to the Python get method

Returns

An S3Object

---

Method get_many()

Get many objects from S3 in parallel

Usage

S3$get_many(keys, return_missing = FALSE, return_info = TRUE)

Arguments

keys

List of object keys

return_missing

If TRUE, return missing objects instead of raising exceptions

return_info

If TRUE, fetch content-type and user metadata

Returns

List of S3Object instances

---

Method get_recursive()

Get many objects from S3 recursively in parallel

Usage

S3$get_recursive(keys, return_info = FALSE)

Arguments

keys

List of prefixes to download recursively

return_info

If TRUE, fetch content-type and user metadata

Returns

List of S3Object instances

---

Method get_all()

Get all objects under the prefix set in the S3 constructor

Usage

S3$get_all(return_info = FALSE)

Arguments

return_info

If TRUE, fetch content-type and user metadata

Returns

List of S3Object instances

---

Method list_paths()

List the next level of paths in S3

Usage

S3$list_paths(keys = NULL)

Arguments

keys

List of paths to list

Returns

List of S3Object instances

---

Method list_recursive()

List all objects recursively under the given prefixes

Usage

S3$list_recursive(keys = NULL)

Arguments

keys

List of prefixes to list

Returns

List of S3Object instances

---

Method put()

Put a single object to S3

Usage

S3$put(key, obj, overwrite = TRUE, content_type = NULL, metadata = NULL, ...)

Arguments

key

Object key

obj

Object to store

overwrite

If TRUE, overwrite existing object

content_type

MIME type for the object

metadata

A list of additional headers to be stored as metadata

...

Additional arguments passed to the Python put method

Returns

URL of the stored object

---

Method put_many()

Put many objects to S3

Usage

S3$put_many(key_objs, overwrite = TRUE)

Arguments

key_objs

List of key-object pairs or S3PutObjects

overwrite

If TRUE, overwrite existing objects

Returns

List of (key, url) pairs for uploaded objects

---

Method put_files()

Upload many local files to S3

Usage

S3$put_files(key_paths, overwrite = TRUE)

Arguments

key_paths

List of key-path pairs or S3PutObjects

overwrite

If TRUE, overwrite existing objects

Returns

List of (key, url) pairs for uploaded files

---

Method info()

Get metadata about a single object in S3

Usage

S3$info(key = NULL, return_missing = FALSE)

Arguments

key

Object key

return_missing

If TRUE, return a missing object instead of raising an exception

Returns

An S3Object

---

Method info_many()

Get metadata about many objects in S3 in parallel

Usage

S3$info_many(keys, return_missing = FALSE)

Arguments

keys

List of object keys

return_missing

If TRUE, return missing objects instead of raising exceptions

Returns

List of S3Object instances

---

Method close()

Close the S3 client and delete temporary files

Usage

S3$close()