Skip to main content

destinations.impl.filesystem.factory

filesystem Objects

class filesystem(Destination[FilesystemDestinationClientConfiguration,
"FilesystemClient"])

[view_source]

__init__

def __init__(bucket_url: str = None,
credentials: t.Union[FileSystemCredentials, t.Dict[str, t.Any],
t.Any] = None,
layout: str = DEFAULT_FILE_LAYOUT,
extra_placeholders: t.Optional[TExtraPlaceholders] = None,
current_datetime: t.Optional[TCurrentDateTime] = None,
destination_name: t.Optional[str] = None,
environment: t.Optional[str] = None,
**kwargs: t.Any) -> None

[view_source]

Configure the filesystem destination to use in a pipeline and load data to local or remote filesystem.

All arguments provided here supersede other configuration sources such as environment variables and dlt config files.

The bucket_url determines the protocol to be used:

  • Local folder: file:///path/to/directory
  • AWS S3 (and S3 compatible storages): `s3://bucket-name
  • Azure Blob Storage: `az://container-name
  • Google Cloud Storage: `gs://bucket-name
  • Memory fs: memory://m

Arguments:

  • bucket_url - The fsspec compatible bucket url to use for the destination.
  • credentials - Credentials to connect to the filesystem. The type of credentials should correspond to the bucket protocol. For example, for AWS S3, the credentials should be an instance of AwsCredentials. A dictionary with the credentials parameters can also be provided.
  • layout str - A layout of the files holding table data in the destination bucket/filesystem. Uses a set of pre-defined and user-defined (extra) placeholders. Please refer to https://dlthub.com/docs/dlt-ecosystem/destinations/filesystem#files-layout extra_placeholders (dict(str, str | callable)): A dictionary of extra placeholder names that can be used in the layout parameter. Names are mapped to string values or to callables evaluated at runtime.
  • current_datetime DateTime | callable - current datetime used by date/time related placeholders. If not provided, load package creation timestamp will be used.
  • **kwargs - Additional arguments passed to the destination config

This demo works on codespaces. Codespaces is a development environment available for free to anyone with a Github account. You'll be asked to fork the demo repository and from there the README guides you with further steps.
The demo uses the Continue VSCode extension.

Off to codespaces!

DHelp

Ask a question

Welcome to "Codex Central", your next-gen help center, driven by OpenAI's GPT-4 model. It's more than just a forum or a FAQ hub – it's a dynamic knowledge base where coders can find AI-assisted solutions to their pressing problems. With GPT-4's powerful comprehension and predictive abilities, Codex Central provides instantaneous issue resolution, insightful debugging, and personalized guidance. Get your code running smoothly with the unparalleled support at Codex Central - coding help reimagined with AI prowess.