mindmap

what's a data lake

• A data lake is a centralized repository that allows you to store all your structured and unstructured data at any scale
• It's a concept, similar to cloud computing, but not a specifc technology
• It's an architectural approach that allows enterprises consolidate large heterogeneous data assets at scale and uncover actionable insights from the consolidated data through various types of analytics

ingest metadata of datasets

• ingest binary files from s3
• ingest tables(parquet, csv) from s3
• ingest custom datasets from s3
  • virtual files
  • files with special metadata contains in its content

query metadata of datasets

handle data between different storage sources by clickhouse

1. uploading local csv data
2. uploading local parquet data
3. s3 parquet data
4. project data from postgresql
5. process from multiple source

demos

• import gaia3 into s3 with parquet format

custom connectors

• jdbc
• s3 with parquet
• multiple s3 source/sink