Dev Data Pipeline Architecture Guide
====================================

Create a data pipeline architecture guide for [use case].

Pipeline details:
- Use case: [ETL/real-time analytics/ML feature engineering]
- Data sources: [list]
- Data volume: [GB/TB per day]
- Latency requirement: [real-time/near-real-time/batch]
- Destination: [data warehouse/ML platform/dashboard]
- Tech stack: [cloud provider/tools]

DATA PIPELINE ARCHITECTURE

1. ARCHITECTURE PATTERNS

   BATCH PROCESSING
   - When: [daily/hourly jobs acceptable]
   - Tools: [Apache Spark/dbt/AWS Glue]
   - Pattern: Extract → Transform → Load (ETL)
   - Pros: [simple/cheaper/easier to debug]
   - Cons: [data latency]

   STREAM PROCESSING
   - When: [real-time or near-real-time needed]
   - Tools: [Kafka/Flink/Spark Streaming/Kinesis]
   - Pattern: Continuous ingestion and processing
   - Pros: [low latency]
   - Cons: [complex/expensive]

   LAMBDA ARCHITECTURE
   - Combines batch and streaming
   - Batch layer: [historical accuracy]
   - Speed layer: [recent data fast]
   - Serving layer: [merged results]

2. PIPELINE COMPONENTS

   INGESTION
   - Source connectors: [Kafka Connect/Fivetran/custom]
   - Schema registry: [Confluent/Glue]
   - Data validation at ingestion: [critical]

   TRANSFORMATION
   - Raw → Bronze: [as-is from source]
   - Bronze → Silver: [cleaned and validated]
   - Silver → Gold: [business-ready aggregates]
   - Tool: [dbt/Spark/Pandas]

   STORAGE
   - Raw data lake: [S3/GCS/ADLS]
   - Processed: [Snowflake/BigQuery/Redshift]
   - Serving: [Redis/Postgres for low-latency]

3. DATA QUALITY
   - Null checks: [required fields]
   - Type validation: [schema enforcement]
   - Range checks: [business rules]
   - Referential integrity: [key validation]
   - Freshness: [data age alerts]
   - Volume: [row count anomaly detection]

4. ORCHESTRATION
   - Tool: [Airflow/Prefect/Dagster]
   - DAG structure: [dependencies between tasks]
   - Retry logic: [attempts and backoff]
   - Alerts on failure: [email/Slack]
   - SLA monitoring: [expected completion time]

5. MONITORING
   - Pipeline success rate: [%]
   - Processing latency: [time]
   - Data freshness: [lag]
   - Error rates: [by stage]
   - Cost per pipeline run: [$]

6. DISASTER RECOVERY
   - Data retention: [days in raw storage]
   - Backfill capability: [can reprocess history]
   - Idempotency: [safe to rerun]
   - Checkpoint/restart: [from failure point]

Source: https://promptzyo.com/prompt/dev-data-pipeline-architecture-guide