What does this skill do?

The Senior Data Engineer skill helps an agent build scalable, dependable data systems — data pipelines, ETL/ELT, and data infrastructure — with expertise across the modern data stack.

It guides architecture decisions like batch vs streaming and Lambda vs Kappa, data modeling (dimensional, slowly changing dimensions, data vault), and data quality and observability.

It works across Python, SQL, and Spark with orchestration in Airflow, Prefect, or Dagster, transformation in dbt, and warehouses like Snowflake, BigQuery, and Databricks.

  • Designs batch and streaming data pipelines

  • Builds dimensional and data-vault models

  • Implements data quality checks and monitoring

  • Applies DataOps practices — testing, contracts, CI/CD

How it works

1

Choose the architecture

The agent evaluates batch vs streaming, Lambda vs Kappa, and warehouse vs lakehouse based on latency, volume, and complexity.

2

Design the pipeline

It designs the ETL/ELT flow and data model — dimensional, slowly changing dimensions, or data vault — for the use case.

3

Add quality and orchestration

It orchestrates with Airflow, Prefect, or Dagster and adds data-quality checks with tools like Great Expectations and dbt tests.

4

Operate and optimize

It applies DataOps practices — testing, contracts, CI/CD, observability — and tunes pipeline and Spark performance.

Best use cases for this skill

  • Pipeline design: Build a batch or streaming ETL/ELT pipeline.

  • Data modeling: Create dimensional models and slowly changing dimensions.

  • Data quality: Add validation and monitoring between pipeline stages.

  • Performance tuning: Optimize Spark jobs and reduce pipeline runtime.

Frequently asked questions

What does the Senior Data Engineer skill do?

It helps an agent build scalable data pipelines, ETL/ELT systems, and data infrastructure, including data modeling, orchestration, data quality, and DataOps.

When should an agent use it?

Use it when designing data architectures, building data pipelines, optimizing data workflows, implementing data governance, or troubleshooting data issues.

Which tools does it cover?

Python, SQL, and Spark; orchestration with Airflow, Prefect, or Dagster; dbt for transformation; and warehouses like Snowflake, BigQuery, Redshift, and Databricks.

How does it decide batch vs streaming?

It compares latency, volume, and complexity, and evaluates Lambda versus Kappa architectures to fit the requirements.

Use from source

Ask your Buda agent to install this community skill.

Copy this prompt into your own Buda agent. It includes the source repository and install command so the agent can complete the setup in natural language.

Prompt to send to your Buda agent
Please install the "Senior Data Engineer Skill" skill into this Buda agent.

Source repository: https://github.com/alirezarezvani/claude-skills
Install command: npx skills add alirezarezvani/claude-skills --skill senior-data-engineer -y -a universal

Please run the install command, verify the skill is available, and tell me any setup notes or follow-up steps from the source repository.
Explore marketplace

Install workflow

  1. 1Copy the prompt
  2. 2Send it to your Buda agent
  3. 3Done