Browse categories
Explore
Fiverr Pro
English
$
USD
I will build robust and scalable data pipelines using pyspark, databricks, and airflow
About this Gig
Need scalable, production-ready data pipelines? I build high-performance data architectures using Databricks, PySpark, SQL, Apache Airflow, and Apache Kafka. From simple file ingestion to complex real-time streaming, I deliver optimized, cost-efficient cloud solutions.
What I Offer:
- End-to-End ETL/ELT: Secure data pipelines using PySpark and SQL.
- Databricks Optimization: Delta Lake, Delta Live Tables (DLT), and Unity Catalog setup.
- Orchestration: Automated Apache Airflow DAGs with error handling.
- Real-Time Streaming: Low-latency event streaming via Apache Kafka.
- Performance Tuning: Slashing your cloud compute costs with optimized queries.
Why Choose Me?
- Production-grade, clean, and reusable code.
- Enterprise cloud experience (AWS / Azure).
- Free architectural documentation included.
Tools & Platforms:
Fivetran
•
Kafka Connect
•
Microsoft SSIS
•
Other
FAQ
1. What information or access do you need to get started?
I need a clear description of your data sources, the expected final output format, and secure, temporary access to your environment (such as Databricks personal access tokens, cloud storage buckets, or Airflow environments)
2. Do you handle both batch processing and real-time streaming?
Yes. I build standard batch ETL pipelines using PySpark and Databricks SQL, as well as real-time, low-latency streaming pipelines using Apache Kafka and Databricks Delta Live Tables (DLT).
3. Will the data pipelines be fully automated?
Yes. I will write custom Apache Airflow DAGs to orchestrate, schedule, monitor, and automate your entire data pipeline, including robust error handling and failure alerts.
4. Will I receive the source code and documentation?
Yes. You will receive 100% ownership of all source code (such as PySpark notebooks, SQL scripts, and Airflow DAG files), along with clean, easy-to-read technical documentation on how to run it.
