I will do azure databricks data engineer , pyspark, sql, etl, dbt f

Name: do azure databricks data engineer , pyspark, sql, etl, dbt f
Brand: Fiverr
Availability: InStock

Noor Ahmad

India

I speak English

Azure Databricks, Data Engineer, PySpark, Azure, SQL, ETL, Oracle

Data Engineer with 3+ years of IT experience in enterprise application and database support, specializing in Oracle SQL/PLSQL, data processing, ETL, and database operations. Hands-on experience with A...

I build and maintain ETL/ELT data pipelines using Databricks, PySpark, Python, and SQL.
Work with Azure and AWS cloud platforms, mainly using ADLS Gen2 and Amazon S3 for data storage.
Develop Databricks notebooks to extract, transform, clean, and load data from different sources.
Use PySpark to process large datasets and perform joins, aggregations, filtering, and transformations.
Work with Delta Lake to create reliable and scalable data tables.
Design Bronze, Silver, and Gold layers to organize and process data efficiently.
Connect Databricks with databases such as Oracle, SQL Server, MySQL, and PostgreSQL.
Create SQL queries for data extraction, transformation, validation, and data analysis.
Build both batch and incremental data pipelines depending on the business requirement.
Schedule and automate pipelines using Databricks Workflows and Jobs.
Use Azure Data Factory for pipeline orchestration and integration with Azure services.
Perform data quality checks, including handling null values, duplicates, incorrect data, and schema changes.
Troubleshoot failed pipelines and investigate data and Spark job issues.
Optimize PySpark jobs using techniques such as partitioning, caching.