Hi,
This is Diksha Chaudhary working with Novia Infotech. We have the below contract job opportunity with one of our direct clients and would like to check if you have any resources available. Please send across the resume of your consultants along with the contact information at the earliest to diksha.c@noviainfotech.com.
Role: Azure Databricks Data Engineer
Location: Louisville, Kentucky
Work Model: Onsite / Hybrid (as per client requirement)
Duration: 12+ Months
Description:
We are seeking an experienced Azure Databricks Data Engineer with strong expertise in Azure data engineering, Databricks, PySpark, Delta Lake, Azure Data Factory, and enterprise data pipeline development. The ideal candidate will have hands-on experience building scalable cloud-based data ingestion, transformation, and integration solutions in Microsoft Azure environments.
This role requires strong analytical problem-solving skills, hands-on development expertise, and the ability to work closely with architects, business teams, and engineering stakeholders in Agile delivery environments.
Key Responsibilities
Data Pipeline Engineering
- Design, develop, and maintain scalable enterprise data pipelines in Microsoft Azure.
- Build ingestion frameworks to collect data from multiple internal and external source systems.
- Design optimized ETL/ELT workflows for large-scale data processing.
- Ensure high availability, performance, and reliability of enterprise data pipelines.
Azure Data Engineering
Work with Microsoft Azure data ecosystem including:
- Azure Databricks
- Azure Data Factory (ADF / ADB as referenced)
- Delta Lake
- PySpark
- Azure storage/data services
- Azure integration services
Responsibilities include:
- Data ingestion
- Pipeline orchestration
- Transformation workflows
- Data integration architecture
Data Transformation & Processing
- Transform, cleanse, enrich, and validate enterprise data using:
- PySpark
- Databricks notebooks
- Delta Lake architecture
- Perform data quality checks to ensure consistency and accuracy.
- Evolve existing notebooks, scripts, and workflows based on business requirements.
Troubleshooting & Optimization
- Analyze and troubleshoot data-related issues.
- Investigate pipeline failures, transformation issues, and performance bottlenecks.
- Optimize data processing jobs for scalability and efficiency.
- Support operational monitoring and issue resolution.
DevOps & CI/CD
Support Azure DevOps engineering practices including:
- Stories / backlog management
- Bug tracking
- Issue management
- CI/CD deployment pipelines
- Release coordination
Data Architecture Collaboration
- Partner with data architects and cross-functional stakeholders to define data requirements.
- Align data pipelines and transformation logic with enterprise data models.
- Support scalable architecture design decisions.
Security & Compliance
- Ensure data engineering solutions align with:
- Enterprise security standards
- Data governance requirements
- Compliance controls
- Implement secure data handling and access controls.
Agile Delivery
- Participate in Agile/Scrum development processes.
- Support sprint planning, backlog refinement, development execution, and release delivery.
Required Skills
- Azure Databricks
- PySpark
- Delta Lake
- Azure Data Factory
- Azure Data Engineering
- ETL / ELT Development
- Data Pipeline Engineering
- Azure DevOps
- CI/CD
- Data Troubleshooting
- Data Transformation
- Agile Delivery
Preferred Skills
- Azure Integration Services
- Enterprise data architecture
- Data governance
- Performance optimization
- Notebook development
- Cloud security best practices
Experience
5–10+ years of total experience, including:
- Strong Azure data engineering experience
- Hands-on Databricks / PySpark expertise
- Enterprise ETL / cloud data pipeline implementation experience
|
' |
Diksha Chaudhary |
E: diksha.c@noviainfotech.com |
|
You received this message because you are subscribed to the Google Groups "NoviaJobs" group.
To unsubscribe from this group and stop receiving emails from it, send an email to noviajobs+unsubscribe@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/noviajobs/CAEm%3D8YWMKoN%2BFAycu2KqRE7kU%2BBZxV1%2Bw3HGd7vzTiutkXSC5A%40mail.gmail.com.
No comments:
Post a Comment