Hi,
Senior
Data Engineer – Azure (Python / Distributed Data Processing)
Location:
Cincinnati, OH
Duration:
12+ Months
Experience
Required: 10+ Years
Competencies:
Microsoft Azure | Python | Agile Way of Working
Position Overview
We are seeking a highly experienced Senior Data Engineer with strong expertise in Microsoft Azure, Python, and distributed data processing. The ideal candidate will take ownership of stories and drive them to completion across the full SDLC, delivering scalable and high-quality data solutions.
This role involves building and orchestrating enterprise data pipelines, enhancing CI/CD processes, and ensuring data quality and observability in cloud-based environments.
Key Responsibilities
Data Engineering & Pipeline Development
- Develop distributed data processing pipelines (PySpark and/or Snowpark).
- Orchestrate multi-step data transformation workflows.
- Build transformation logic using Object-Oriented Programming (OOP) principles.
- Implement data quality checks for ingested and post-processed data.
- Ensure data observability via monitoring and alerting solutions.
- Maintain and enhance existing applications and data pipelines.
Cloud & DevOps Enablement
- Build and manage cloud resources using Infrastructure as Code (IaC).
- Enhance and maintain CI/CD pipelines for production deployments.
- Manage dependencies (Conda, venv, etc.).
- Debug and performance tune enterprise applications.
Testing & Code Quality
- Perform unit, integration, and regression testing.
- Implement automated testing frameworks (PyTest or similar).
- Follow software engineering best practices and SOLID principles.
- Use Git/GitHub for version control and collaboration.
Agile & Collaboration
- Participate in sprint planning, estimation, and retrospective sessions.
- Mentor junior engineers and contribute to knowledge sharing.
- Collaborate with cross-functional teams on internal and external-facing applications.
- Continuously improve engineering processes and delivery standards.
Required Skills & Experience
- 10+ years of overall IT experience.
- Bachelor’s degree in Computer Science, MIS, Mathematics, Business Analytics, or related technical field.
- 2+ years of professional data development experience.
- 2+ years of SQL development experience.
- Strong proficiency in:
- Python and Python frameworks
- Object-Oriented Programming (OOP)
- Distributed data processing (PySpark / Snowpark)
- CI/CD pipelines
- Git/GitHub
- Automated testing (PyTest)
- Experience with:
- Data Quality processes
- Data Observability tools
- Infrastructure as Code
- Performance tuning
- Experience with cloud technologies (Azure preferred; AWS/GCP acceptable).
- Understanding of Agile/Scrum principles.
Preferred Skills
- Experience with Java and Spring Framework.
- Experience working in large-scale enterprise data environments.
- Strong analytical and problem-solving capabilities.
|
Arjun
BHATT |
E: arjun.b@noviainfotech.com M: (972) 961 2004 Linkedin: linkedin.com/in/arjun-bhatt-3a86001b5 |
You received this message because you are subscribed to the Google Groups "NoviaJobs" group.
To unsubscribe from this group and stop receiving emails from it, send an email to noviajobs+unsubscribe@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/noviajobs/CANVQszEEHCpZPXVtSRZSunWCsW%2BNBrUd6RObrkERuWN0Wzav5w%40mail.gmail.com.
No comments:
Post a Comment