Hi,
My name is Rohit Chauhan, and I am a Staffing Specialist at Novia Infotech LLC. I am reaching out to you on an exciting job opportunity with one of our clients.
Job Title: Lead Data Engineer
Location:
Malvern, PA
Experience: 15+ Years
Job Summary
We are seeking an experienced Tech Lead – Data Engineer with deep expertise in Java, AWS, Python, and PySpark, along with strong experience in event-driven architectures. The ideal candidate will lead the design and development of scalable batch and real-time data pipelines, enabling high-quality data delivery for analytics, reporting, and machine learning initiatives.
Key Responsibilities
- Architect, design, and build event-driven data pipelines using AWS services such as Kinesis, MSK (Kafka), Lambda, Step Functions, SQS/SNS, Glue, and EMR
- Develop and maintain ETL/ELT workflows using Python and PySpark ensuring scalability and performance
- Optimize Spark-based transformations, partitioning strategies, and large-scale distributed processing
- Design and manage data lakes and data warehouses using S3, Glue Catalog, Athena, and Redshift
- Build robust real-time streaming solutions with checkpointing, state management, idempotency, and schema evolution
- Implement data quality, monitoring, and observability frameworks using tools like CloudWatch, Datadog
- Ensure data security and governance through IAM, encryption (KMS), and secure networking practices
- Develop reusable frameworks, internal libraries, and CI/CD pipelines for automated deployments
- Collaborate with data scientists, analysts, and business stakeholders to deliver reliable and well-modeled datasets
- Lead architecture discussions, conduct design reviews, and mentor junior engineers
Required Qualifications
- 15+ years of experience in Data Engineering
- Strong programming expertise in Python and PySpark for large-scale data processing
- Hands-on experience with AWS services: S3, Glue, EMR, Lambda, Step Functions, Kinesis/MSK, DynamoDB, Athena, Redshift
- Proven experience in building event-driven and streaming data pipelines
- Strong SQL skills for complex transformations and analytics
- Experience with workflow orchestration tools like Apache Airflow or Step Functions
- Hands-on experience with CI/CD, Git, and Infrastructure as Code (Terraform or CloudFormation)
- Deep understanding of distributed systems, data modeling, and Spark performance tuning
- Experience with cloud cost optimization and data platform scalability
- Strong knowledge of data security, encryption, networking, and compliance best practices
Preferred Skills
- Experience with Java-based backend systems
- Exposure to real-time analytics and machine learning data pipelines
- Familiarity with data governance and cataloging tools
|
Rohit Chauhan IT Recruiter A: 4421 Avenida Ln, McKinney, TX, 75070
|
You received this message because you are subscribed to the Google Groups "NoviaJobs" group.
To unsubscribe from this group and stop receiving emails from it, send an email to noviajobs+unsubscribe@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/noviajobs/CAJ0-OE_PQ3zirG-cPp4%2BzS3RXjezsuqUDcnDKYPkoGBHWf%2BFxg%40mail.gmail.com.
No comments:
Post a Comment