Urgent requirement of Senior Data Engineer – Databricks / PySpark for Atlanta, GA

Hi,

This is Diksha Chaudhary working with Novia Infotech. We have the below contract job opportunity with one of our direct clients and would like to check if you have any resources available. Please send across the resume of your consultants along with the contact information at the earliest to diksha.c@noviainfotech.com.

Role: Senior Data Engineer – Databricks / PySpark

Location: Atlanta, GA

Duration: 12+ Months

 

Description:

We are seeking a Senior Data Engineer with strong hands on expertise in Databricks, PySpark, and large scale data engineering platforms. The ideal candidate will design and develop scalable data pipelines, support big data analytics platforms, and enable advanced analytics and machine learning initiatives. This role requires deep expertise in Spark architecture, data ingestion pipelines, and distributed data processing within cloud or big data environments.

Key Responsibilities

1. Data Engineering and Data Pipeline Development
   Design and develop scalable data pipelines to ingest, clean, transform, and aggregate large volumes of structured and unstructured data.
2. Databricks Platform Development
   Develop and manage data solutions using Databricks including notebooks, Delta Lake architecture, and Delta Live Tables.
3. Data Processing and Transformation
   Implement data processing pipelines using PySpark, Spark SQL, and Python to enable high performance big data processing.
4. Data Integration and Data Ingestion
   Develop mechanisms to ingest data from heterogeneous data sources and normalize datasets for analytics, reporting, and machine learning.
5. Data Quality and Governance
   Implement data validation frameworks and statistical data quality procedures to ensure accuracy and reliability of data sources.
6. Data Modeling and Analytics Support
   Work closely with data scientists and business intelligence teams to develop data models supporting research, reporting, and machine learning applications.
7. Big Data Infrastructure Development
   Design and support scalable data infrastructure capable of processing high volume and high velocity datasets.
8. Streaming Data Processing
   Develop streaming pipelines using technologies such as Apache Kafka and Spark Streaming.
9. Data Engineering Best Practices
   Establish and maintain data engineering best practices for ETL development, pipeline monitoring, and performance optimization.
10. Business Collaboration
    Engage with business stakeholders to gather data requirements and design solutions aligned with business objectives.
11. Team Leadership and Mentorship
    Lead and mentor junior data engineers while contributing as a senior technical resource on data engineering initiatives.
12. Data Insights and Reporting
    Analyze large datasets to derive insights and support leadership with strategic recommendations.

Required Skills

1. 7+ years of overall IT experience.
2. Minimum 5+ years of experience in data engineering or ETL development roles.
3. Strong hands on experience with Databricks platform and Databricks UI.
4. Strong expertise in PySpark and Spark SQL.
5. Experience working with Delta Lake architecture including Delta Lake with Python and Spark SQL.
6. Experience working with Delta Live Tables and Unity Catalog.
7. Strong understanding of Apache Spark architecture and distributed data processing concepts.
8. Experience working with big data technologies such as Hadoop, Hive, Kafka, Sqoop, or HDFS.
9. Experience building scalable data pipelines and ETL frameworks.
10. Strong SQL skills with the ability to write intermediate complexity queries.
11. Experience working with Agile development methodologies including Scrum, Kanban, or SAFe.

Preferred Skills

1. Experience working with cloud platforms such as AWS, Azure, or Google Cloud Platform.
2. Experience working with streaming platforms such as Apache Kafka or Amazon MSK.
3. Experience with BI platforms such as Tableau, Power BI, Spotfire, or Oracle Analytics Cloud.
4. Experience with enterprise data platforms such as Teradata, DB2, BigQuery, or mainframe systems.
5. Experience working with Git for version control.
6. Experience working with ETL tools such as IBM DataStage or similar tools.
7. Experience working with serverless cloud technologies such as AWS Lambda, S3, Kinesis, or AWS Glue.

Experience Required

1. Minimum 6 to 8+ years of experience in data engineering, big data development, or enterprise data platform development.
2. Strong recent hands on experience with Databricks and Apache Spark technologies.

 

 

'

Diksha Chaudhary US IT Recruiter www.noviainfotech.com

E: diksha.c@noviainfotech.com A: McKinney, TX, 75070

 

 

--
You received this message because you are subscribed to the Google Groups "NoviaJobs" group.
To unsubscribe from this group and stop receiving emails from it, send an email to noviajobs+unsubscribe@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/noviajobs/CAEm%3D8YWrPTMWq0JR_nF5UNDuXPHo4oc2den--UBAEwueYMjwpQ%40mail.gmail.com.