Max Rate: $50
Local Candidates Only - F2F
Job Title: Senior
Big Data Developer / Senior Data Engineer (Spark Scala)
Location: O'Fallon, MO
Work Model: 100% Onsite
Duration: 12+ Months
Job Summary
We are seeking a highly skilled Senior Big Data Developer / Senior Data Engineer with deep expertise in Apache Spark, Scala, PySpark, and real-time streaming architectures. The ideal candidate will design, build, and support large-scale batch and streaming data processing systems with a strong emphasis on Kafka-based streaming, Apache NiFi ingestion, and distributed object storage platforms such as Apache Ozone and Ceph.
This role is ideal for engineers who enjoy solving complex challenges related to scalability, low latency, performance optimization, fault tolerance, and production reliability in enterprise data platforms.
Mandatory Skills
Programming Languages
- Scala
- Python (PySpark)
- SQL
- Java (Core)
- Unix Shell Scripting
Big Data Technologies
- Apache Spark Core
- Spark SQL
- Spark Structured Streaming
- Apache Spark Performance Tuning
- Batch Processing
- Real-Time Data Processing
- Distributed Data Processing
Streaming Technologies
- Apache Kafka
- Kafka Streaming Architecture
- Event-Driven Architecture
- Real-Time Streaming Pipelines
- Kafka Offsets and Partitions
- Event Replay & Reprocessing
Data Ingestion & Orchestration
- Apache NiFi
- Apache Airflow
- Data Ingestion Frameworks
- Workflow Orchestration
- Data Routing & Flow Management
Storage Technologies
- Apache Ozone
- Ceph
- Distributed Object Storage
- Object Storage Concepts
- Large-Scale Data Storage Platforms
Databases & Data Processing
- Structured Data
- Semi-Structured Data
- ETL Pipelines
- ELT Pipelines
- Data Validation
- Data Reconciliation
- Metadata Management
DevOps & Platform Engineering
- Linux
- Git
- CI/CD Pipelines
- Monitoring & Logging Tools
- Automated Testing
- Production Support
Key Responsibilities
Big Data Application Development
- Design, develop, and maintain large-scale Apache Spark applications using:
- Scala
- PySpark
- Spark SQL
- Build highly scalable and fault-tolerant data processing frameworks.
- Develop reusable components and data engineering frameworks.
Real-Time Streaming Architecture
- Design and implement streaming data pipelines using:
- Apache Kafka
- Spark Structured Streaming
- Build low-latency streaming applications supporting enterprise-scale data workloads.
- Develop event-driven architectures for real-time analytics and data processing.
Advanced Streaming Patterns
- Implement:
- Stateful Streaming
- Windowing
- Watermarking
- Late Data Handling
- Checkpointing
- Develop event replay and reprocessing mechanisms using Kafka offsets and partitions.
- Ensure reliability and data consistency across streaming platforms.
Data Ingestion & Orchestration
- Build and maintain ingestion and routing workflows using:
- Apache NiFi
- Apache Airflow
- Implement Kafka-based ingestion patterns and orchestration pipelines.
- Support large-scale data movement and processing frameworks.
ETL & ELT Pipeline Development
- Develop end-to-end ETL and ELT pipelines.
- Ensure:
- Low Latency
- Scalability
- Fault Tolerance
- High Availability
- Design reusable and maintainable data processing frameworks.
Spark Performance Optimization
- Optimize Spark applications through:
- Partitioning Strategies
- Memory Management
- Shuffle Optimization
- Efficient File Formats
- CPU Optimization
- I/O Tuning
- Improve overall platform performance and resource utilization.
Storage Platform Integration
- Integrate Spark workloads with:
- Apache Ozone
- Ceph
- Distributed Object Storage Platforms
- Design scalable storage architectures for analytics workloads.
Data Quality & Governance
- Ensure:
- Data Quality
- Data Consistency
- Auditability
- Metadata Capture
- Implement:
- Validation Frameworks
- Reconciliation Processes
- Data Governance Controls
Production Support & Reliability Engineering
- Support mission-critical production systems.
- Perform:
- Monitoring
- Incident Management
- Root Cause Analysis
- Capacity Planning
- Performance Optimization
- Maintain strict SLA compliance.
Engineering Excellence
- Contribute to:
- Coding Standards
- Reusable Frameworks
- Architecture Reviews
- Code Reviews
- Technical Documentation
- Promote engineering best practices and continuous improvement initiatives.
Required Qualifications
- Bachelor's Degree in:
- Computer Science
- Engineering
- Information Technology
- Related Technical Discipline
- 6–8+ years of experience in:
- Big Data Engineering
- Data Engineering
- Distributed Systems
- Real-Time Streaming Platforms
- Enterprise Data Processing Systems
- Strong hands-on experience with:
- Apache Spark
- Scala
- PySpark
- Spark Structured Streaming
- Apache Kafka
- Apache NiFi
- Strong expertise in:
- SQL Development
- ETL/ELT Pipelines
- Distributed Data Processing
- Linux
- Git
- CI/CD
- Experience working with:
- Structured Data
- Semi-Structured Data
- Object Storage Platforms
- Large-Scale Analytics Systems
- Strong analytical, troubleshooting, and problem-solving skills.
Preferred Qualifications
Preferred Technologies
- Apache Ozone
- Ceph
- Apache Airflow
- AWS DevOps & Automation
- Monitoring & Observability Platforms
Preferred Experience
- Exactly-Once Processing Semantics
- At-Least-Once Streaming Semantics
- Streaming System Observability:
- Consumer Lag
- Throughput Monitoring
- Backpressure Management
- Mission-Critical Production Systems
- High Availability Data Platforms
- Automated Testing Frameworks
Key Competencies
- Big Data Engineering
- Real-Time Streaming Architecture
- Distributed Systems Design
- Performance Optimization
- Fault-Tolerant Data Processing
- Production Support
- System Reliability Engineering
- Problem Solving
- Collaboration & Communication
- Technical Leadership
Shrey Sharma
Sr. Technical Recruiter
LinkedIn: https://www.linkedin.com/in/shrey-sharma-322992370/
Novia
Infotech, 4421 Avenida Ln, McKinney, TX, 75070, www.noviainfotech.com
You received this message because you are subscribed to the Google Groups "NoviaJobs" group.
To unsubscribe from this group and stop receiving emails from it, send an email to noviajobs+unsubscribe@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/noviajobs/CA%2BRQAPPEfPX%3DzUa62RiY-FaxUdymo66QgZP16a91M%3DpmtrSs4A%40mail.gmail.com.
No comments:
Post a Comment