Senior Data Pipeline Developer
Unreal Gigs
- San Francisco, CA
- Permanent
- Full-time
- Data Ingestion: Develop and maintain data ingestion processes to collect data from diverse sources, including databases, APIs, files, and streaming sources, ensuring seamless data flow and interoperability.
- Data Transformation: Transform and cleanse data as it moves through the pipeline, applying business rules, data enrichment, and validation to meet business requirements and enable downstream analytics and reporting.
- Orchestration and Automation: Implement workflow orchestration and automation solutions to schedule and manage data pipeline workflows, reducing manual intervention and improving operational efficiency.
- Performance Optimization: Optimize data pipeline performance through parallel processing, partitioning, and other techniques, ensuring scalability and responsiveness for large-scale data processing needs.
- Monitoring and Alerting: Implement monitoring and alerting systems to track data pipeline performance and health, proactively identifying and resolving issues to minimize downtime and data loss.
- Error Handling and Retry Mechanisms: Implement error handling and retry mechanisms to handle data processing failures and ensure data reliability and consistency.
- Documentation and Best Practices: Document data pipeline designs, processes, and best practices, providing clear and comprehensive documentation to facilitate understanding and collaboration among team members.
- Collaboration: Collaborate with cross-functional teams, including data engineers, data scientists, and business analysts, to understand requirements and deliver data pipeline solutions that meet business needs.
- Mentorship and Development: Mentor junior developers, sharing expertise and best practices in data pipeline development, and facilitate knowledge sharing sessions within the team.
- Bachelor's degree or higher in Computer Science, Engineering, Mathematics, or related field.
- 5+ years of experience in data engineering, with a focus on designing, building, and optimizing data pipeline solutions.
- Proficiency in data pipeline technologies such as Apache Kafka, Apache NiFi, Apache Airflow, or similar.
- Strong programming skills in languages such as Python, Java, or Scala, with experience in data processing frameworks like Apache Spark or Apache Beam.
- Experience with cloud platforms such as AWS, Azure, or Google Cloud Platform, and services like AWS Glue, Azure Data Factory, or Google Dataflow.
- Strong understanding of data integration concepts and techniques, with experience integrating data from diverse sources and systems.
- Strong problem-solving skills and analytical thinking, with the ability to troubleshoot complex data pipeline issues and optimize system performance.
- Excellent communication and collaboration skills, with the ability to work effectively in cross-functional teams and communicate technical concepts to non-technical stakeholders.
- Competitive salary: The industry standard salary for Senior Data Pipeline Developers typically ranges from $170,000 to $230,000 per year, depending on experience and qualifications.
- Comprehensive health, dental, and vision insurance plans.
- Flexible work hours and remote work options.
- Generous vacation and paid time off.
- Professional development opportunities, including access to training programs, conferences, and workshops.
- State-of-the-art technology environment with access to cutting-edge tools and resources.
- Vibrant and inclusive company culture with opportunities for growth and advancement.
- Exciting projects with real-world impact at the forefront of data-driven innovation.