Data Engineer
Federal Reserve Bank
- New York City, NY
- $116,500-200,400 per year
- Permanent
- Full-time
- Design, develop, monitor, and maintain data pipelines in an AWS Gov Cloud ecosystem with AWS, Databricks, Delta Lake and Trino as the underlying platforms.
- Collaborate with cross-functional teams to understand data needs and translate them into effective data pipeline solutions.
- Establish data quality checks and ensure data integrity and accuracy throughout the data lifecycle.
- Automate testing of the data pipelines and configure as part of CICD.
- Optimize data processing and query performance for large-scale datasets within AWS and Databricks environments.
- Document data engineering processes, architecture, and configurations.
- Troubleshooting and debugging data-related issues on the AWS Databricks platform.
- Integrating Databricks with other AWS products such as SNS, SQS, and MSK.
- Bachelor's or master's degree in computer science, Information Technology, or a related field.
- Strong technical proficiency to contribute to data engineering task in an AWS and Databricks ecosystem.
- Strong technical proficiency with Spark, Trino, Python, PySpark and SQL
- Proven ability in Gitlab with CI/CD.
- Proven ability in AWS Services like S3, RDS, Lambda, SQS, SNS, MSK is required.
- Strong SQL skills to perform data analysis and understanding of source data.
- Proven ability with data pipeline orchestration tools
- Proven ability with ETL tools and Relational databases
- Proven ability to troubleshoot complex data issues and implement effective solutions.
- Proven ability in staying updated with industry trends and emerging technologies in data engineering.