
Software Engineer III - AWS & Data Bricks
- Chicago, IL
- $114,000-155,000 per year
- Permanent
- Full-time
- Implements data solutions to make high-quality data available for analytics and reporting,
- Collaborates with data analysts, architects, engineers and business stakeholders to understand data requirements.
- Ensures data quality and consistency by identifying and resolving data issues and creating data reconciliations.
- Optimizes data workflows and processing for performance, scalability, and reliability.
- Monitors data pipelines and proactively addresses issues to minimize downtime and disruptions.
- Documents data engineering processes, data lineage, and data dictionaries
- Stays current with data engineering technologies, best practices, and industry trends.
- Designs and develops complex data pipelines, ETL (Extract, Transform, Load) processes and decision support systems' data procedures.
- Adds to team culture of diversity, equity, inclusion, and respect
- Formal training or certification on Data Engineering concepts and 3+ years applied experience.
- Experience using technologies such as Databricks , Pyspark, and AWS
- Experience in working with AWS (Lambda, Step Function, SQS, SNS, API Gateway, secrets manager and storage services )
- Strong software engineering and object-oriented programming skills with expertise in Python and Terraform
- Familiar with development tools such as Jenkins, Jira, Git/Stash, spinnaker
- Hands on experience with open-source frameworks/libraries, such as Apache NiFi, Apache Airflow and Autosys.
- Strong understanding of REST API development using FASTAPI or equivalent frameworks.
- Familiarity with unit testing frameworks such as pytest or unittest.
- Advanced at SQL (e.g., joins and aggregations)
- Extensive experience in statistical data analysis, with the ability to select appropriate tools and identify data patterns for effective analysis, as well as experience throughout the data lifecycle.
- Data modeling skills.
- Familiarity with Kubernetes, Kafka.
- Experience with containers and container-based deployment environment (Docker, Kubernetes, etc.)
- Exposure to Oracle Database, Pl/SQL programming & Informatica.
Chicago,IL $114,000.00 - $155,000.00 / year
eQuest