Lead Data Engineer, Databricks
Delaware North
- Buffalo, NY
- $95,900-129,400 per year
- Permanent
- Full-time
- Medical, dental, and vision insurance
- 401(k) with up to 4% company match
- Annual performance bonus based on level, as well as individual, company, and location performance
- Paid vacation days and holidays
- Paid parental bonding leave
- Tuition and/or professional certification reimbursement
- Generous friends-and-family discounts at many of our hotels and resorts
- Architect and optimize data pipelines for performance and cost-efficiency within the Databricks environment
- Design and implement ETL processes using Databricks jobs to process and transform raw data into a usable format for analysis
- Develop and optimize Spark jobs for data processing, analysis, and machine learning tasks
- Pinpoint and resolve efficiency issues, enhancing the speed of data processing and optimizing the use of resources
- Establish and uphold security protocols to safeguard confidential data, aligning with applicable Legal and CyberSecurity standards
- Bachelor's degree in computer science, engineering, or a similar STEM related field
- Minimum of 5 years of experience in data engineering with a strong focus on Databricks
- Experience designing and implenting ETL processes using Databricks.
- Deep understanding of Apache Spark, Delta Lake, and related big data technologies, and proficiency in modern programming languages such as Python and Scala
- Strong knowledge of SQL and experience with relational, NoSQL, Data Warehouse, and Data Lake methodologies
- Experience with version control systems, SDLC best practices, Agile methodologies, and cloud architecture, AWS preferred
- Experience with Spark jobs for data processing, analysis, and machine learning tasks using Databricks.
- Must be legally authorized to work in the US without sponsorship.