Senior Data Engineer
OnMed
- White Plains, NY
- $160,000-180,000 per year
- Permanent
- Full-time
- Own the architecture, development, and maintenance of our data integration solutions using Databricks and Azure cloud solutions, including Azure Data Factory and Azure API Management.
- Own the Databricks environment – across information management and integration capabilities with the goal to support scalability and partner integrations.
- Information Management: Collaborate with CTO and other Engineering Leaders to apply best practices across our cloud solutions, including Databricks environment optimization, governance, ongoing performance monitoring, and guidance to the business intelligence team.
- Integration: This role owns the integration architecture of the organization in tandem with the design, development, optimization, and ongoing performance monitoring of interfaces and integration touchpoints.
- Collaborate with cross-functional teams to understand data requirements, provide best practices and deliver efficient data integration solutions. Reviews and make recommendations to leadership to achieve our engineering and business goals.
- Develop and optimize necessary SQL queries to support data integration and transformation processes.
- Implement and manage data pipelines, ensuring data quality, consistency, and reliability.
- Collaborate with leaders to understand our business and proactively generating data analytics dashboards, reports, data pipeline to provide value-add insights. Oversee the work of other peers to ensure data quality, consistency and reliability to support short and long-term needs.
- Leverage AI tools to enhance and optimize monitoring and troubleshooting of data integration workflows to ensure seamless operation and ongoing maintenance.
- Document data integration processes and provide technical support to stakeholders.
- Stay updated with the latest trends and best practices in data integration, Databricks, and Azure technologies.
- Perform other related role's responsibilities as assigned.
- Strong experience with and can apply Agile engineering practices.
- Advanced Python, including experience building APIs, scripting ETL processes, and automating workflows.
- Ability to see the big picture and service as the integration architect in tandem with getting granular about design and development – macro- and micro-level execution.
- Deep knowledge of SQL and experience with complex query optimization.
- Strong expertise in Databricks and Azure API Management.
- Experience with data modeling, ETL processes, and data warehousing.
- Familiarity with data governance and security best practices.
- Proficiency in creating technical documentation, SOPs, and maintaining clear communication with cross-functional teams.
- A problem-solving mindset, with the ability to take ownership and drive solutions from design to deployment.
- Technical experience leveraging AI tools for data analytics and database maintenance will be a plus.
- Bachelor's degree in Computer Science, Information Technology, or a related field.
- At least 7+ years of relevant professional experience as a Data Engineer with a strong analytics background, and 4+ years of experience with Agile engineering practices.
- Proven track record with Software/Data Engineering, building robust data systems (prior leadership or mentorship experience is a plus)
- Technical Skills Required: Databricks, Python and SQL.
- Technical Skills Nice-to-Have: Azure or AWS (experience solving big data problems), C# and PySpark, and ML/Machine Learning background.
- Experience working with AI tools highly desirable.
- Experience working with healthcare data or in the healthcare industry is a plus.