Sr Azure Streaming Data Engineer

• Design and implement scalable, low-latency data pipelines to ingest real-time data from various sources (REST APIs, MQTT feeds, IoT sensors, etc.) using Azure streaming services such as Azure Event Hubs, Azure IoT Hub, Azure Stream Analytics, and Azure Functions.
• Build streaming or micro-batch processing jobs (using Stream Analytics or Synapse Spark Structured Streaming) that cleanse, transform, and write event data into Delta Lake tables on ADLS Gen2 in the Raw and Enriched zones.
• Ensure that incoming data is captured in the Raw layer and promptly processed into Enriched for use in analytics.
• Enable real-time and near-real-time analytics by integrating streaming outputs with Power BI (GCC) real-time datasets and dashboards.
• Ensure that critical events (e.g. facility sensor alerts, airline flight alerts, passenger processing kiosk messages) are pushed to Power BI or other subscribers with minimal latency for situational awareness.
• Coordinate with the GIS team to incorporate location data (ArcGIS) into streaming insights.
• Apply Azure best practices in all solutions - including secure networking (VNet integration, private endpoints for Event Hubs/Storage), authentication/authorization (managed identities, RBAC), encryption, and scalable design (throughput units, partitioning strategies, etc.).
• Ensure compliance with any relevant security or regulatory standards in this public-sector environment while optimizing for performance.
• Work closely with the Data Architect to review designs and align with the overall lakehouse framework.
• Provide technical leadership on streaming to more junior engineers if needed, and collaborate with stakeholders in operations and IT to understand real-time data needs.
• Partake in an agile-lite development process (task refinement, sprint planning, demos) and document your solutions for knowledge sharing.

• Self-motivated, quick learner, and adaptable to new technologies and legacy systems.

• 7+ years of data engineering experience with at least 2-3 years focused on real-time/streaming data pipeline development.
• Proven expertise in Azure streaming technologies (Azure Event Hubs, IoT Hub, Azure Stream Analytics) and related patterns (pub/sub, event processing).
• Hands-on experience with Azure Synapse Analytics (or Azure Spark environments) to develop data pipelines. Ability to create and manage Synapse pipelines, triggers, and Synapse Spark notebooks for streaming or batch workflows.
• Familiarity with writing to and reading from Delta Lake storage. (Experience with Synapse native Spark is required.)
• Proficiency in Python (PySpark) and/or Scala for Spark, plus experience authoring Azure Functions (in Python, C# or JavaScript) to handle streaming transformations or invoke APIs.
• Solid understanding of SQL for creating views or analyzing streaming results.
• Experience designing end-to-end streaming data solutions in Azure.
• Knowledge of networking, security, and scalability best practices (e.g., Event Hub partitioning, error handling and retry logic in Functions, scaling Stream Analytics jobs).
• Ability to optimize pipelines for high throughput and low latency.
• Experience writing data to data lakes in a structured format (Delta/Parquet) and integrating with BI tools.
• Understanding of how to output streaming data to Power BI (e.g., through Stream Analytics outputs or Power BI REST endpoints) or similar real-time visualization platforms.
• Excellent communication skills to work with cross-functional teams.
• Ability to translate business requirements for real-time data into technical solutions.
• Experience in a lead or architect capacity on data projects is required, as this is a senior role needing self-direction and mentorship abilities.

• Microsoft Azure certifications such as Azure Data Engineer and/or Azure Solutions Architect.
• Familiarity with other streaming and IoT technologies (e.g., Kafka, MQTT brokers, Azure Event Grid, Azure Data Explorer)
• Experience with big data file formats and tools (Apache Parquet/Delta, Kafka, Spark) and implementing CI/CD for data pipelines (Azure DevOps Pipelines, YAML deployments).
• Knowledge of infrastructure-as-code (ARM/Bicep or Terraform) for deploying data services is a plus.
• Prior experience in secure or regulated environments (government agencies, utilities, healthcare, etc.) dealing with sensitive data. Understanding of compliance requirements and how to design data solutions within those constraintg.

• Applicants should be eligible to work in the US without requiring sponsorship.
• This is a full-time, long-term and hybrid opportunity from the US.
• This role will share time between billable projects and internal solution and innovation work and will heavy communication customers.
• Pay range: $140,000 - $184,000. Individual pay is determined by multiple factors, including job-related knowledge, skills, experience, and relevant education or training.

• Flex PTO, plus 11 paid holidays!
• 12 weeks paid Parental Leave.
• Flexible working hours!
• Fully paid medical/dental/vision benefits for you and your family
• Numerous company and team building events.
• Professional Development Fund for continuing education and training!
• Short & Long-Term Disability fully paid for Life Insurance.
• 401(k)- with employer matching up to 4% of salary contributions.

• Community and Connection: Foster team connection; contribute to a collaborative environment.
• Exceptional Experiences: Strive for outstanding communication experiences; align actions with delivering excellence.
• Embrace New Ideas: Demonstrate openness to innovation; adapt and learn quickly.
• Ownership Mentality: Take responsibility and ownership of tasks and projects; drive issues to resolution.
• Walk the Talk: Consistently align actions, communication, and problem-solving with company values.

Kopius