Lead Databricks Data Architect
- Atlanta, GA
- Permanent
- Full-time
Experience: 8 – 10 YearsJob Type: Full TimeJob OverviewWe are seeking an experienced Lead Databricks Data Architect to design and implement robust, scalable, and secure cloud-based data solutions. This role involves working with Azure cloud services, leading development of data ingestion pipelines, and optimizing data architecture to meet evolving business needs.Key Responsibilities
- Architect and build end-to-end data solutions using Azure Databricks, Azure Data Factory, and Azure SQL Database.
- Lead the development of real-time and batch data pipelines using Delta Live Tables (DLT) and Structured Streaming.
- Collaborate with stakeholders to translate business requirements into scalable data warehouse solutions on Azure.
- Provide strategic leadership for large-scale data programs and ensure alignment with organizational goals.
- Drive performance optimization of Spark jobs and Databricks workflows through advanced tuning techniques.
- Build and maintain metadata-driven ingestion frameworks to support automation and scalability.
- Work cross-functionally with data scientists, analysts, and other stakeholders to deliver business-aligned data solutions.
- Implement data governance, quality checks, and security measures to maintain data integrity and compliance.
- Monitor and enhance cloud-based data infrastructure for performance and cost-efficiency.
- Lead or support migration and upgrade efforts of legacy applications (e.g., .NET Framework) to cloud platforms.
- Utilize Azure DevOps tools including Git, pipelines, and work item management.
- Troubleshoot issues on Windows Server environments (e.g., IIS, Event Viewer, Performance Monitor).
- Participate in Agile/Scrum ceremonies including sprint planning, retrospectives, and daily standups.
- Strong proficiency in:
- Azure Databricks, ADF, Azure SQL
- Apache Spark (job tuning and internals)
- Structured Streaming, Delta Live Tables
- Experience in:
- .NET Frameworks 4.8.2 and Core 6/8
- ASP.NET MVC, Blazor, WebForms
- Web services & API integrations
- Microsoft SQL Server (DDL, DML, stored procedures)
- Nota Fiscal integration (Preferred in finance/logistics/related domains)
- Familiarity with:
- Azure authentication and key vault
- Agile/Lean methodologies
- Proven experience designing and delivering enterprise-level cloud data platforms.
- Ability to lead cross-functional teams and mentor junior developers.
- Excellent communication, documentation, and stakeholder management skills.