Senior Solution Architect – Databricks

Responsibilities

End-to-End Data Architecture

• Collaborate with Databricks PS and customer

• on comprehensive end-to-end data architecture on Databricks platform

• Design and implement data ingestion strategies from various source systems

• Architect scalable storage solutions leveraging Delta Lake and lakehouse architecture

• Develop efficient data processing frameworks using Apache Spark and Databricks workflows

• Create data consumption layers for analytics, reporting, and machine learning use cases

• Ensure seamless data flow across the entire data lifecycle

Governance, Security & Compliance

• Collaborate with Databricks PS on implementing robust governance frameworks

• Design and enforce security policies and access controls within Databricks environments

• Ensure compliance with regulatory requirements and industry standards

• Implement data lineage, audit trails, and monitoring capabilities

• Optimize performance through best practices in cluster configuration and query optimization

• Establish data quality standards and validation processes

Data Modelling & Design

• Define business-aligned data models that support organizational objectives

• Design Delta table structures optimized for performance and scalability

• Create dimensional and normalized data models based on business requirements

• Implement medallion architecture (Bronze, Silver, Gold layers) for data refinement

• Develop data schemas that support both analytical and operational workloads

• Document data models and maintain data dictionaries

Technical Leadership & Collaboration

• Lead technical discussions with clients and stakeholders to understand business requirements

• Provide technical guidance to development teams on Databricks best practices

• Work closely with Infrastructure (Infra), Applications (Apps), and Cyber teams for integrated delivery

• Mentor junior architects and engineers on Databricks technologies

• Present solution architectures and technical recommendations to client leadership

Solution Implementation

• Oversee implementation of Databricks solutions from design through deployment

• Ensure solutions meet performance, scalability, and reliability requirements

• Support proof-of-concepts and pilot implementations

• Conduct architecture reviews and ensure alignment with enterprise standards

Skillset(Must have)

Essential Technical Skills

• Databricks Platform: Proven hands-on experience with Databricks platform including workspace administration, cluster management, and workflow orchestration

• Unity Catalog: Demonstrated experience implementing Unity Catalog for unified governance, including metastore configuration, catalog/schema design, and access control policies

• Data Modelling: Strong expertise in designing and implementing data models, including dimensional modeling, data vault, and lakehouse architectures

• Delta Lake: Deep understanding of Delta Lake features including ACID transactions, time travel, and optimization techniques

• Apache Spark: Proficiency in Spark SQL, DataFrames, and performance tuning

• Programming: Strong skills in Python, SQL, and/or Scala for data processing

• Cloud Platforms: Experience with Azure, AWS, or GCP cloud services

Additional Technical Skills

• Data pipeline development and ETL/ELT processes

• Data governance frameworks and metadata management

• Performance optimization and troubleshooting

• CI/CD practices for data platforms

• Data quality and validation frameworks

• Understanding of data security and compliance requirements

Professional Experience

• Minimum 8-10 years in data architecture, engineering, or analytics roles

• At least 3-5 years of hands-on experience with Databricks platform

• Databricks Certified Associate Developer for Apache Spark (mandatory)

• Proven track record implementing Unity Catalog in enterprise environments

• Experience designing and implementing complex data models for large-scale systems

• Background working with Databricks Professional Services or similar partnerships preferred

• Experience across multiple industry sectors (Public Service, Financial Services, Healthcare, etc.) is advantageous

Certifications (Preferred) - mandatory to have at least one certification

• Data Engineer Professional

• Cloud platform certifications (Azure Data Engineer, AWS Data Analytics, or GCP Data Engineer)

• Relevant data management or analytics certifications

Soft Skills

• Excellent communication and presentation abilities

• Strong analytical and problem-solving capabilities

• Ability to work collaboratively with Partners and cross-functional teams

• Client-focused mindset with relationship-building skills

• Self-motivated with ability to manage multiple priorities

• Experience working in agile environments