Required Qualifications
Essential Technical Skills
- Databricks Platform: Proven hands-on experience with Databricks platform including workspace administration, cluster management, and workflow orchestration
- Unity Catalog: Demonstrated experience implementing Unity Catalog for unified governance, including metastore configuration, catalog/schema design, and access control policies
- Data Modelling: Strong expertise in designing and implementing data models, including dimensional modeling, data vault, and lakehouse architectures
- Delta Lake: Deep understanding of Delta Lake features including ACID transactions, time travel, and optimization techniques
- Apache Spark: Proficiency in Spark SQL, DataFrames, and performance tuning
- Programming: Strong skills in Python, SQL, and/or Scala for data processing
- Cloud Platforms: Experience with Azure, AWS, or GCP cloud services
Additional Technical Skills
- Data pipeline development and ETL/ELT processes
- Data governance frameworks and metadata management
- Performance optimization and troubleshooting
- CI/CD practices for data platforms
- Data quality and validation frameworks
- Understanding of data security and compliance requirements
Professional Experience
- Minimum 8-10 years in data architecture, engineering, or analytics roles
- At least 3-5 years of hands-on experience with Databricks platform
- Proven track record implementing Unity Catalog in enterprise environments
- Experience designing and implementing complex data models for large-scale systems
- Background working with Databricks Professional Services or similar partnerships preferred
- Experience across multiple industry sectors (Public Service, Financial Services, Healthcare, etc.) is advantageous
Certifications (Preferred)
- Data Engineer Professional
- Databricks Certified Associate Developer for Apache Spark
- Cloud platform certifications (Azure Data Engineer, AWS Data Analytics, or GCP Data Engineer)
- Relevant data management or analytics certifications
Soft Skills
- Excellent communication and presentation abilities
- Strong analytical and problem-solving capabilities
- Ability to work collaboratively with Partners and cross-functional teams
- Client-focused mindset with relationship-building skills
- Self-motivated with ability to manage multiple priorities
- Experience working in agile environments
Key Responsibilities
End-to-End Data Architecture
- Collaborate with Databricks PS and customer
- on comprehensive end-to-end data architecture on Databricks platform
- Design and implement data ingestion strategies from various source systems
- Architect scalable storage solutions leveraging Delta Lake and lakehouse architecture
- Develop efficient data processing frameworks using Apache Spark and Databricks workflows
- Create data consumption layers for analytics, reporting, and machine learning use cases
- Ensure seamless data flow across the entire data lifecycle
Governance, Security & Compliance
- Collaborate with Databricks PS on implementing robust governance frameworks
- Design and enforce security policies and access controls within Databricks environments
- Ensure compliance with regulatory requirements and industry standards
- Implement data lineage, audit trails, and monitoring capabilities
- Optimize performance through best practices in cluster configuration and query optimization
- Establish data quality standards and validation processes
Data Modelling & Design
- Define business-aligned data models that support organizational objectives
- Design Delta table structures optimized for performance and scalability
- Create dimensional and normalized data models based on business requirements
- Implement medallion architecture (Bronze, Silver, Gold layers) for data refinement
- Develop data schemas that support both analytical and operational workloads
- Document data models and maintain data dictionaries
Technical Leadership & Collaboration
- Lead technical discussions with clients and stakeholders to understand business requirements
- Provide technical guidance to development teams on Databricks best practices
- Work closely with Infrastructure (Infra), Applications (Apps), and Cyber teams for integrated delivery
- Mentor junior architects and engineers on Databricks technologies
- Present solution architectures and technical recommendations to client leadership
Solution Implementation
- Oversee implementation of Databricks solutions from design through deployment
- Ensure solutions meet performance, scalability, and reliability requirements
- Support proof-of-concepts and pilot implementations
- Conduct architecture reviews and ensure alignment with enterprise standards
