Essential Technical Skills
- Data Engineering: Strong foundation in data engineering principles, ETL/ELT processes, and data pipeline design patterns
- PySpark: Proven hands-on experience developing data pipelines using PySpark, including DataFrames API, Spark SQL, and performance optimization
- Databricks Platform: Practical experience with Databricks workspace, cluster management, notebooks, and job orchestration
- Workspace AI Agent: Knowledge of Databricks Workspace AI Agent capabilities and integration
- Data Modelling: Experience implementing data models including dimensional modeling, data vault, or lakehouse architectures
- Delta Lake: Understanding of Delta Lake features including ACID transactions, schema evolution, and optimization techniques
- Python: Strong Python programming skills for data processing and automation
Additional Technical Skills
SQL proficiency for data querying and transformation
- Experience with cloud platforms (Azure, AWS, or GCP)
- Understanding of data governance and security best practices
- Knowledge of streaming data processing (Structured Streaming)
- Familiarity with DevOps practices and CI/CD pipelines
- Experience with version control systems (Git)
- Understanding of data quality frameworks and testing methodologies
Professional Experience
- Minimum 8 years in data engineering or related roles
- At least 2-3 years of hands-on experience with Databricks platform
- Proven track record of refactoring legacy code to modern frameworks
- Experience building and maintaining production data pipelines at scale
- Background working across multiple data sources and formats
- Experience in agile development environments
Required Certifications
- Databricks Certified Data Engineer Associate OR Databricks Certified Data Engineer Professional
Additional Certifications (Preferred)
- Databricks Certified Associate Developer for Apache Spark
- Cloud platform certifications (Azure Data Engineer Associate, AWS Certified Data Analytics, or Google Cloud Professional Data Engineer)
- Relevant data engineering or big data certifications
Soft Skills
- Strong problem-solving and analytical thinking abilities
- Excellent communication skills to explain technical concepts clearly
- Ability to work collaboratively in cross-functional teams
- Self-motivated with strong attention to detail
- Adaptable to changing priorities and technologies
- Client-focused mindset with commitment to quality delivery
