1. ETL Development
- Design, develop, and maintain end-to-end ETL/ELT processes using Azure Data Factory (ADF) and Azure Databricks.
- Write complex, high-performance T-SQL queries, stored procedures, and functions to transform raw data into actionable insights. Optimize data loads for performance and scalability within Azure SQL Database and Synapse Analytics.
2. Quality Assurance & Testing
- Develop and execute comprehensive Data Validation scripts to check for data completeness, transformation logic, and referential integrity. Conduct Source-to-Target mapping validation and performance testing on large datasets. Identify, document, and track data anomalies and bugs, working closely with the engineering team to resolve bottlenecks.
3. Infrastructure & Collaboration
- Work within an Agile environment, collaborating with Data Architects and Business Analysts to understand requirements. Manage version control and CI/CD deployments using Azure DevOps. Monitor pipeline health and set up automated alerts for failure or data drift.
Technical Requirements
SQL Mastery: Deep expertise in SQL (T-SQL/PL-SQL) for data manipulation, optimization, and complex joins.
Azure Data Stack: Hands-on experience with Azure Data Factory, Azure Data Lake Storage (ADLS), and Azure SQL Database. (Knowledge of Databricks/PySpark is a major plus).
Testing Tooling: Experience with data testing tools (e.g., dbt, Great Expectations) or custom Python/SQL testing scripts.
Cloud Infrastructure: Understanding of Azure environment security, Managed Identities, and Key Vaults.
Education: Degree in Computer Science, Information Technology, or a related field, plus 3-5 years of relevant experience.
