Data Engineer

Key Responsibilities

• Design, develop, and optimize ETL/ELT pipelines using AWS services (Glue, Athena, S3, Redshift, Iceberg).
• Implement monitoring and alerting mechanisms for data workflows to ensure reliability and timely issue resolution.
• Perform database and schema optimization for improved performance, scalability, and cost efficiency.
• Manage data validation, taxonomy updates, and metadata consistency across environments.
• Refactor, clean, and maintain data jobs, tables, and legacy pipelines to align with best practices.
• Oversee infrastructure upgrades, including EKS cluster updates and database cleanup activities.
• Automate job scheduling and parameterization for efficient data processing.
• Collaborate with cross-functional teams to integrate external datasets while ensuring compliance with data governance standards.

Required Skills

• Strong proficiency in AWS cloud services: Glue, S3, Athena, Redshift, IAM, and EKS.
• Hands-on experience in ETL/ELT pipeline design, optimization, and troubleshooting.
• Solid knowledge of SQL, Python, and data transformation frameworks.
• Strong understanding of data modeling, schema design, and performance tuning.
• Familiarity with DevOps practices, including automation, CI/CD, and monitoring solutions.
• Proven ability to manage data quality, validation, and governance processes.

Nice to Have

• Experience with data cataloging and metadata management tools (e.g., AWS Glue Data Catalog, DataHub).
• Exposure to real-time or near real-time data processing.
• Knowledge of multi-layered data architectures (bronze/silver/gold or raw/staging/production).