Senior Data Engineer

Role and Responsibilities

Data Architecture & Storage

• Design and architect data storage solutions-including databases, data lakes, and data warehouses-using AWS services such as Amazon S3, Amazon RDS, Amazon Redshift, and Amazon DynamoDB, along with Databricks Delta Lake.
• Integrate Informatica IDMC for metadata management and data cataloging.

Data Pipelines & Processing

• Create, manage, and optimize data pipelines for ingestion, processing, and transformation using AWS services (AWS Glue, AWS Data Pipeline, AWS Lambda), Databricks for advanced data processing, and Informatica IDMC for integration and data quality.
• Develop ETL processes to cleanse, transform, and enrich data for analytical use, leveraging Databricks Spark capabilities and Informatica IDMC.

Data Integration & Governance

• Integrate data from internal and external sources into AWS and Databricks environments, ensuring consistency and quality.
• Utilize Informatica IDMC for data integration, transformation, governance, and compliance with data privacy regulations.

Performance Monitoring & Optimization

• Monitor and optimize performance of data processing and queries across AWS and Databricks environments.
• Utilize Informatica IDMC for optimizing workflows and ensuring scalability and performance efficiency.

Security & Compliance

• Implement security best practices and encryption methods to protect sensitive data across AWS and Databricks.
• Leverage Informatica IDMC for data governance and compliance with privacy standards.

Automation

• Implement automation for data ingestion, transformation, monitoring, and other routine tasks using AWS Step Functions, AWS Lambda, Databricks Jobs, and Informatica IDMC workflow automation.

Documentation

• Maintain comprehensive documentation for data infrastructure, pipelines, and configurations within AWS and Databricks environments.
• Ensure metadata management through Informatica IDMC.

Cross-Functional Collaboration

• Collaborate with data scientists, analysts, and software engineers to understand data requirements and deliver end-to-end solutions across AWS, Databricks, and Informatica IDMC.

Issue Resolution & Support

• Identify and resolve data-related issues to maintain data availability and integrity across AWS, Databricks, and Informatica environments.

Cost Optimization

• Optimize resource usage across AWS, Databricks, and Informatica IDMC to control costs while meeting scalability and performance needs.

Continuous Improvement & Innovation

• Stay up-to-date with AWS, Databricks, and Informatica IDMC features and data engineering best practices.
• Recommend and implement new technologies and techniques as needed.

Requirements / Qualifications

• Bachelor's or Master's degree in Computer Science, Data Engineering, or a related field.
• Minimum 5 years of experience in data engineering with hands-on expertise in AWS services, Databricks, and/or Informatica IDMC.
• Proficiency in programming languages such as Python, Java, or Scala for pipeline development.
• Ability to evaluate technical solutions and make recommendations to resolve data issues, particularly related to performance for complex transformations and long-running data processes.
• Strong knowledge of SQL and NoSQL databases.
• Familiarity with data modeling and schema design.
• Excellent analytical, problem-solving, communication, and collaboration skills.
• Certifications (AWS Data Analytics Specialty, Databricks, Informatica) are a strong plus.

Preferred Skills

• Experience with big data technologies such as Apache Spark and Hadoop on Databricks.
• Knowledge of containerization and orchestration tools like Docker and Kubernetes.
• Familiarity with data visualization tools such as Tableau or Power BI.
• Understanding of DevOps principles for managing and deploying data pipelines.
• Experience with version control systems (Git) and CI/CD pipelines.
• Knowledge of data governance and cataloguing tools, especially Informatica IDMC.