Roles & Responsibilities
We are looking for a Lead Big Data Engineer to spearhead the development of scalable and secure big data solutions across our organization. This is a strategic leadership role that combines deep technical expertise with a passion for mentoring, architecture, and data governance. As a key member of our data team, you will drive innovation, ensure best practices, and help shape the future of our data infrastructure.
Key Responsibilities:
- Design, build, and optimize data pipelines and ETL/ELT workflows using SQL and Python.
- Lead architecture and design reviews, including the development of Entity Relationship Diagrams (ERDs) and system blueprints.
- Work closely with analysts, data scientists, and software engineers to translate business needs into scalable data solutions.
- Provision and manage cloud infrastructure using Terraform and other Infrastructure-as-Code (IaC) tools.
- Design and maintain CI/CD pipelines, particularly leveraging GitHub Actions, for seamless deployment of data applications.
- Utilize AWS services such as S3, Glue, Lambda, RDS, and Lake Formation to build robust cloud-native data platforms.
- Handle both batch and real-time data ingestion and transformation with efficiency and scalability.
- Support strategic initiatives around data privacy, quality, and security; implement data encryption, masking, and hashing techniques.
- Enforce modern software development practices including version control, testing, and deployment standards.
- Optimize system performance and troubleshoot production data issues proactively.
- Guide and mentor junior data engineers and support team knowledge-sharing efforts.
Required Skills & Experience:
- Strong command of SQL for complex querying and data modeling.
- Advanced Python programming skills focused on data engineering applications.
- Proven experience with Terraform for cloud infrastructure deployment.
- Familiarity with CI/CD tools, especially GitHub Actions.
- Expert-level knowledge of AWS cloud services and architecture.
- Experience developing and interpreting ERDs and participating in architectural decision-making.
- Effective communication and leadership skills with prior experience managing or mentoring engineers.
Preferred Qualifications:
- Hands-on experience with big data tools such as Apache Spark, Hive, or Kafka.
- Familiarity with Docker and Kubernetes for containerization and orchestration.
- Knowledge of data governance frameworks and compliance standards (e.g., GDPR, HIPAA).
- Prior experience implementing enterprise-scale data quality and security initiatives.
