Data Engineer

Data Engineer

Key Responsibilities:

Design and develop scalable data pipelines across Hadoop (Hive, Impala,Spark, Kafka, Iceberg) and Teradata environments.

Build ingestion and transformation frameworks using Java, Spark, Python and

shell scripts.

Develop full stack applications and internal tools using Python, Shell scripting,

and modern web frameworks (e.g., Flask, React).

Create APIs and microservices to expose data and ML models securely to

downstream systems and user interfaces.

Collaborate with data scientists to operationalize ML models using Cloudera

Machine Learning (CML)

Build and deploy GenAI/LLM-powered applications for intelligent data

interaction, summarization, and automation.

Implement enterprise-grade security controls including RBAC, LDAP,

Kerberos, Apache Ranger, and row-level access.

Tune and optimize data applications for performance across Hadoop and

Teradata, ensuring efficient resource utilization.

Support sandbox environments for prototyping, enabling users to build ML

models, dashboards, and data pipelines.

Required Skills & Experience:

Data Engineering:Strong experience with Hadoop ecosystem (Hive, Impala,

Spark, Kafka, Iceberg, Ranger, Atlas), Teradata and data pipeline

orchestration.

Full Stack Development:Proficiency in Python, Shell scripting, REST APIs,

and web frameworks (Flask, React, etc.).

Machine Learning & AI:Hands-on experience with ML platforms (CML),

Spark MLlib, Python ML libraries (scikit-learn, XGBoost), and model

deployment.

GenAI/LLM Applications:Familiarity with building applications using large

language models (e.g., OpenAI, Hugging Face, LangChain) for enterprise use

cases.

Security & Governance:Experience with enterprise data security (LDAP,

Kerberos, RBAC), data masking, and access control.

Performance Tuning:Proven ability to optimize data applications and

queries in large-scale environments (Hadoop, Teradata).

Tools & Platforms:Cloudera Data Platform (CDP), Informatica, QlikSense,

Apache Oozie, Git, CI/CD pipelines.

Soft Skills:Strong analytical and problem-solving skills, excellent

communication, and ability to work in cross-functional teams.