Key Responsibilities:
- Design and develop scalable data pipelines across Hadoop (Hive, Impala, Spark, Kafka, Iceberg) and Teradata environments.
- Build ingestion and transformation frameworks using Java, Spark, Python and shell scripts.
- Develop full stack applications and internal tools using Python, Shell scripting, and modern web frameworks (e.g., Flask, React).
- Create APIs and microservices to expose data and ML models securely to downstream systems and user interfaces.
- Collaborate with data scientists to operationalize ML models using Cloudera Machine Learning (CML)
- Build and deploy GenAI/LLM-powered applications for intelligent data interaction, summarization, and automation.
- Implement enterprise-grade security controls including RBAC, LDAP, Kerberos, Apache Ranger, and row-level access.
- Tune and optimize data applications for performance across Hadoop and Teradata, ensuring efficient resource utilization.
- Support sandbox environments for prototyping, enabling users to build ML models, dashboards, and data pipelines.
Required Skills & Experience:
- Data Engineering: Strong experience with Hadoop ecosystem (Hive, Impala, Spark, Kafka, Iceberg, Ranger, Atlas), Teradata and data pipeline orchestration.
- Full Stack Development: Proficiency in Python, Shell scripting, REST APIs, and web frameworks (Flask, React, etc.).
- Machine Learning & AI: Hands-on experience with ML platforms (CML), Spark MLlib, Python ML libraries (scikit-learn, XGBoost), and model deployment.
- GenAI/LLM Applications: Familiarity with building applications using large language models (e.g., OpenAI, Hugging Face, LangChain) for enterprise use cases.
- Security & Governance: Experience with enterprise data security (LDAP, Kerberos, RBAC), data masking, and access control.
- Performance Tuning: Proven ability to optimize data applications and queries in large-scale environments (Hadoop, Teradata).
- Tools & Platforms: Cloudera Data Platform (CDP), Informatica, QlikSense, Apache Oozie, Git, CI/CD pipelines.
- Soft Skills: Strong analytical and problem-solving skills, excellent communication, and ability to work in cross-functional teams.
