Role Overview:
ETL & Data Engineer to design, build, and maintain robust data pipelines and backend services that power AI-driven operations. The role involves working with high-volume IT and cloud data, optimizing ETL processes, and integrating with AIOps platforms and ML pipelines.
Key Responsibilities:
- Build and maintain scalable ETL pipelines for batch and real-time data ingestion, transformation, and loading from diverse sources (IT infrastructure, cloud, monitoring systems, APIs).
- Implement data validation, cleansing, and normalization for consistent AI model input.
- Develop backend services and APIs to support data ingestion, metadata management, and configuration.
- Optimize ETL jobs for performance, fault tolerance, and low latency.
- Integrate with AIOps platforms and ML pipelines using REST APIs or event-driven architectures.
- Schedule and monitor ETL workflows using tools like Airflow, Prefect, or Dagster.
- Support CI/CD pipelines for deploying ETL services and full-stack applications.
Required Skills & Tools:
- Programming & Scripting: Python, Go (Golang), Java, Ruby, JavaScript/TypeScript (Next.js)
- ETL & Data Engineering: Apache NiFi, Spark, Airflow, Flink, Kafka, Talend
- Orchestration: Airflow, Prefect, Dagster
- Data Storage & Lakes: PostgreSQL, MongoDB, Elasticsearch, Snowflake, BigQuery, S3, GCS, Azure Blob
- Streaming Platforms: Kafka, Kinesis, Pub/Sub
Good to Have:
- Experience with AIOps & Observability tools like Splunk, Dynatrace, AppDynamics, New Relic, Elastic Stack
- Familiarity with ITSM systems (ServiceNow) and CMDB integrations
- Understanding of metrics, logs, and traces for AI-driven operations
