Data Engineer with Spark, Scala +Kafka
- Posted 10 days ago
- Be among the first 10 applicants
Job Description
Data Engineer with Scala , Kafka, GCP
- Should know how to develop, deploy , monitor and optimize Etl workflows for both Batch and Streaming with Scala, Spark, Airflow, BQ , Kafka , Dataproc and GCP environments
- Experience in Medallion architecture with proper Data quality checks and summarization activities
- Experience in performance optimizations, Observability and runbook creations for continuous optimizations.
- Good knowledge of Airflow orchestrations including dependency handing, idempotent execution etc.
- Exposure to data governance and information security would be a plus
- Through knowledge of handing streaming data including back pressure handling etc.
- Minimum Experience – 6+ years
More Info
About Company
Skills:
Ansible, Golang, Unix, Maven, Prometheus, Kafka, Grafana, Stash, Docker, Java, Rust, Hadoop, Dynamodb, Big Data, J2EE, Jenkins, Git, Linux, Spark, Elastic Search, Splunk, Kubernetes, Flink, Enterprise Redis DB, Ray Streaming, Humio
Skills:
snowflake , VMware, Google Cloud Platform, PostgreSQL, Sql, Terraform, MySQL, Spark, Microsoft Azure, Puppet, Oracle, AWS, Hyper-V, Chef, dbt
Skills:
Distributed Systems, Scala, Apache Kafka, Apache Spark, Sql, ELT, Etl, Data Pipelines, Cloud Databricks
Skills:
secure coding , Sql, Spring Batch, Data Modeling, Java, Python, Elasticsearch, Airflow, dbt
Skills:
Spark, Sql, Spring Batch, Java, Python, Kubernetes, Elasticsearch, Kafka, Airflow, dbt