T
AWS Databrics ,ADF, Pyspark , Python , Terraform
T
- Posted a day ago
- Be among the first 10 applicants
Job Description
Hiring: AWS Databricks Data Engineer
We are looking for an experienced AWS Databricks Data Engineer with strong expertise in Databricks, Azure Data Factory (ADF), PySpark, Python, Terraform, and AWS Cloud.
If you have experience building scalable cloud-based data pipelines and are looking for your next opportunity, we'd love to connect!
Job Details
- Role: AWS Databricks Data Engineer
- Experience: 6–9 Years
- Locations: Indore / Pune / Hyderabad
- Notice Period: Immediate Joiners / 30 / 60 / 90 Days
- Key Skills: AWS, Databricks, ADF, PySpark, Python, Terraform
Key Responsibilities
- Design, develop, and maintain scalable data engineering solutions using Databricks.
- Develop high-performance data processing pipelines using PySpark and Python.
- Build and manage ETL/ELT workflows using Azure Data Factory (ADF).
- Work with AWS cloud services for data storage, processing, integration, and deployment.
- Design and implement efficient data ingestion and transformation frameworks.
- Develop and maintain cloud infrastructure using Terraform / Infrastructure as Code (IaC).
- Optimize Databricks jobs, Spark workloads, and data pipelines for performance and scalability.
- Implement data quality, monitoring, logging, and error-handling mechanisms.
- Troubleshoot production issues related to data pipelines and distributed processing.
- Collaborate with architects, analysts, DevOps teams, and business stakeholders to deliver reliable data solutions.
- Follow CI/CD, version control, security, and cloud engineering best practices.
Required Skills
- 6–9 years of experience in Data Engineering / Cloud Data Engineering.
- Strong hands-on experience with Databricks.
- Strong development experience in PySpark and Python.
- Hands-on experience with Azure Data Factory (ADF).
- Good experience working with AWS Cloud.
- Hands-on knowledge of Terraform and Infrastructure as Code.
- Strong understanding of ETL/ELT, data pipelines, data transformation, and distributed data processing.
- Experience working with large datasets and optimizing Spark-based workloads.
- Good understanding of SQL, data modeling, and cloud data architectures.
- Strong analytical, troubleshooting, and communication skills.
More Info
Key Skills
About Company
Skills:
Python, LangChain LangGraph CrewAI, Testing and clean Git workflows, FastAPI or similar frameworks, API integrations REST webhooks, Prompt structure and output handling, Understanding of how RAG systems use data, LLM API integration OpenAI Anthropic, Basic BI analytics tools, Docker CI-CD, Exposure to vector databases pgvector Pinecone, AWS GCP experience, Strong SQL and database design, Airflow Prefect or pipeline orchestration
Skills:
code coverage , Integration Testing, Solid Principles, Design Patterns, Agile Methodologies, Unit Testing, Sql, Microservices, Version Control, Django, Cloud, Python, CLEAN Code, Automated tests, NoSQL Databases, CI CD pipelines
Skills:
Docker, FastAPI, Datadog, Python, Lambda, ECS, AWS, LangChain, Vector Databases, Uvicorn, ddtrace
Skills:
Google Cloud Platform (GCP), Django, Numpy, Pandas, Flask, Data Structures And Algorithms, Python, Testing, Version Control, Tensorflow, Matplotlib, Pytorch, Rest Apis, Azure, AWS, Scikit-learn, Software development best practices, CI/CD
Skills:
RDS, Pyspark, AWS Glue, Dynamodb, Emr, Sql, Lambda, Cloudwatch, Sqs, Iam, Sns, Python, AWS, Lake Formation, CloudTrail