Design, build, productionize, and support scalable document parsing and content extraction pipelines on a modern hybrid cloud platform, handling structured and unstructured data including PDFs, PPTX, Excel, Word, images, scanned documents, videos, audios, charts, tables etc
Design and implement OCR, VLM, and multimodal AI pipelines leveraging state-of-the-art models (e.g., GPT- [4/5]o, vision-language models, Text-to-Speech) for accurate content extraction, layout understanding, and table/chart interpretation
Develop and optimize embedding models, including fine-tuning and domain adaptation to support organization-specific terminology, keywords, and semantic search use cases
Implement knowledge ingestion, indexing, and retrieval pipelines for RAG (Retrieval-Augmented Generation) and agentic GenAI systems, ensuring high recall, precision, and low-latency retrieval
Make an Impact by:
Build and maintain data ingestion pipelines for batch and streaming data sources using tools/library like Kafka, LlamaIndex, LangChain, PyMuPDF, MSAL etc.
Implement layout-aware parsing to accurately preserve document structure, hierarchy, and reading order (using OCR, VLM, and multimodal AI solutions).
Extract accurate numeric and semantic data from complex charts and irregular tables while preventing hallucinations.
Assist in integrating data from diverse source systems (files, APIs, databases, streaming)
Implement, fine-tune, and optimize embedding models to support organization-specific terminology, improving semantic search, retrieval accuracy, and downstream GenAI performance.
Help maintain metadata and pipeline documentation for transparency and traceability
Participate in integrating pipelines with tools such as Microsoft Fabric, Databricks, Delta Lake, and other platform components
Build and maintain knowledge base and RAG solution on cloud (Azure)
Implement and operate knowledge base storage, lifecycle management and embedding/vectorization
Contribute to automation efforts using version control and CI/CD workflows
Apply basic data governance and access control policies during implementation.
Skills for Success:
Bachelor's degree in Computer Science, Engineering, or a related field
1-3 years of experience in data engineering or data platform development. Fresh graduates encouraged to apply as well.
Proven ability to independently build basic batch or streaming document processing pipelines
Hands-on experience with Python, PyTorch, PEFT, LoRA, OCR, VLM, LLM, prompting, Text-to-Speech models and SQL for data transformation and validation
Familiarity with Apache Spark (especially PySpark) and large-scale data processing concepts
Experience with implementing knowledge base and RAG solutions for agentic AI use cases
Self-starter with strong problem-solving skills and a keen attention to detail
Able to work independently while collaborating effectively with senior engineers and other stakeholders
