Data Engineer

Job Description : We are seeking a highly skilled and experienced Data Engineer with a strong background in Python, machine learning (ML) frameworks, and big data technologies. In this pivotal role, you will design, build, and maintain robust data pipelines to handle massive datasets, enabling advanced analytics and ML applications within our supply chain and e-commerce domains. You will collaborate closely with data scientists, analysts, and product teams to drive data-driven decision-making and innovation. Key Responsibilities : - Design, build, and maintain scalable and efficient data pipelines to ingest, process, and transform large volumes of data from various sources (transactions, event data, supply chain, inventory, etc. - Implement ELT (Extract, Load, Transform) processes to ensure data quality and consistency for downstream analytics and ML applications. - Optimize data pipelines for performance, reliability, and cost-effectiveness. - Utilize big data technologies such as Spark, Hadoop, and Databricks to process and analyze large datasets. - Develop and deploy data pipelines on cloud-based data platforms and services (AWS, GCP, Azure), leveraging cloud-native tools and services. - Manage and optimize cloud data infrastructure for scalability and performance. - Collaborate with data scientists to integrate ML models into data pipelines and production systems. - Develop and maintain data pipelines to support ML model training, validation, and deployment. - Utilize ML frameworks such as TensorFlow, PyTorch, and Scikit-learn for data preprocessing and feature engineering. - Design and implement real-time data processing systems using technologies such as Kafka or Kinesis. - Develop streaming data pipelines to support real-time analytics and decision-making. - Design and implement data warehousing solutions to support business intelligence and reporting. - Collaborate with data analysts to develop and implement A/B testing frameworks for validating hypotheses and optimizing growth campaigns. - Support data analysis for product recommendations, and UI/UX experiments. - Apply data engineering and ML expertise to solve real-world problems in supply chain and e-commerce domains, such as demand forecasting, inventory optimization, and customer behavior analysis. - Develop data solutions to improve supply chain efficiency, reduce costs, and enhance customer experience. - Collaborate with cross-functional teams, including data scientists, analysts, and product managers, to deliver data-driven solutions. - Communicate technical concepts and solutions effectively to both technical and non-technical stakeholders. - Contribute to code reviews, and promote best practices. Qualifications and Skills : Education : - Bachelor's or Master's degree in Computer Science, Data Science, or a related field. Experience : - Proven experience as a Data Engineer, with a focus on big data and ML. - Strong hands-on experience in Python programming. - Expertise in big data technologies (Spark, Hadoop, Databricks). - Proficiency in cloud-based data platforms and services (AWS, GCP, Azure). - Experience with ML frameworks (TensorFlow, PyTorch, Scikit-learn). - Experience with real-time data processing (Kafka, Kinesis). - Experience with data warehousing solutions and ELT processes. Technical Skills : - Strong understanding of data modeling, data warehousing, and database design. - Proficiency in SQL and NoSQL databases. - Experience with data visualization tools (e.g., Tableau, Power BI) is a plus. - Experience with containerization tools like Docker and Kubernetes is a plus (ref:hirist.tech)

Location: bangalore, IN

Posted Date: 5/1/2025