Data Engineer - Spark/Hadoop

About the Role : We are looking for a skilled and motivated Data Engineer to design, build, and maintain robust data pipelines and ETL processes that power insights and decisions across our q-commerce platform. You will work closely with data analysts, product teams, and engineering to ensure the availability, accuracy, and performance of our data infrastructure. This role is ideal for someone who thrives in a fast-paced environment and is passionate about solving complex data problems at scale. Key Responsibilities : - Design and develop scalable ETL/ELT pipelines to ingest, transform, and store data from various sources. - Build and maintain data warehouses, data lakes, and analytics platforms. - Work with structured and unstructured data across SQL, NoSQL, and cloud storage systems. - Optimize data workflows for performance, scalability, and cost-efficiency. - Collaborate with data analysts and data scientists to support their data needs and ensure data accuracy. - Implement data quality checks, validation routines, and monitoring systems. - Manage and monitor data jobs, ensuring high availability and minimal latency. - Integrate with third-party APIs and internal services for real-time and batch data processing. - Maintain documentation of data architecture, schemas, and pipelines. - Ensure data security, privacy, and compliance with relevant policies and regulations. Required Skills & Qualifications : - Experience: 35 years in data engineering or a similar role. - Strong proficiency in SQL and experience with relational databases like PostgreSQL, MySQL, or Redshift. - Hands-on experience with ETL tools and workflow orchestration platforms such as Airflow, Luigi, or Prefect. - Experience with big data frameworks (e.g., Spark, Hadoop) is a plus. - Proficiency in Python, Scala, or Java for data manipulation and pipeline development. - Familiarity with cloud platforms like AWS (S3, EMR, Glue, Redshift, Lambda), GCP, or Azure. - Experience with NoSQL databases (e.g., MongoDB, Cassandra) and streaming tools like Kafka or Kinesis. - Understanding of data modeling, partitioning, and indexing strategies for performance tuning. - Strong debugging and problem-solving skills. - Experience working in agile environments and using tools like Git, JIRA, and Confluence. (ref:hirist.tech)

Location: navi-mumbai, IN

Posted Date: 5/1/2025