Senior Big Data Engineer - Spark/Hadoop

About the Role : We are seeking a highly skilled and experienced Senior Big Data Engineer to join our dynamic team. In this role, you will be responsible for designing, developing, and maintaining our data infrastructure and pipelines. You will work with large datasets, implement scalable solutions, and ensure data quality and reliability. The ideal candidate will have a strong background in big data technologies, a passion for data, and the ability to drive innovation. Responsibilities : Design and Development : - Design and implement scalable and robust data pipelines for data ingestion, processing, and storage. - Develop and maintain data warehouses and data lakes using cloud-based and on-premise technologies. - Build and optimize ETL/ELT processes to ensure efficient data flow. - Develop and implement data quality checks and monitoring systems. Infrastructure Management : - Manage and optimize big data infrastructure, including clusters and storage systems. - Ensure high availability and performance of data platforms. - Implement and maintain data security and access control policies. Performance Optimization : - Identify and resolve performance bottlenecks in data pipelines and systems. - Optimize queries and data structures for efficient data retrieval. - Monitor and analyze system performance to ensure optimal resource utilization. Collaboration and Communication : - Collaborate with data scientists, analysts, and other engineers to understand data requirements and deliver solutions. - Communicate effectively with stakeholders to provide updates and insights on data projects. - Document technical designs, processes, and best practices. Innovation and Research : - Stay up-to-date with the latest big data technologies and trends. - Evaluate and recommend new tools and technologies to improve data infrastructure and processes. - Contribute to the development of data engineering best practices. Required Technical Skills : Big Data Technologies : - Apache Hadoop (HDFS, MapReduce, YARN) - Apache Spark (Spark Core, Spark SQL, Spark Streaming) - Apache Kafka - Apache Hive, Apache Impala Cloud platforms : - AWS (EMR, S3, Glue, Redshift), Google Cloud Platform (Dataproc, BigQuery, Dataflow), or Azure (HDInsight, Data Lake Storage, Azure Synapse Analytics) Programming Languages : - Python (Pandas, NumPy, PySpark) - Scala - SQL Data Warehousing and Databases : - Data warehousing concepts and principles - Relational databases (e.g., PostgreSQL, MySQL) - NoSQL databases (e.g., Cassandra, MongoDB) ETL/ELT Tools : - Apache Airflow, Luigi, or similar orchestration tools. Version Control : - Git Operating Systems : - Linux/Unix Preferred Skills : - Experience with containerization (Docker, Kubernetes). - Experience with data visualization tools (e.g., Tableau, Power BI). - Experience with machine learning pipelines. - Understanding of data governance and compliance. - Experience with streaming data processing. Qualifications : - Bachelor's or Master's degree in Computer Science, Engineering, or a related field. - 5-8 years of experience1 in big data engineering. - Proven experience in designing and implementing large-scale data solutions. - Strong problem-solving and analytical skills. - Excellent communication and collaboration skills. (ref:hirist.tech)

Location: in, IN

Posted Date: 5/1/2025