Data Engineer - Hadoop/Python/Spark

Skills : - Hadoop - Python - Spark - PySpark - ETL (Extract, Transform, Load) Roles & Responsibilities : - Data Ingestion: Develop and maintain data pipelines for ingesting raw data from various sources into the Hadoop ecosystem. - Data Processing: Utilize Python and Spark to process and transform large volumes of data efficiently, ensuring scalability and performance. - Data Modeling: Design and implement data models that facilitate efficient querying and analysis of structured and unstructured data. - ETL Development: Develop ETL (Extract, Transform, Load) processes to cleanse, transform, and integrate data from different sources into data warehouses or data lakes. - Performance Tuning: Optimize data processing workflows and queries to improve performance and reduce processing times. - Data Quality Assurance: Implement data quality checks and validation processes to ensure the accuracy, completeness, and consistency of data. - Data Governance: Enforce data governance policies and standards to ensure compliance with regulatory requirements and industry best practices. - Documentation: Maintain documentation for data pipelines, data models, and ETL processes to facilitate knowledge sharing and troubleshooting. - Collaboration: Collaborate with cross-functional teams including data scientists, analysts, and business stakeholders to understand data requirements and deliver solutions that meet business needs. - Continuous Improvement: Stay updated with emerging technologies and best practices in data engineering, and proactively identify opportunities for process optimization and automation. (ref:hirist.tech)

Location: India, IN

Posted Date: 5/3/2024