Data Engineer - SQL/PySpark

Job Title : Data : Chennai, India Immediate Joiners Preferred: Notice period of up to 15 Keywords : Python Py spark ,SQL, with azure, data pipeline, Data warehouse (like snowflake) ETL control Job Description : We are seeking a highly skilled Data Engineer to join our team in Chennai. The ideal candidate should have strong expertise in Python, PySpark, SQL, Azure services, data pipelines, and data warehouse technologies like Snowflake. If you're passionate about building robust ETL processes and ensuring data quality and reliability, we want to hear from you. Responsibilities : Design, develop, and maintain scalable data pipelines using Python and PySpark. Collaborate with cross-functional teams to understand data requirements and design appropriate solutions. Implement ETL processes to extract, transform, and load data from various sources into our data warehouse (e.g., Snowflake). Optimize data pipelines for performance, reliability, and scalability. Ensure data quality and integrity throughout the ETL process. Monitor and troubleshoot production data pipelines to ensure smooth operation. Work closely with stakeholders to understand business needs and translate them into technical requirements. Provide technical guidance and support to junior team members. Requirements : Bachelor's degree in computer science, Engineering, or a related field. Proven experience as a Data Engineer or similar role. Strong programming skills in Python and proficiency in SQL. Hands-on experience with PySpark for big data processing. Familiarity with cloud platforms, particularly Azure. Experience building and maintaining data pipelines. Knowledge of data warehouse concepts and experience working with platforms like Snowflake. Strong understanding of ETL principles and best practices. Excellent problem-solving and communication skills. Ability to work independently and as part of a team in a fast-paced environment. Expert knowledge in any of the following ETL processes ETL control tables, Error logging, Auditing. Solid working experience on writing ETL jobs using standard components. Ability to spin new ETL jobs or modify the existing job based on requirements. Ability to support & troubleshoot any existing ETL jobs for failures and suggest fix to avoid failures. Build and maintain complex ETL and data pipelines to load and maintain data. Preferred Qualifications : Experience with other big data technologies such as Apache Spark, Hadoop, or Kafka. Familiarity with containerization technologies like Docker and orchestration tools like Kubernetes. Knowledge of data governance and compliance standards. If you meet the above qualifications and are ready to take on this exciting challenge, we encourage you to apply and become part of our dynamic team (ref:hirist.tech)

Location: Chennai, IN

Posted Date: 4/24/2024