Data Engineer - Google Cloud Platform

Responsibilities : Data Pipeline Development & Management : - Design, build, and maintain highly scalable and efficient data pipelines on Google Cloud Platform (GCP). - Develop and implement ETL/ELT processes to ingest, transform, and load large datasets. - Optimize data workflows for performance, reliability, and cost-effectiveness. GCP Service Utilization : - Leverage and optimize Google Cloud Platform services, including BigQuery, Dataflow, Pub/Sub, Cloud Storage, and Cloud Functions. - Utilize Dataflow for stream and batch data processing. - Implement Pub/Sub for real-time data ingestion and messaging. - Design and manage data warehouses within BigQuery. Data Integrity & Security : - Ensure data integrity, quality, and consistency across all data pipelines. - Implement and maintain data security measures, adhering to best practices and compliance requirements. - Monitor data pipelines for errors and anomalies, and implement proactive measures to prevent issues. Collaboration & Communication : - Collaborate effectively with cross-functional teams, including data scientists, analysts, and software engineers, in an agile environment. - Communicate technical concepts clearly and concisely to both technical and non-technical stakeholders. - Participate in code reviews and contribute to the development of best practices. Troubleshooting & Problem Solving : - Troubleshoot and resolve complex data-related issues, ensuring minimal downtime and disruption. - Perform root cause analysis and implement long-term solutions to prevent recurring issues. Required Skills and Experience : Experience : - 4 years of hands-on experience as a Data Engineer, with a strong emphasis on Google Cloud Platform (GCP). GCP Expertise : - Extensive experience with core GCP services, including: - BigQuery (data warehousing and analytics) - Dataflow (stream and batch data processing) - Pub/Sub (real-time messaging) - Cloud Storage. - Cloud Functions. - Strong understanding of GCP data architecture and best practices. Programming & Scripting : - Proficiency in Python for data processing and automation. - Strong SQL skills for data querying and manipulation. Infrastructure & Automation : - Knowledge of Infrastructure as Code (IaC) using Terraform (preferred). - Experience with CI/CD pipelines for automated deployments. Problem Solving & Communication : - Strong problem-solving and analytical skills. - Excellent communication and collaboration skills. - Ability to work Qualifications : Experience : - 5 years of experience in GCP Data Engineering. - Experience in optimizing data workflows and implementing complex ETL/ELT processes. Advanced Skills : - Experience with DataProc. - Experience with Airflow or Cloud Composer. - Experience with Data Catalog. (ref:hirist.tech)

Location: pune, IN

Posted Date: 5/1/2025