Senior AWS Data Engineer - Python Programming

Hiring for one of our Big4 client : Job Title : Senior AWS Data Engineer Location : Bangalore Job Type : Full-Time Overview : We are looking for an experienced Senior AWS Data Engineer (P1) with 5 years of hands-on experience in building scalable, reliable data pipelines, and data engineering solutions using AWS services. The ideal candidate will have expertise in AWS Glue, Lambda, Redshift, Step Functions, CloudWatch, and advanced Python/PySpark skills. You will be responsible for designing, implementing, and optimizing ETL/ELT pipelines, managing data integration workflows, and ensuring high-performance data solutions on AWS. This role requires deep technical expertise in cloud-based data platforms, excellent problem-solving skills, and the ability to lead complex data projects while working collaboratively with business stakeholders. Key Responsibilities : Lead ETL/ELT Pipeline Development : - Architect, design, and implement advanced ETL/ELT pipelines using AWS Glue, Lambda, PySpark, and SQL, ensuring efficient data integration and transformation. Data Warehousing & Optimization : - Optimize Amazon Redshift data warehouse performance, design schema structures, and develop efficient data models to handle large-scale data & Workflow Automation : - Use AWS Step Functions, Airflow, and CloudWatch for orchestrating data workflows, automating tasks, and ensuring smooth pipeline operations. Cloud Services Integration : - Leverage a broad set of AWS services, including API Gateway, S3, SQS, SNS, SES, DMS, CloudFormation, CDK, and IAM, to integrate various data sources, manage permissions, and automate data processes. Technical Leadership : - Provide guidance and mentorship to junior engineers, help them grow in their technical skills, and ensure best practices for coding, testing, and deployment. Solution Design & Development : - Work closely with business analysts and product owners to translate functional requirements into high-performance, scalable technical solutions on AWS. Data Quality & Monitoring : - Utilize CloudWatch for monitoring data pipelines, ensure optimal performance, and troubleshoot issues in production environments. Security & Compliance : - Implement best practices for data security, access control using IAM, and ensure compliance with data governance and regulatory & Process Standardization : - Create comprehensive technical documentation, including system designs, data models, and pipeline configurations. Standardize best practices across the team. Primary Skills Required : AWS Services Expertise : - Extensive experience with AWS services, including Glue, Lambda, Step Functions, Redshift, S3, API Gateway, SQS, SNS, SES, DMS, CloudFormation, CDK, IAM, and VPC. Programming : - Strong proficiency in Python, PySpark, and SQL, with hands-on experience in developing data transformation scripts and automation for large-scale data processing. ETL/ELT Pipelines : - Proven experience designing and implementing ETL/ELT pipelines using AWS Glue, Lambda, and other AWS services to efficiently process and transform data. Data Warehousing : - Expertise in Amazon Redshift for data warehousing, including schema design, query optimization, performance tuning, and data & Workflow Management : - Advanced experience with AWS Step Functions, Airflow, and CloudWatch to manage and orchestrate data workflows, monitor pipeline health, and ensure process efficiency. Cloud Infrastructure & Automation : - Strong experience with CloudFormation and CDK for infrastructure automation, and managing resources in AWS. Security & Permissions Management : - Deep knowledge of IAM for managing security and access control, ensuring secure data & Debugging : - Expertise in monitoring data pipelines using CloudWatch, identifying bottlenecks, and resolving issues in data processes. Additional Skills (Nice to Have) : - Experience with Data Migration Service (DMS) for database replication and migration. - Familiarity with Airflow or other orchestration frameworks for data workflows. - Strong understanding of data governance and compliance standards in cloud environments. - Knowledge of Agile development methodologies and proficiency with Git for version control. - AWS certifications in relevant areas (e.g., AWS Certified Solutions Architect, AWS Certified Data Analytics). (ref:hirist.tech)

Location: bangalore, IN

Posted Date: 5/1/2025