AWS Data Engineer - Solution Design & Implementation

AWS Data Engineer - Consultant Bengaluru, Karnataka, India Locations : Bangalore, Coimbatore, Delhi NCR, Mumbai Experience : 3 to 5 years We are seeking experienced AWS Data Engineers to join our consulting team and drive impactful data solutions for our clients. As an AWS Data Engineer Consultant, you will design, implement, and maintain robust data pipelines and analytics solutions using a wide range of AWS services. The ideal candidate will have a strong background in AWS data services, big data technologies, and programming languages, with a passion for solving complex data challenges. Key Responsibilities : Solution Design and Implementation : - Design and implement scalable, high-performance data pipelines and data warehousing solutions using AWS services such as Glue, EMR, Lambda, S3, Redshift, and related technologies. - Develop and optimize ETL/ELT processes to ingest, transform, and load data from diverse sources into data lakes and data warehouses. - Build and maintain data lakes on Amazon S3 using technologies like Delta Lake or Apache Iceberg, ensuring data quality, governance, and accessibility. - Create and manage analytics solutions using Amazon Athena, Redshift Spectrum, and other query services for efficient data analysis and reporting. - Design and implement database solutions using Amazon Aurora, RDS (PostgreSQL, MySQL), DynamoDB, and other database services, considering factors like performance, scalability, and cost. - Develop serverless data processing workflows using AWS Lambda and Step Functions for orchestration and automation. Data Engineering & Programming : - Write efficient, maintainable, and testable code using Python, PySpark, and SQL (including PostgreSQL). - Develop data processing jobs using Spark on EMR for large-scale data transformation and analysis. - Implement data quality checks, data validation, and data cleansing procedures to ensure data accuracy and reliability. - Utilize infrastructure-as-code (IaC) principles and tools (e.g., AWS CloudFormation, Terraform) to automate the provisioning and management of data infrastructure. Collaboration & Consulting : - Collaborate with data scientists, analysts, and business stakeholders to understand their data requirements and provide technical expertise. - Work closely with clients to gather business requirements, define project scope, and deliver solutions that meet their needs. - Provide technical guidance and support to junior data engineers. - Participate in pre-sales activities, including solution design, estimation, and proposal development. Data Operations & Optimization : - Optimize data architectures for performance, scalability, reliability, and cost-efficiency. - Troubleshoot and resolve data pipeline and infrastructure issues, ensuring minimal downtime and data loss. - Monitor data pipeline performance and implement optimizations to improve throughput and reduce latency. - Implement data security best practices to protect sensitive data and ensure compliance with industry standards (e.g., GDPR, HIPAA). - Implement CI/CD pipelines for data engineering workflows to automate testing, deployment, and monitoring. Technical Skills : AWS Services : Core : Glue, EMR, Lambda, S3, EC2, IAM Data Warehousing & Analytics : Redshift, Athena, Kinesis Databases : Aurora (PostgreSQL, MySQL), RDS (PostgreSQL, MySQL), DynamoDB Workflow & Orchestration : Step Functions, Airflow on MWAA Monitoring & Logging : CloudWatch Infrastructure as Code : CloudFormation, Terraform (preferred) Big Data Technologies : Hadoop, Spark - Data Lake Technologies : Delta Lake, Apache Iceberg Programming : Proficiency : Python, PySpark Scripting : Bash Databases : - Strong SQL skills (PostgreSQL, ANSI SQL) - NoSQL databases (DynamoDB) Data Warehousing and Analytics : - Data modeling (dimensional, star schema) - ETL/ELT processes, data pipeline design Data Lake Architecture : - Data lake design principles, data governance DevOps : - Version control (Git) - CI/CD pipelines - Agile methodologies Required Qualifications : - Bachelor's degree in Computer Science, Information Technology, or a related field. - 3 to 5 years of hands-on experience as a Data Engineer, with at least 60% of that experience focused on the AWS platform. - Strong proficiency in designing, building, and deploying data solutions using AWS data services (Glue, EMR, Lambda, S3, Redshift). - Solid experience with data lake technologies, particularly Delta Lake or Apache Iceberg. - Expertise in database systems, including Amazon Aurora (PostgreSQL), Amazon RDS (PostgreSQL), and Amazon DynamoDB. - Proficiency in Python and PySpark programming for data processing and analysis. - Strong SQL skills and experience with PostgreSQL (or a similar relational database). - Experience with AWS Step Functions (or similar workflow orchestration tools) for building data processing pipelines. - Familiarity with data modeling, schema design, and data warehousing concepts. - Knowledge of data security and compliance requirements and best practices. - Excellent problem-solving and analytical skills, with a focus on delivering high-quality solutions. - Strong communication, collaboration, and interpersonal abilities. Preferred Qualifications : AWS Certifications : - AWS Certified Data Engineer - Associate - AWS Certified Solutions Architect - Associate or Professional - AWS Certified Database - Specialty - Experience with real-time data processing using Amazon Kinesis or Apache Kafka. - Knowledge of machine learning workflows on AWS (e.g., Amazon SageMaker). - Experience with containerization technologies (Docker, Kubernetes). - Experience with other cloud platforms (e.g., Azure, GCP). - Contributions to open-source projects related to data engineering. Benefits : - Work with one of the Big 4 in India - Competitive salary and benefits package. - Opportunity to work on cutting-edge data technologies and challenging projects. - Collaborative and supportive work environment. - Professional development and growth opportunities. (ref:hirist.tech)

Location: in, IN

Posted Date: 5/1/2025