Staff Software Engineer

As a Staff Software Engineer for the Pricing and Assortment Trust team, I build robust distributed systems for crawling, extracting large volumes of assortment and pricing data. The goal is price and product intelligence from a variety of eCommerce sites. In this role, you will help us always meet the customer trust on price and choice expectations of our customers by designing and maintaining resilient, distributed data collection applications that overcome throttling and blocking challenges inherent in the real-world, high-volume : - Serve as a technical expert and help us build and scale our e-commerce data extraction systems. - In this role, you will design, optimize, and operate high-performance API reverse engineering solutions and web scrapers to extract product data, pricing, inventory, and other key insights from major online retailers and marketplaces. - Develop and optimize large-scale web scraping pipelines to extract millions of e-commerce product pages daily. - Reverse-engineer e-commerce websites to identify the best scraping strategies, including API interception, headless browsing, and dynamic content rendering. - Build distributed crawling solutions using AWS (EC2 Lambda, S3), Kubernetes, or other cloud computing environments. - Implement anti-bot evasion techniques, including IP rotation, browser fingerprinting, CAPTCHA solving, and request throttling. - Monitor, debug, and maintain scraping scripts to ensure uninterrupted data extraction, adapting to site changes in real-time. - Process and clean extracted data efficiently, optimizing storage in SQL/NoSQL databases or cloud-based data lakes. - Collaborate with data and engineering teams to integrate scraped data into analytics pipelines and real-time monitoring : - At least 7 years of experience working in web scraping, data extraction, or related fields. - Proven experience in high-volume, high-frequency data collection, capable of extracting millions of records per hour while maintaining efficiency and reliability. - Strong experience with headless browsers (Playwright, Puppeteer, Selenium) for scraping JavaScript-heavy sites. - Proficiency in Python, Node.js, or similar scripting languages. Java experience is a bonus. - Deep understanding of e-commerce site structures, including pagination and API calls. - Experience with proxy management, IP rotation, and bot mitigation techniques at scale. - Hands-on experience with AWS (EC2 Lambda, S3 CloudWatch), Kubernetes, or containerized workloads for distributed scraping. - Familiarity with data storage formats (JSON, CSV, Parquet) and database technologies (SQL, NoSQL, DynamoDB, Elasticsearch, etc. ). Nice to Have : - Expertise with reverse engineering mobile apps and mobile app APIs. - Expertise in real-time data processing pipelines and message queues (Kafka, RabbitMQ, SQS). - Understanding of pricing intelligence and price monitoring - Contributions to open-source web scraping frameworks. (ref:hirist.tech)

Location: bangalore, IN

Posted Date: 5/1/2025