Data Engineer, Scrape and Analysis Pipeline

Questah Staffing

Agency
Questah Staffing
Work location
Remote
Employment type
Contract
Published

Description

Maintain the scraping and analysis commands that collect the evidence behind the BestWaitlistApps.com feature ratings and write it to the comparison data. This role is with BestWaitlistApps.com. Nice to have: Experience calling large language model APIs from application code. Experience with image analysis.

Responsibilities

Maintain the scraper that renders each vendor's pages and takes screenshots. Maintain the commands that analyze the scraped text and the screenshots and write the detected features to the comparison tables. Compare the AI analysis with the stored feature marks and report where they disagree. Maintain the gap report that lists features that tracked products have and the catalog lacks. Fix failed scrapes and keep a log of what each scrape stored.

Requirements

Experience with PHP and Laravel console commands. Experience with Node and a headless browser for scraping. Comfortable with SQL and schema migrations.

Apply to this job

Applications to this job go to Questah Staffing. Create a contractor account with Questah Staffing, or sign in, and you come back to this page to apply.