AI Response Labeler / Annotator
Blueprint · Data · Junior · 23d ago
$26k – $30k
per year
Open to candidates across LATAM.
Applies on the original listing · via Himalayas
Who can apply
Read straight from the listing, so you know before you spend time on it.
Open to candidates across LATAM.
“This is a remote role open to candidates in Latin American countries.”
Requires 9:00 a.m. to 5:00 p.m. Pacific Time during 30-day onboarding, followed by standard local business hours in Latin America.
- Async-first
- The listing doesn't promise async-first working.
- Contractor / B2B
- Employment is managed through a local PEO or Employer of Record (EOR) partner.
- Global stipend
- No home office or equipment budget mentioned.
- Annual retreat
- No company-paid get-togethers mentioned.
Read from the listing automatically. Always confirm the details with the company before you accept an offer.
About the role
In this role, you will evaluate the quality of AI-generated responses by conducting side-by-side comparisons using detailed annotation rubrics. You will assess model outputs across varied tasks—such as Q&A, search relevance, multi-turn conversations, and file-based tasks—judging factual accuracy, instruction following, reasoning, tone, and clarity. Candidates will complete a structured 30-day onboarding program with daily benchmarks of approximately 25 tasks per day.
What you'll do
- Perform side-by-side comparisons of AI-generated outputs and choose the superior response based on established guidelines.
- Evaluate responses for factual accuracy, reasoning, relevance, completeness, clarity, safety, and instruction adherence.
- Identify critical flaws including hallucinations, unsupported claims, weak logic, and missed prompt constraints.
- Draft concise, evidence-based rationales justifying evaluation decisions when required.
- Maintain daily productivity standards of approximately 25 completed tasks while meeting quality benchmarks.
What you bring
- Excellent written English comprehension and communication skills.
- Strong critical-thinking skills with sound judgment in evaluating factuality and nuanced differences in text quality.
- High attention to detail and ability to apply complex, scenario-specific guidelines consistently.
- Comfort with structured, repetitive evaluation tasks in a high-volume work environment.
- Ability to work 9:00 a.m. to 5:00 p.m. Pacific Time during the initial ~30-day qualification period.
About Blueprint
Blueprint is a technology solutions firm headquartered in Bellevue, Washington, that connects strategy and execution across AI, cloud, data, and emerging technology.
View Blueprint profile & open roles →