Open to candidates across EMEA, APAC.
Direct from the employer's own careers page
Who can apply
Read straight from the listing, so you know before you spend time on it.
Open to candidates across EMEA, APAC.
“Europe, Middle East, Africa, Asia-Pacific (EMEA, APAC)”
A fully remote team of around 270 people working across 30+ countries.
- Async-first
- Circle operates highly asynchronously, documents heavily, and does not track working hours.
- Contractor / B2B
- No mention of hiring international contractors or invoicing.
- Global stipend
- No home office or equipment budget mentioned.
- Annual retreat
- Circle hosts all-company retreats twice a year in international destinations.
Read from the listing automatically. Always confirm the details with the company before you accept an offer.
About the role
The AI Quality engineering team builds the foundation for measuring, diagnosing, and improving the quality of Circle's AI-powered features in production. As a player-coach, you will design evaluation infrastructure, trace failures across multi-turn agent pipelines, build evaluation datasets, and lead a growing team of engineers while remaining hands-on with the codebase.
What you'll do
- Build and own evaluation infrastructure, including CI/CD pipelines, scorers, and datasets for Circle's AI agents.
- Diagnose agent pipeline failures across plan creation, execution, and tool selection, turning findings into prototypes or priorities for core AI engineering.
- Develop annotation workflows, simulated conversations, and AI-generated datasets to expand evaluation coverage.
- Run structured experiments across prompts, models, and harnesses to optimize cost, latency, and agent performance.
- Lead and manage the technical direction and day-to-day priorities of the AI Quality engineering team while staying hands-on.
- Collaborate closely with AI Core engineers to provide confidence that product changes reliably improve quality.
What you bring
- 7+ years of experience building and shipping production software, ideally with complex, tool-using LLM agent systems.
- Comfortable working in Ruby on Rails and Python, or capable of quickly picking them up.
- Proven experience building ML/AI evaluation pipelines, scorers, dashboards, or CI/CD using frameworks like LangSmith or Braintrust.
- Experience designing datasets, annotation workflows, or labeling pipelines for ML/AI evaluation.
- Proficiency in English at a CEFR Level C2 / ILR Level 5.
About Circle
Circle is an all-in-one community and creator platform enabling businesses and creators to unite their audience through chat, courses, livestreams, and events. It operates as a fully remote team of 270+ people across more than 30 countries.
View Circle profile & open roles →