Freelance Agent Evaluation Engineer

About the Role
the company connects specialists with project-based AI opportunities for leading tech companies, focusing on testing, evaluating, and improving AI systems. Participation is project-based and not permanent employment.

We are building a dataset to evaluate AI coding agents by assessing how well a model handles real-world developer tasks. In this role, you will create challenging tasks and evaluation criteria within realistic simulated environments.

Responsibilities

• Build realistic developer environments: create a virtual company with a codebase, infrastructure, and context (tickets, docs, conversations) that forms a believable development history.
• Design tasks from intermediate states of these environments: craft the prompt, define what