Freelance Agent Evaluation Engineer

Remote from Spain·Part-time·Added today

Mindrift

41 open roles

Apply

What they offer

Up to US$40 an hour
Gross, as stated in the ad.
Part-time hours
Per the ad.
Fully remote
Per the ad.

What they ask for

Have the right to work in Spain
Mindrift doesn't mention sponsorship in the ad.
Work in English
English is required, per the ad.
Have 3+ years of experience
Mid-level role.

The role

Mindrift connects specialists with project-based AI opportunities for leading tech companies, focused on testing, evaluating, and improving AI systems. Participation is project-based, not permanent employment.

We're building a dataset to evaluate AI coding agents - how well a model handles real-world developer tasks.

What you'll do

You'll create challenging tasks and evaluation criteria within realistic simulated environments:

Build realistic developer environments - a virtual company with codebase, infrastructure, and context (tickets, docs, conversations) that forms a believable development history
Design tasks from intermediate states of these environments - craft the prompt, define what "solved" means, and ensure the task is solvable by an AI agent
Write tests that verify agent solutions - accept all valid approaches and reject incorrect ones, neither too strict nor too lenient
Iterate on tasks and tests based on QA feedback - review agent solutions, analyze failures, and refine until the evaluation is fair and robust

What this is NOT

Not data labeling
Not prompt engineering
Not writing code from scratch - the agent writes most of the code; you guide and evaluate

Why this freelance opportunity might be a great fit for you?

Take part in a part-time, remote, freelance project that fits around your primary professional or academic commitments.
Work on advanced AI projects and gain valuable experience that enhances your portfolio. - Influence how future AI models understand and communicate in your field of expertise.

Requirements

What we're looking for

5+ years in software development
Core stack: Python (FastAPI), JavaScript/TypeScript (React), Docker, Postgres, Kafka, Redis
Experience writing tests (functional, integration)
English proficiency - B2+

Why this is hard:
Frontier models are already good at coding. Creating a task that genuinely challenges the best models is non-trivial. You need to deeply understand where models fail and what scenarios reveal the difference between a good and a bad solution. Tasks have many valid solutions - writing tests that accept all correct solutions and reject incorrect ones is harder than it sounds.

Languages

Please submit your CV in English and indicate your level of English proficiency.

Pay & benefits

What you'll get

Educational qualifications

A Master's Degree in Computer Science, Software Engineering, Data Science / Data Analytics, Artificial Intelligence / Machine Learning, Computational Linguistics / Natural Language Processing (NLP), Information Systems or other related fields.
Bachelor's degree is accepted if only candidate has 5 years of experience in the field.

Academic and/or Professional Experience

Candidates should have a minimum of 3 years of professional experience in related roles or domain - specifically for QA-automation/testing or cybersecurity roles

How it works
Apply → Pass qualification(s) → Join a project → Complete tasks → Get paid

Compensation:
Paid per accepted task. Your rate depends on the qualification tier you reach and how efficiently you complete tasks - up to the equivalent of $40/hr. Because payment is per task, a faster pace raises your effective hourly rate.

How they hire

2 stepsa task to do

Submitting your CV
Completing assessments or interviews
In your own time
Language assessments, skills tests or an automated AI-led interview, alone or combined, done on your own schedule.
Verifying your identity · In your own time · 1–5 days
Joining a project and onboarding
In your own time
Review project guidelines and join the project Discord, or wait in a talent pool until a project opens.

Identity verification through Persona usually takes a few days.
The wait for a project varies: active projects onboard quickly, talent-pool placements can be delayed.

About Mindrift

Mindrift (formerly Toloka AI) is a global AI data platform that connects businesses with a distributed network of freelance professionals who train, evaluate, and improve AI models. The company operates a marketplace where companies post tasks ranging from data annotation and AI evaluation to software engineering, and independent experts complete them remotely. With a workforce of millions worldwide, Mindrift is a key player in the AI data supply chain, serving clients that include major tech companies.

Mindrift has a significant presence in Spain, with numerous remote freelance roles open to professionals across the country, including positions for AI training specialists, software engineers, and data scraping experts. The company's fully remote work model is particularly attractive to international professionals, as it allows them to work from anywhere in Spain without the need for relocation. Their platform offers flexible, project-based work, making it a viable option for skilled individuals looking to supplement their income or transition to freelance work.

Industry: AI Data
Founded: 2014
Employees: 1,000–5,000
Headquarters: Amsterdam, Netherlands
In Spain: Madrid, Barcelona
Website: mindrift.ai

Good to know if you are moving

All roles are fully remote, so you can work from anywhere in Spain without needing to relocate.
The platform offers flexible, freelance-based work, ideal for professionals seeking autonomy over their schedule.
Work is project-based and paid per task, providing a direct way to monetize your skills.
The company operates globally, so you'll collaborate with a diverse, international community of experts.
As a freelance platform, there is no traditional career ladder or full-time employment contract.

All open roles at Mindrift