Competitive Data Scientist
About the Role
A specialized AI evaluation initiative focuses on developing challenging data science and machine learning tasks that assess the reasoning, implementation accuracy, and problem-solving capabilities of coding agents. The work emphasizes reproducibility, verified reference solutions, and rigorous evaluation methods.
This opportunity is ideal for data scientists, machine learning practitioners, quantitative analysts, and applied mathematics specialists with experience in competitive data science or research-driven modeling. Candidates should demonstrate practical programming ability, sound mathematical reasoning, and the ability to distinguish robust solutions from superficially plausible results.
The work involves building controlled computational environments, implementing benchmark solutions, evaluating coding agent performance, and improving task quality based on observed failures. Engagement is fully remote, with flexible scheduling and an expected commitment of approximately 15 hours per week.
What You'll Do
- Build reproducible computational environments with relevant datasets, dependencies, and configuration requirements.
- Implement verified reference solutions for data science and machine learning tasks.
- Develop automated tests and evaluation criteria to measure solution correctness and completeness.
- Execute coding agents against benchmark tasks and investigate their implementation failures.
- Refine task specifications and evaluation procedures based on observed agent behavior.
- Detect coding errors, flawed assumptions, data leakage, and methodological weaknesses.
- Validate outputs against expected results and document verification procedures.
- Explain solution logic, task constraints, and evaluation requirements in clear technical documentation.
Requirements
- Demonstrated experience in data science, machine learning engineering, quantitative modeling, applied mathematics, or related technical research.
- Practical experience with competitive data science platforms such as Kaggle, DrivenData, Zindi, AIcrowd, Numerai, DataSource.ai, or CodaLab/Codabench.
- Strong mathematical reasoning and problem-solving skills.
- Proficiency in programming sufficient to implement, debug, and independently verify analytical solutions.
- Ability to evaluate model outputs, identify methodological errors, and assess solution correctness.
- Understanding of reproducible experimentation, data preparation, and dependency management.
- Ability to design objective tests and translate technical requirements into verifiable acceptance criteria.
- Strong technical communication skills, including the ability to document reasoning and verification steps.
- Availability for approximately 15 hours of contract work per week.
- Ability to work independently in a fully remote environment with a flexible schedule.
- Preferred: Strong Python programming experience, including automated testing, debugging, and working with unfamiliar codebases.
- Preferred: Experience evaluating AI coding agents, developing benchmark tasks, or validating machine learning systems.
- Preferred: Familiarity with software testing frameworks, version control, and reproducible research workflows.