AI, Machine Learning & Deep Learning Research

Designing Rigorous ML Experiments (Leakage-Aware)

Designs an experimental protocol for machine learning or deep learning: data splitting, baselines, metrics, tuning, repetition and statistical tests, with a systematic check for data leakage.

Expert Full project prompt Multi-step prompt Not rated yet

Purpose of the prompt

Most exaggerated results in machine learning papers come from data leakage or unfair comparisons. The prompt builds a protocol that tackles this explicitly and can go into the methodology chapter.

When to use it?

Before training any model in a research project, and when reviewing the experimental methodology of a paper.

Full prompt text

Text in square brackets [ ] is a variable you replace with your research details, or fill it in from “Customise the prompt” below.

You are a senior machine learning researcher and a reviewer for conferences in the field, specialising in [Specific field].

Task: [Task]
Data: [Data description]
Proposed models: [Proposed models]
Computing resources: [Computing resources]

Design the experimental protocol:
1. Data splitting: the method (at patient, user, site or time level) and why, the proportions, and an external test set if possible.
2. Data leakage check: a specific list for this data (duplicate samples, preprocessing before splitting, selection on the test set, future information).
3. Baselines: simple, strong and recent, ensuring a fair comparison (the same tuning budget).
4. Metrics: primary and secondary metrics suited to class imbalance, and calibration metrics if the application is medical.
5. Hyperparameter tuning: the method, search space and budget, and on which set.
6. Repetition and uncertainty: number of seeds, confidence intervals (bootstrap), and the statistical test for comparing models.
7. Error analysis and subgroup analysis (fairness).
8. A reproducibility checklist: versions, seeds, settings and hardware.
9. The empty results table that I will fill in.

Academic integrity rules (mandatory):
- Do not invent references, author names, DOIs, links, experimental results or statistics. Where a source is needed, write [source needed].
- Do not claim to have read a study or file whose text I have not given you.
- Clearly separate what comes from the material I provided from your own conclusions and suggestions.
- If you are missing essential information, ask me for it before continuing instead of assuming it.
- Do not predict performance figures.

Write the answer in [Answer language].

Customise the prompt

Fill in only what suits you. The line of an empty optional field is removed from the final text, and empty required fields stay in brackets for you to complete later.

Preview of the customised prompt

You are a senior machine learning researcher and a reviewer for conferences in the field, specialising in [Specific field].

Task: [Task]
Data: [Data description]
Proposed models: [Proposed models]
Computing resources: [Computing resources]

Design the experimental protocol:
1. Data splitting: the method (at patient, user, site or time level) and why, the proportions, and an external test set if possible.
2. Data leakage check: a specific list for this data (duplicate samples, preprocessing before splitting, selection on the test set, future information).
3. Baselines: simple, strong and recent, ensuring a fair comparison (the same tuning budget).
4. Metrics: primary and secondary metrics suited to class imbalance, and calibration metrics if the application is medical.
5. Hyperparameter tuning: the method, search space and budget, and on which set.
6. Repetition and uncertainty: number of seeds, confidence intervals (bootstrap), and the statistical test for comparing models.
7. Error analysis and subgroup analysis (fairness).
8. A reproducibility checklist: versions, seeds, settings and hardware.
9. The empty results table that I will fill in.

Academic integrity rules (mandatory):
- Do not invent references, author names, DOIs, links, experimental results or statistics. Where a source is needed, write [source needed].
- Do not claim to have read a study or file whose text I have not given you.
- Clearly separate what comes from the material I provided from your own conclusions and suggestions.
- If you are missing essential information, ask me for it before continuing instead of assuming it.
- Do not predict performance figures.

Write the answer in [Answer language].

Variables you can customise

[Specific field]
Required
[Task]
Required
[Data description]
Required
[Proposed models]
Optional
[Computing resources]
Optional
[Answer language]
Optional, choose from: Arabic, English, Arabic with English terms in parentheses, default value: Arabic with English terms in parentheses

Usage example

Example input

Skin cancer classification from skin images, 10,000 images from 3,000 patients, 7 imbalanced classes.

Expected output

A nine-section experimental protocol with a tailored leakage checklist and an empty results table.

How to use it

  • Put the protocol in the methodology chapter.
  • Use it later to write the evaluation report.

Tips for better results

  • State whether each patient or user has several samples; this is the most common source of leakage.

Limitations and warnings

  • The protocol is general; check the reporting guidelines in your field (such as TRIPOD+AI or CLAIM for medical applications).

Suggested tools for using this prompt

Tools from the directory of research and academic publishing tools that are actually linked to this prompt's task, with the reason each tool is suggested.

Google Colab

Tool: Primary Free with usage limits

A hosted Jupyter Notebook service from Google that runs in the browser with no setup and offers computing resources, including GPUs.

Research function: Writing and running Python code in notebooks saved to Google Drive.

Why it is suggested: To run experiments on cloud GPUs.

Semantic Scholar

Tool: Supporting Free

A free, AI-powered academic search engine from the Allen Institute for AI that indexes more than 200 million papers.

Research function: Searching for and discovering scientific papers, with automatic summaries and highlights that help you grasp a paper quickly.

Why it is suggested: To find recent baselines for the same task.

ChatGPT

Tool: Supporting Free with usage limits

A general AI assistant from OpenAI used for brainstorming, drafting and summarising.

Research function: General text chat that helps you generate and organise ideas and improve your wording.

Why it is suggested: A general language model where you can paste the prompt directly and continue the conversation to improve the result.

Claude

Tool: Supporting Free with usage limits

A general AI assistant from Anthropic used for analysis, drafting, summarising and working with documents.

Research function: General text chat that helps you analyse texts, organise ideas and improve your wording.

Why it is suggested: A general language model where you can paste the prompt directly and continue the conversation to improve the result.

KNIME

Tool: Optional Free

A free, open-source analytics platform for building data processing and analysis workflows visually.

Research function: Building visual workflows for data preparation, analysis and machine learning, with optional coding.

Why it is suggested: To build visual processing pipelines for simple experiments.

In research workflows

Ratings

Not rated yet
Rate this prompt

One rating per visitor per prompt; submitting again updates your earlier rating. Comments appear after review.

Similar prompts