Check effectiveness

How a query is judged to have changed the data, and how to read the verdicts.

Every query costs site and data management time. A check whose queries are almost always answered "data correct as per source" adds work and no quality, and sites learn to ignore it. This report ranks edit checks by the share of their queries that changed nothing, and shows the fields where people keep raising the same query by hand: those are candidates for a new edit check. Run it on the query listing at an interim point, before an edit check revision, or when a study's query rate looks too high.

What to upload

• Query listing: all queries, closed and cancelled ones included. Columns are recognised by name: query ID, site, subject, status, opening and closing dates, form, field, check name, query text, answer, "raised by" (system or a person), and a data-changed flag if your EDC exports one.
• Audit trail (optional): subject, field, date, old and new value. It is the surest way to tell whether the queried value changed.

How the outcome of a query is judged

For each closed query, the first available evidence wins:

1. a data-changed column in the listing (Y/N, yes/no, да/нет);
2. the audit trail: a change of that subject's field between the opening and the closing date of the query, where the new value differs from the old one;
3. the answer text: words of change ("corrected", "updated", "entered", "исправлено", "внесено") mean a change, and win over words of confirmation; "data correct", "confirmed", "as per source", "верно", "no change" mean no change.

If none of these tell, the outcome is "not known" and counted apart. The summary shows how many queries were judged by each kind of evidence. Answer text is the weakest evidence: a site can write "confirmed" and still change the value. Add the audit trail when you can.

Grouping and verdicts

Queries are grouped by the check name column. Without it, by field and the query text with numbers and quoted values blanked out, so "Age 40 is outside 18–65" and "Age 71 is outside 18–65" fall in one group. Queries raised by a person (the "raised by" column says so, and there is no check name) are grouped by form and field.

• Noisy: 80% or more of the check's queries changed nothing or were cancelled. Retune its limits, change its wording, or drop it.
• Weak: 50% or more changed nothing. Review it.
• Useful: most of its queries led to a data change.
• Too few queries: fewer than the threshold (10 by default) to judge.

A manual form and field with at least the threshold of queries, half or more of which led to a change, is marked candidate for an edit check.

The "site with most" column shows when a check's queries come mainly from one site: then the problem may be the site's practice rather than the check.

Try it

Example files: queries.csv, audit_trail.csv. Upload the query listing alone first: the age range check comes out noisy, the blood pressure check weak, the missing AE date check useful, and the manual dose queries a candidate for a new check. Then add the audit trail: the outcome of the age queries is judged by it instead of the answer text.

Sources

• Tufts CSDD: industry studies showing that most queries do not change data.
• SCDM, Good Clinical Data Management Practices: edit check design and data validation plan.

Nothing is stored. Files are read in memory and discarded; the audit trail records counts only. Changing or dropping a check is a decision for the study team under your data validation plan; this report is the evidence for it, not the decision.