measure-repository/SOPs/Validation/SOP-VAL-001-Psychometric-Validation.md

# Standard Operating Procedure: Psychometric Validation of Clinical Outcome Measures

| Document ID | SOP-VAL-001 |
|-------------|---------|
| Title | Psychometric Validation of Clinical Outcome Measures |
| Revision | 1.0 |
| Effective Date | [DATE] |
| Author | [AUTHOR] |
| Approved By | [APPROVER] |
| Department | Outcomes Research |

---

## 1. Purpose

This procedure establishes requirements for conducting psychometric validation studies of clinical outcome measures to ensure they demonstrate appropriate measurement properties for their intended use.

## 2. Scope

This procedure applies to:
- New outcome measure development
- Validation of existing measures in new populations
- Adaptation of measures for new contexts or modes of administration
- All outcome measure types (PRO, ClinRO, ObsRO, PerfO)

## 3. Responsibilities

### 3.1 Principal Investigator/Measure Developer
- Design validation study protocol
- Ensure appropriate statistical expertise
- Review and interpret validation results
- Document validation evidence

### 3.2 Biostatistician
- Develop statistical analysis plan
- Conduct psychometric analyses
- Generate validation reports
- Advise on sample size and methodology

### 3.3 Quality Manager
- Review validation protocols for regulatory compliance
- Maintain validation documentation
- Track validation status of all measures

## 4. Definitions

| Term | Definition |
|------|------------|
| Reliability | The degree to which a measure is free from measurement error |
| Internal Consistency | The extent to which items within a scale measure the same construct (Cronbach's alpha) |
| Test-Retest Reliability | Consistency of scores when measure is administered to the same individuals at different times |
| Inter-Rater Reliability | Agreement between different raters/observers (for ClinRO, ObsRO) |
| Validity | The degree to which a measure assesses what it purports to measure |
| Content Validity | Evidence that measure items represent all aspects of the construct |
| Construct Validity | Evidence that measure relates to other measures as theoretically expected |
| Criterion Validity | Agreement between measure and a gold standard |
| Responsiveness | Ability to detect meaningful change over time |
| MCID | Minimal Clinically Important Difference - smallest change considered important |
| Floor/Ceiling Effects | Clustering of scores at bottom or top of scale, limiting ability to detect change |

## 5. Procedure

### 5.1 Validation Study Planning

5.1.1. Define validation objectives:
   - Target population
   - Intended use and context
   - Mode of administration (paper, electronic, interview)
   - Key measurement properties to evaluate

5.1.2. Develop validation protocol including:
   - Background and rationale
   - Study design and timeline
   - Participant eligibility criteria
   - Sample size justification
   - Data collection procedures
   - Statistical analysis plan
   - Success criteria for validation

5.1.3. Select comparison measures:
   - Established measures of same construct (convergent validity)
   - Measures of different constructs (discriminant validity)
   - Clinical indicators or gold standards (criterion validity)

5.1.4. Obtain necessary regulatory approvals (IRB, informed consent)

5.1.5. Document validation plan in Form FRM-VAL-001

### 5.2 Reliability Assessment

#### 5.2.1 Internal Consistency Reliability

5.2.1.1. Analyze baseline data from main study sample

5.2.1.2. Calculate Cronbach's alpha for each scale/subscale

5.2.1.3. Acceptance criteria:
   - Alpha ≥ 0.70 for group comparisons
   - Alpha ≥ 0.90 for individual decision-making
   - Alpha < 0.95 (if higher, may indicate item redundancy)

5.2.1.4. Examine item-total correlations (typically ≥ 0.30)

5.2.1.5. Assess scale dimensionality using factor analysis

#### 5.2.2 Test-Retest Reliability

5.2.2.1. Administer measure twice to stable subsample

5.2.2.2. Time interval: typically 2-14 days
   - Short enough that true change is unlikely
   - Long enough to prevent memory effects

5.2.2.3. Calculate intraclass correlation coefficient (ICC)

5.2.2.4. Acceptance criteria:
   - ICC ≥ 0.70 for group comparisons
   - ICC ≥ 0.90 for individual decision-making

5.2.2.5. Calculate standard error of measurement (SEM)

5.2.2.6. Generate Bland-Altman plots to assess agreement

#### 5.2.3 Inter-Rater Reliability (for ClinRO, ObsRO)

5.2.3.1. Have multiple raters assess same participants

5.2.3.2. Calculate ICC or weighted kappa as appropriate

5.2.3.3. Acceptance criteria:
   - ICC or kappa ≥ 0.70

5.2.3.4. Identify sources of disagreement for training improvement

### 5.3 Validity Assessment

#### 5.3.1 Content Validity

5.3.1.1. Conduct qualitative research with target population:
   - Concept elicitation interviews
   - Cognitive debriefing of items
   - Assessment of comprehensibility and relevance

5.3.1.2. Obtain expert panel review:
   - Clinical experts
   - Psychometricians
   - Patient representatives

5.3.1.3. Document evidence in content validity report

5.3.1.4. For FDA submissions, follow FDA PRO Guidance requirements

#### 5.3.2 Construct Validity

5.3.2.1. Convergent validity:
   - Correlate with established measures of same construct
   - Expected correlation: typically r ≥ 0.50-0.70

5.3.2.2. Discriminant validity:
   - Correlate with measures of different constructs
   - Expected correlation: typically r < 0.30

5.3.2.3. Known-groups validity:
   - Compare scores across groups expected to differ
   - Use appropriate statistical tests (t-test, ANOVA)
   - Calculate effect sizes (Cohen's d, eta-squared)

5.3.2.4. Factorial validity:
   - Conduct confirmatory factor analysis (CFA)
   - Assess model fit (CFI > 0.90, RMSEA < 0.08, SRMR < 0.08)

#### 5.3.3 Criterion Validity

5.3.3.1. If gold standard exists, calculate:
   - Sensitivity and specificity
   - Positive and negative predictive values
   - ROC curves and AUC

### 5.4 Responsiveness Assessment

5.4.1. Collect data at baseline and follow-up from participants expected to change

5.4.2. Calculate change scores

5.4.3. Assess responsiveness using:
   - Effect sizes (Cohen's d, standardized response mean)
   - Correlation with external indicators of change
   - Receiver operating characteristic (ROC) analysis

5.4.4. Determine Minimal Clinically Important Difference (MCID):
   - Anchor-based methods (correlation with patient global ratings)
   - Distribution-based methods (0.5 SD, 1 SEM)
   - Multiple methods recommended

### 5.5 Interpretability Assessment

5.5.1. Assess score distribution:
   - Floor effects: >15% scoring at minimum
   - Ceiling effects: >15% scoring at maximum
   - Skewness and kurtosis

5.5.2. Develop score interpretation guidelines:
   - Clinical cutoff scores
   - Severity categories
   - Normative data (if appropriate)

5.5.3. Document MCID and other interpretability anchors

### 5.6 Validation Report

5.6.1. Prepare comprehensive validation report including:
   - Study objectives and methods
   - Participant characteristics
   - All psychometric analyses results
   - Tables and figures
   - Discussion of strengths and limitations
   - Conclusions and recommendations for use

5.6.2. File validation report as Form FRM-VAL-002

5.6.3. Update measure status in Validation Tracking Database

5.6.4. For regulatory submissions, prepare according to FDA guidance

### 5.7 Ongoing Validation Activities

5.7.1. Plan for continued evidence generation:
   - Validation in additional populations
   - Assessment in different contexts or settings
   - Cross-cultural validation
   - Longitudinal measurement invariance

5.7.2. Monitor published validation evidence for measures in use

5.7.3. Review and update validation status annually

## 6. Related Documents

- FRM-VAL-001: Validation Study Protocol Template
- FRM-VAL-002: Psychometric Validation Report Template
- FRM-VAL-003: Validation Tracking Database
- SOP-DM-001: Data Management for Validation Studies
- SOP-LIC-001: License Management

## 7. References

- FDA (2009). Guidance for Industry: Patient-Reported Outcome Measures: Use in Medical Product Development to Support Labeling Claims
- Mokkink LB, et al. (2010). The COSMIN checklist for assessing the methodological quality of studies on measurement properties of health status measurement instruments. Quality of Life Research, 19(4), 539-549
- Reeve BB, et al. (2013). ISOQOL recommends minimum standards for patient-reported outcome measures used in patient-centered outcomes and comparative effectiveness research. Quality of Life Research, 22(8), 1889-1905
- Streiner DL, Norman GR, Cairney J (2015). Health Measurement Scales: A Practical Guide to Their Development and Use (5th ed.). Oxford University Press
- DeVellis RF (2017). Scale Development: Theory and Applications (4th ed.). SAGE Publications

---

## Revision History

| Rev | Date | Description | Author |
|-----|------|-------------|--------|
| 1.0 | [DATE] | Initial release | [AUTHOR] |