Table 1 Reconsidering the psychometrics of QOL assessment in light of response shift and appraisal

RELIABILITY

Internal consistency

High homogeneity. Items on a scale are chosen to demonstrate high inter-correlation.

Item content may be too narrow; fail to capture important aspects of QOL. QOL items that are necessarily general and unspecified remain difficult to interpret.

Determine what frames of reference and sampling strategies are systematically induced by specific items and measurement approaches.

Tune items and instructions to achieve desired appraisal parameters. Adjust analyses for differences in appraisal.

Inter-observer agreement

Convergence in QOL ratings made by two or more observers (i.e., self, family, provider).

Differences in perspective so pervasive that this is often ignored or not considered to be a psychometric issue.

Direct measurement of all appraisal parameters to determine whether they explain differences in perspectives.

Ask raters to assume criteria for appraisal used by other observers to calibrate agreement in ratings.

Test-retest stability

High stability over short periods of time. Low stability indicative of measurement error.

QOL is contingent upon appraisal, so low stability may represent change in the appraisal process rather than error of measurement.

True test-retest reliability requires individuals to use a consistent frame of reference, to consider the same types of experiences, to maintain their standards for evaluating these experiences, and to prioritize experiences in the same way.

Impose and test strict assumptions about similarity of appraisal parameters. Establish test-retest stability over a timeframe in which changes in appraisal would not be expected.

VALIDITY

Construct validity

High correlation with some other QOL measures.

Equivalent to internal consistency reliability. Content may be too narrow. There is no a priori theory to dictate how such measures ought to relate to one another.

Makes it possible to frame a priori questions about how measures ought to behave, and to develop stronger and more consistent evidence about construct validity.

QOL Contingent True Score: 'Internal construct validity' examines whether a QOL measure elicits the desired process of appraisal from respondents. 'External construct validity' involves the relationship of QOL with objective criteria or other QOL measures, in light of established appraisal parameters.

Responsiveness

QOL changes in conjunction with health state changes.

Movement up or down on a QOL scale tells us little about processes underlying that observed change.

Observed overt scores are always constructed by individuals based upon the recall and appraisal of relevant experience.

Must be able to distinguish patients who are feeling better from those who have changed their mind about what it means to feel bad.