The Psychometrician’s Handbook / Resources

Resources

Take it apart

The dataset behind every figure in Volume 1, the code that produces them, the glossary and the full reference list. Free to use, with attribution.

Dataset and code

One synthetic dataset, every figure

Volume 1 runs on a single simulated clinical outcome assessment dataset, adcorr: 200 subjects, 18 parameters, 10,400 rows in long CDISC ADaM BDS. Every figure and every number in the volume comes out of it. The data are synthetic and no patient contributed to them.

Everything, as a zip
adcorr.csv
The dataset itself 1.2 MB README_data_dictionary.md
Data dictionary, and what each parameter was built to show 6 KB 01_simulate_coa_data.R
Generates the dataset, and every code block in the volume 19 KB 02_reproduce_figures.R
Reproduces the figures, written to be taken apart and reused 19 KB

R 4.x. The simulation script installs whatever it needs. With set.seed(2026) you get byte-identical data, so the numbers in the volume are the numbers you will get.

The data are long, so pivot first

# one row per subject, per parameter, per visit
adcorr <- read.csv("adcorr.csv")

wide <- adcorr |>
  filter(AVISIT == "Baseline") |>
  pivot_wider(id_cols = USUBJID,
              names_from = PARAMCD,
              values_from = AVAL)

cor(wide$TARGTOT, wide$RELATEDA)   # 0.73

The volume itself

Volume 1: Correlations

volume-1-correlations.pdf3.3 MB volume-1-correlations.pptx3.9 MB

Glossary

18 terms from Volume 1

Lifted from the volume appendix, so the wording here and in the deck stay the same.

Between-subject correlation

Association between individuals’ average levels of two variables. Reflects stable differences between people.

Within-subject correlation

Association between deviations from each person’s own mean across occasions. Reflects co-variation for an individual over time.

Total correlation

A standard correlation across all observations, ignoring nesting. Conflates both between subjects and within subject correlations; not interpretable alone.

Nesting

A structure in which observations belong to a higher unit (timepoints within a person). Violates the independence assumption.

Person-mean centering

Subtracting each person’s mean from their observations to separate within- from between-subject variability.

Intraclass correlation (ICC)

Proportion of total variance from stable between-subject differences; here, the degree of clustering within people.

Ecological fallacy

Inferring individual-level dynamics from group-level associations (Robinson, 1950; Curran & Bauer, 2011).

Simpson’s Paradox

Statistical phenomenon in which a trend observed within a patient, reverses or disappears when between-patients trend is analyzed.

Dependence

Two observations are dependent when knowing one changes what you'd expect for the other. Two visits from the same patient, unlike two different patients.

Continuous variable

A variable that can take any infinite number of real values within a range.

Ordinal variable

A variable with categories that have a logical order but the exact distance between the categories is unknown.

Binary variable

A variable with two categories only.

Corrected (item-total correlation)

« Corrected » means the item is excluded from the total score it's compared against, so it isn't partly correlated with itself.

Clinician Global Impression of Severity (CGIS)

A single item questionnaire asking the clinician to rate the severity of a patient's condition, usually at the time of assessment.

Clinician Global Impression of Change (CGIC)

A single item questionnaire asking the clinician to rate the improvement or worsening in a patient's condition since the start of a study.

Patient Global Impression of Severity (PGIS)

A single item questionnaire asking the patient about their condition, usually referenced over the last week.

Patient Global Impression of Change (PGIC)

A single item questionnaire asking the patient to rate the improvement or worsening in their condition since the start of a study.

Lord’s Paradox

The longitudinal analogue of Simpson's Paradox: adjusting for baseline can flip the direction of a change effect.

References

54 sources cited in Volume 1

Take the whole list into your reference manager. RIS imports into Zotero, EndNote, Mendeley and Papers; BibTeX suits LaTeX and Overleaf; the CSV is for everything else.

RIS · Zotero, EndNote12 KB BibTeX · LaTeX11 KB CSV · spreadsheet14 KB

These are parsed from the volume appendix, which is typed prose rather than a database, so check the fields before citing from them. Every record keeps the original line in full, so nothing is lost if a field is split wrongly.

  • Anscombe, F. J. (1973). Graphs in statistical analysis. Am Stat, 27(1), 17-21.
  • Bakdash, J. Z., & Marusich, L. R. (2017). Repeated measures correlation. Front Psychol, 8, 456.
  • Barnett, A. G., van der Pols, J. C., & Dobson, A. J. (2005). Regression to the mean: what it is and how to deal with it. Int J Epidemiol, 34(1), 215-220
  • Barnett, A. G., van der Pols, J. C., & Dobson, A. J. (2005). Regression to the mean: what it is and how to deal with it. International Journal of Epidemiology, 34(1), 215-220.
  • Bates, D., et al. (2015). Fitting linear mixed-effects models using lme4. J Stat Softw, 67(1), 1-48
  • Bland, J. M., & Altman, D. G. (1986). Statistical methods for assessing agreement between two methods of clinical measurement.
  • Bland, J. M., & Altman, D. G. (1995). Calculating correlation coefficients with repeated observations.
  • Borsboom, D., Mellenbergh, G. J., & van Heerden, J. (2003). The theoretical status of latent variables. Psychological Review, 110(2), 203-219.
  • Campbell, D. T., & Fiske, D. W. (1959). Convergent and discriminant validation by the multitrait–multimethod matrix. Psychological Bulletin, 56(2), 81-105.
  • Cappelleri, J. C., et al. (2013). Patient-Reported Outcomes: Measurement, Implementation and Interpretation. CRC Press
  • Chen, W.-H., & Thissen, D. (1997). Local dependence indexes for item pairs using item response theory. J Educ Behav Stat, 22(3), 265-289.
  • Clark, L. A., & Watson, D. (1995). Constructing validity: Basic issues in objective scale development. Psychological Assessment, 7(3), 309–319
  • Clifton, L., & Clifton, D. A. (2019). The correlation between baseline score and post-intervention score, and its implications for statistical analysis.
  • Cohen, J. (1988). Statistical Power Analysis for the Behavioral Sciences (2nd ed.). Erlbaum
  • Cohen, J., Cohen, P., West, S. G., & Aiken, L. S. (2003). Applied Multiple Regression / Correlation Analysis for the Behavioral Sciences (3rd ed.). Erlbaum
  • Coon, C. D., & Cappelleri, J. C. (2016). Interpreting change in scores on PRO instruments. Ther Innov Regul Sci, 50(1), 22-29
  • Cronbach, L. J., & Furby, L. (1970). How we should measure “change”: Or should we? Psychological Bulletin, 74(1), 68-80
  • Curran, P. J., & Bauer, D. J. (2011). The disaggregation of within-person and between-person effects in longitudinal models of change. Annu Rev Psychol, 62, 583-619.
  • Devji, T., et al. (2020). Credibility of anchor-based estimates of minimal important differences.
  • EMA (2016). Appendix 2, anticancer medicinal products: PRO measures in oncology.
  • Fayers, P. M., & Machin, D. (2015).Quality of life: the assessment, analysis and reporting of patient-reported outcomes; John Wiley & Sons
  • FDA (2009). PRO Measures: Use in Medical Product Development to Support Labeling Claims.
  • FDA (2025). Patient-Focused Drug Development: Fit-for-Purpose COAs
  • Fisher, R. A. (1921). On the probable error of a coefficient of correlation deduced from a small sample.
  • Griffiths, P., Sims, et al., (2023). How strong should my anchor be for estimating group and individual level meaningful change? A simulation study assessing anchor correlation strength and the impact of sample size, distribution of change scores and methodology on establishing a true meaningful change threshold. Quality of Life Research, 32(5), 1255-1264. (simulation).
  • Guyatt G. H. (2000). Making sense of quality-of-life data. Medical care, 38(9 Suppl), II175–II179.
  • Hays, R. D., Farivar, S. S., & Liu, H. (2005). Approaches and recommendations for estimating minimally important differences for health-related quality of life measures.
  • Hinkle, D. E., Wiersma, W., & Jurs, S. G. (2003). Applied Statistics for the Behavioral Sciences (5th ed.)
  • Koo, T. K., & Li, M. Y. (2016). A Guideline of Selecting and Reporting Intraclass Correlation Coefficients for Reliability Research. Journal of chiropractic medicine, 15(2), 155–163
  • Little, R. J. A., & Rubin, D. B. (2019). Statistical Analysis with Missing Data (3rd ed.). Wiley.
  • Lord, F. M. (1967). A paradox in the interpretation of group comparisons. Psychological Bulletin, 68(5), 304-305.
  • Lord, F. M., & Novick, M. R. (1968). Statistical Theories of Mental Test Scores.
  • McGraw, K. O., & Wong, S. P. (1996). Forming inferences about some intraclass correlation coefficients. Psychol Methods, 1(1), 30-46.
  • Mukaka, M. M. (2012). A guide to appropriate use of correlation coefficient in medical research. Malawi Med J, 24(3), 69-71.
  • Nunnally, J. C., & Bernstein, I. H. (1994). Psychometric Theory (3rd ed.). McGraw-Hill (correction for attenuation).
  • Nunnally, J. C., & Bernstein, I. H. (1994). Psychometric Theory (3rd ed.). McGraw-Hill.
  • Olsson, U. (1979). Maximum likelihood estimation of the polychoric correlation coefficient.
  • Ozer, D. J. (1985). Correlation and the coefficient of determination. Psychological Bulletin, 97(2), 307-315.
  • Piedmont, R.L. (2014). Inter-item Correlations. In: Michalos, A.C. (eds) Encyclopedia of Quality of Life and Well-Being Research.
  • Pinheiro, J. C., & Bates, D. M. (2000). Mixed-Effects Models in S and S-PLUS. Springer.
  • Prinsen, C. A. C., et al. (2018). COSMIN guideline for systematic reviews of PROMs. Qual Life Res, 27, 1147-1157
  • Revelle, W. (2024). psych: Procedures for Psychological, Psychometric, and Personality Research. R package (polychoric, ICC).
  • Revicki, D., et al. (2008). Determining responsiveness and minimally important differences for PROs. J Clin Epidemiol, 61(2), 102-109
  • Robinson, W. S. (1950). Ecological correlations and the behavior of individuals. Am Sociol Rev, 15, 351-357
  • Sackett, P. R., & Yang, H. (2000). Correction for range restriction: An expanded typology. J Appl Psychol, 85(1), 112-118.
  • Schober, P., Boer, C., & Schwarte, L. A. (2018). Correlation coefficients: appropriate use and interpretation. Anesth Analg, 126(5), 1763-1768
  • Schönbrodt, F. D., & Perugini, M. (2013). At what sample size do correlations stabilize? J Res Pers, 47(5), 609-612.
  • Shrout, P. E., & Fleiss, J. L. (1979). Intraclass correlations: uses in assessing rater reliability. Psychol Bull, 86(2), 420-428
  • Simpson, E. H. (1951). The interpretation of interaction in contingency tables. J R Stat Soc B, 13(2), 238-241.
  • Simpson, E. H. (1951). The interpretation of interaction in contingency tables. Journal of the Royal Statistical Society B, 13(2), 238-241
  • Spearman, C. (1904). The proof and measurement of association between two things. Am J Psychol, 15, 72-101.
  • Sprangers, M. A. G., & Schwartz, C. E. (1999) Integrating response shift into health-related quality of life research. Soc Sci Med, 48(11), 1507-1515.
  • Streiner, D. L., Norman, G. R., & Cairney, J. (2024). Health Measurement Scales: A practical guide to their development and use (6th ed.). Oxford University Press.
  • Stull, D. E., Leidy, N. K., Parasuraman, B., & Chassany, O. (2009). Optimal recall periods for patient-reported outcomes: challenges and potential solutions. Curr Med Res Opin, 25(4), 929-942.