Intraclass correlation

In statistics, the intraclass correlation, or the

intraclass correlation coefficient (ICC),[1] is a
descriptive statistic that can be used when
quantitative measurements are made on units that
are organized into groups. It describes how
strongly units in the same group resemble each
other. While it is viewed as a type of correlation,
unlike most other correlation measures it operates
on data structured as groups, rather than data
structured as paired observations.

The intraclass correlation is commonly used to

quantify the degree to which individuals with a
fixed degree of relatedness (e.g. full siblings)
A dot plot showing a dataset with high intraclass
resemble each other in terms of a quantitative trait
correlation. Values from the same group tend to be
(see heritability). Another prominent application is
similar.
the assessment of consistency or reproducibility of
quantitative measurements made by different
observers measuring the same quantity.

Early ICC definition: unbiased

but complex formula
The earliest work on intraclass correlations
focused on the case of paired measurements, and
the first intraclass correlation (ICC) statistics to be
proposed were modifications of the interclass
correlation (Pearson correlation).

Consider a data set consisting of N paired data

values (xn,1 , xn,2 ), for n = 1, ..., N. The intraclass A dot plot showing a dataset with low intraclass
correlation r originally proposed[2] by Ronald correlation. There is no tendency for values from the
Fisher[3] is same group to be similar.

where
Later versions of this statistic [3] used the degrees of freedom 2N −1 in the denominator for calculating s2
and N −1 in the denominator for calculating r, so that s2 becomes unbiased, and r becomes unbiased if s is
known.

The key difference between this ICC and the interclass (Pearson) correlation is that the data are pooled to
estimate the mean and variance. The reason for this is that in the setting where an intraclass correlation is
desired, the pairs are considered to be unordered. For example, if we are studying the resemblance of twins,
there is usually no meaningful way to order the values for the two individuals within a twin pair. Like the
interclass correlation, the intraclass correlation for paired data will be confined to the interval [−1, +1].

The intraclass correlation is also defined for data sets with groups having more than 2 values. For groups
consisting of three values, it is defined as[3]

where

As the number of items per group grows, so does the number of cross-product terms in this expression
grows. The following equivalent form is simpler to calculate:

where K is the number of data values per group, and is the sample mean of the nth group.[3] This form
is usually attributed to Harris.[4] The left term is non-negative; consequently the intraclass correlation must
satisfy

For large K, this ICC is nearly equal to

which can be interpreted as the fraction of the total variance that is due to variation between groups. Ronald
Fisher devotes an entire chapter to intraclass correlation in his classic book Statistical Methods for Research
Workers.[3]
For data from a population that is completely noise, Fisher's formula produces ICC values that are
distributed about 0, i.e. sometimes being negative. This is because Fisher designed the formula to be
unbiased, and therefore its estimates are sometimes overestimates and sometimes underestimates. For small
or 0 underlying values in the population, the ICC calculated from a sample may be negative.

Modern ICC definitions: simpler formula but positive bias

Beginning with Ronald Fisher, the intraclass correlation has been regarded within the framework of
analysis of variance (ANOVA), and more recently in the framework of random effects models. A number
of ICC estimators have been proposed. Most of the estimators can be defined in terms of the random effects
model

where Yij is the ith observation in the jth group, μ is an unobserved overall mean, αj is an unobserved
random effect shared by all values in group j, and εij is an unobserved noise term.[5] For the model to be
identified, the αj and εij are assumed to have expected value zero and to be uncorrelated with each other.
Also, the αj are assumed to be identically distributed, and the εij are assumed to be identically distributed.
2 2
The variance of αj is denoted σα and the variance of εij is denoted σε .

The population ICC in this framework is[6]

With this framework, the ICC is the correlation of two observations from the same group.

[Proof]

For a one way random effects model:

, , s and s independent
and s are independent from s.

The variance of any observation is:

The covariance of two observations from the same group i (for
) is:[7]

In this, we've used properties of the covariance.

Put together we get:

An advantage of this ANOVA framework is that different groups can have different numbers of data
values, which is difficult to handle using the earlier ICC statistics. This ICC is always non-negative,
allowing it to be interpreted as the proportion of total variance that is "between groups." This ICC can be
generalized to allow for covariate effects, in which case the ICC is interpreted as capturing the within-class
similarity of the covariate-adjusted data values.[8]

This expression can never be negative (unlike Fisher's original formula) and therefore, in samples from a
population which has an ICC of 0, the ICCs in the samples will be higher than the ICC of the population.

A number of different ICC statistics have been proposed, not all of which estimate the same population
parameter. There has been considerable debate about which ICC statistics are appropriate for a given use,
since they may produce markedly different results for the same data.[9][10]

Relationship to Pearson's correlation coefficient

In terms of its algebraic form, Fisher's original ICC is the ICC that most resembles the Pearson correlation
coefficient. One key difference between the two statistics is that in the ICC, the data are centered and scaled
using a pooled mean and standard deviation, whereas in the Pearson correlation, each variable is centered
and scaled by its own mean and standard deviation. This pooled scaling for the ICC makes sense because
all measurements are of the same quantity (albeit on units in different groups). For example, in a paired data
set where each "pair" is a single measurement made for each of two units (e.g., weighing each twin in a
pair of identical twins) rather than two different measurements for a single unit (e.g., measuring height and
weight for each individual), the ICC is a more natural measure of association than Pearson's correlation.

An important property of the Pearson correlation is that it is invariant to application of separate linear
transformations to the two variables being compared. Thus, if we are correlating X and Y, where, say,
Y = 2X + 1, the Pearson correlation between X and Y is 1 — a perfect correlation. This property does not
make sense for the ICC, since there is no basis for deciding which transformation is applied to each value in
a group. However, if all the data in all groups are subjected to the same linear transformation, the ICC does
not change.

Use in assessing conformity among observers

The ICC is used to assess the consistency, or conformity, of measurements made by multiple observers
measuring the same quantity.[11] For example, if several physicians are asked to score the results of a CT
scan for signs of cancer progression, we can ask how consistent the scores are to each other. If the truth is
known (for example, if the CT scans were on patients who subsequently underwent exploratory surgery),
then the focus would generally be on how well the physicians' scores matched the truth. If the truth is not
known, we can only consider the similarity among the scores. An important aspect of this problem is that
there is both inter-observer and intra-observer variability. Inter-observer variability refers to systematic
differences among the observers — for example, one physician may consistently score patients at a higher
risk level than other physicians. Intra-observer variability refers to deviations of a particular observer's score
on a particular patient that are not part of a systematic difference.
The ICC is constructed to be applied to exchangeable measurements — that is, grouped data in which there
is no meaningful way to order the measurements within a group. In assessing conformity among observers,
if the same observers rate each element being studied, then systematic differences among observers are
likely to exist, which conflicts with the notion of exchangeability. If the ICC is used in a situation where
systematic differences exist, the result is a composite measure of intra-observer and inter-observer
variability. One situation where exchangeability might reasonably be presumed to hold would be where a
specimen to be scored, say a blood specimen, is divided into multiple aliquots, and the aliquots are
measured separately on the same instrument. In this case, exchangeability would hold as long as no effect
due to the sequence of running the samples was present.

Since the intraclass correlation coefficient gives a composite of intra-observer and inter-observer variability,
its results are sometimes considered difficult to interpret when the observers are not exchangeable.
Alternative measures such as Cohen's kappa statistic, the Fleiss kappa, and the concordance correlation
coefficient[12] have been proposed as more suitable measures of agreement among non-exchangeable
observers.

Calculation in software packages

ICC is supported in the open source software package R (using the function "icc" with the packages psy (ht
tps://cran.r-project.org/web/packages/psy/index.html) or irr (https://cran.r-project.org/web/packages/irr/inde
x.html), or via the function "ICC" in the package psych (http://www.personality-project.org/r/html/ICC.htm
l).) The rptR (https://cran.r-project.org/web/packages/rptR/index.html) package [13] provides methods for
the estimation of ICC and repeatabilities for Gaussian, binomial and Poisson distributed data in a mixed-
model framework. Notably, the package allows estimation of adjusted ICC (i.e. controlling for other
variables) and computes confidence intervals based on parametric bootstrapping and significances based on
the permutation of residuals. Commercial software also supports ICC, for instance Stata or SPSS[14]
Different intraclass correlation coefficient definitions applied to three
scenarios of inter-observer concordance.
Different types of ICC [3] (http://www.nyu.edu/its/statistics/Docs/intracls.html) Archived (https://web.archive.org/we
b/20090303152247/http://www.nyu.edu/its/statistics/Docs/intracls.html) 2009-03-03 at the Wayback Machine

Shrout and Fleiss McGraw and Wong convention

convention [15] Name in SPSS and Stata [16][17]

One-way random, single score

ICC(1,1) One-way random, single measures
ICC(1)

Two-way random, single score Two-way random, single measures, absolute

ICC(2,1)
ICC(A,1) agreement

Two-way mixed, single score

ICC(3,1) Two-way mixed, single measures, consistency
ICC(C,1)
Two-way random, single score Two-way random, single measures,
undefined
ICC(C,1) consistency

Two-way mixed, single score Two-way mixed, single measures, absolute

undefined
ICC(A,1) agreement

One-way random, average score

ICC(1,k) One-way random, average measures
ICC(k)
Two-way random, average score Two-way random, average measures, absolute
ICC(2,k)
ICC(A,k) agreement

Two-way mixed, average score Two-way mixed, average measures,

ICC(3,k)
ICC(C,k) consistency

Two-way random, average score Two-way random, average measures,

undefined
ICC(C,k) consistency
Two-way mixed, average score Two-way mixed, average measures, absolute
undefined
ICC(A,k) agreement

The three models are:

One-way random effects: each subject is measured by a different set of k randomly selected
raters;
Two-way random: k raters are randomly selected, then, each subject is measured by the
same set of k raters;
Two-way mixed: k fixed raters are defined. Each subject is measured by the k raters.

Number of measurements:

Single measures: even though more than one measure is taken in the experiment, reliability
is applied to a context where a single measure of a single rater will be performed;
Average measures: the reliability is applied to a context where measures of k raters will be
averaged for each subject.

Consistency or absolute agreement:

Absolute agreement: the agreement between two raters is of interest, including systematic
errors of both raters and random residual errors;
Consistency: in the context of repeated measurements by the same rater, systematic errors
of the rater are canceled and only the random residual error is kept.

The consistency ICC cannot be estimated in the one-way random effects model, as there is no way to
separate the inter-rater and residual variances.
An overview and re-analysis of the three models for the single measures ICC, with an alternative recipe for
their use, has also been presented by Liljequist et al. (2019).[18]

Interpretation
Cicchetti (1994)[19] gives the following often quoted guidelines for interpretation for kappa or ICC inter-
rater agreement measures:

Less than 0.40—poor.

Between 0.40 and 0.59—fair.
Between 0.60 and 0.74—good.
Between 0.75 and 1.00—excellent.

A different guideline is given by Koo and Li (2016):[20]

below 0.50: poor

between 0.50 and 0.75: moderate
between 0.75 and 0.90: good
above 0.90: excellent

See also
Correlation ratio
Design effect
Effect size#Eta-squared (η2)

References
1. Koch GG (1982). "Intraclass correlation coefficient". In Samuel Kotz and Norman L. Johnson
(ed.). Encyclopedia of Statistical Sciences. Vol. 4. New York: John Wiley & Sons. pp. 213–
217.
2. Bartko JJ (August 1966). "The intraclass correlation coefficient as a measure of reliability".
Psychological Reports. 19 (1): 3–11. doi:10.2466/pr0.1966.19.1.3 (https://doi.org/10.2466%2
Fpr0.1966.19.1.3). PMID 5942109 (https://pubmed.ncbi.nlm.nih.gov/5942109).
S2CID 145480729 (https://api.semanticscholar.org/CorpusID:145480729).
3. Fisher RA (1954). Statistical Methods for Research Workers (https://archive.org/details/statist
icalmethoe7fish) (Twelfth ed.). Edinburgh: Oliver and Boyd. ISBN 978-0-05-002170-5.
4. Harris JA (October 1913). "On the Calculation of Intra-Class and Inter-Class Coefficients of
Correlation from Class Moments when the Number of Possible Combinations is Large".
Biometrika. 9 (3/4): 446–472. doi:10.1093/biomet/9.3-4.446 (https://doi.org/10.1093%2Fbiom
et%2F9.3-4.446). JSTOR 2331901 (https://www.jstor.org/stable/2331901).
5. Donner A, Koval JJ (March 1980). "The estimation of intraclass correlation in the analysis of
family data". Biometrics. 36 (1): 19–25. doi:10.2307/2530491 (https://doi.org/10.2307%2F25
30491). JSTOR 2530491 (https://www.jstor.org/stable/2530491). PMID 7370372 (https://pub
med.ncbi.nlm.nih.gov/7370372).
6. Proof that ICC in the anova model is the correlation of two items: ocram [1] (https://stats.stack
exchange.com/users/3019/ocram), Understanding the intra-class correlation coefficient, URL
(version: 2012-12-05): [2] (https://stats.stackexchange.com/q/45201)
7. dsaxton (https://stats.stackexchange.com/users/78861/dsaxton), Random effects model:
Observations from the same level have covariance $\sigma^2$?, URL (version: 2016-03-22)
link (https://stats.stackexchange.com/a/203052/253)
8. Stanish W, Taylor N (1983). "Estimation of the Intraclass Correlation Coefficient for the
Analysis of Covariance Model". The American Statistician. 37 (3): 221–224.
doi:10.2307/2683375 (https://doi.org/10.2307%2F2683375). JSTOR 2683375 (https://www.j
stor.org/stable/2683375).
9. Müller R, Büttner P (December 1994). "A critical discussion of intraclass correlation
coefficients". Statistics in Medicine. 13 (23–24): 2465–76. doi:10.1002/sim.4780132310 (http
s://doi.org/10.1002%2Fsim.4780132310). PMID 7701147 (https://pubmed.ncbi.nlm.nih.gov/7
701147). See also comment:
Vargha P (1997). "Letter to the Editor". Statistics in Medicine. 16 (7): 821–823.
doi:10.1002/(SICI)1097-0258(19970415)16:7<821::AID-SIM558>3.0.CO;2-B (https://doi.
org/10.1002%2F%28SICI%291097-0258%2819970415%2916%3A7%3C821%3A%3A
AID-SIM558%3E3.0.CO%3B2-B).
10. McGraw KO, Wong SP (1996). "Forming inferences about some intraclass correlation
coefficients". Psychological Methods. 1: 30–46. doi:10.1037/1082-989X.1.1.30 (https://doi.or
g/10.1037%2F1082-989X.1.1.30). There are several errors in the article:
McGraw KO, Wong SP (1996). "Correction to McGraw and Wong (1996)". Psychological
Methods. 1 (4): 390. doi:10.1037/1082-989x.1.4.390 (https://doi.org/10.1037%2F1082-98
9x.1.4.390).
11. Shrout PE, Fleiss JL (March 1979). "Intraclass correlations: uses in assessing rater
reliability". Psychological Bulletin. 86 (2): 420–8. doi:10.1037/0033-2909.86.2.420 (https://do
i.org/10.1037%2F0033-2909.86.2.420). PMID 18839484 (https://pubmed.ncbi.nlm.nih.gov/1
8839484).
12. Nickerson CA (December 1997). "A Note on 'A Concordance Correlation Coefficient to
Evaluate Reproducibility' ". Biometrics. 53 (4): 1503–1507. doi:10.2307/2533516 (https://doi.
org/10.2307%2F2533516). JSTOR 2533516 (https://www.jstor.org/stable/2533516).
13. Stoffel MA, Nakagawa S, Schielzeth J (2017). "rptR: repeatability estimation and variance
decomposition by generalized linear mixed-effects models" (https://doi.org/10.1111%2F204
1-210x.12797). Methods in Ecology and Evolution. 8 (11): 1639–1644. doi:10.1111/2041-
210x.12797 (https://doi.org/10.1111%2F2041-210x.12797). ISSN 2041-210X (https://www.w
orldcat.org/issn/2041-210X).
14. MacLennan RN (November 1993). "Interrater Reliability with SPSS for Windows 5.0". The
American Statistician. 47 (4): 292–296. doi:10.2307/2685289 (https://doi.org/10.2307%2F26
85289). JSTOR 2685289 (https://www.jstor.org/stable/2685289).
15. McGraw KO, Wong SP (1996). "Forming Inferences About Some Intraclass Correlation
Coefficients". Psychological Methods. 1 (1): 30–40. doi:10.1037/1082-989X.1.1.30 (https://d
oi.org/10.1037%2F1082-989X.1.1.30).
16. Stata user's guide release 15 (https://www.stata.com/manuals/r.pdf) (PDF). College Station,
Texas: Stata Press. 2017. pp. 1101–1123. ISBN 978-1-59718-249-2.
17. Howell DC. "Intra-class correlation coefficients" (https://www.uvm.edu/~dhowell/methods9/S
upplements/icc/More%20on%20ICCs.pdf) (PDF).
18. Liljequist D, Elfving B, Skavberg Roaldsen K (2019). "Intraclass correlation - A discussion
and demonstration of basic features" (https://www.ncbi.nlm.nih.gov/pmc/articles/PMC664548
5). PLOS ONE. 14 (7): e0219854. doi:10.1371/journal.pone.0219854 (https://doi.org/10.137
1%2Fjournal.pone.0219854). PMC 6645485 (https://www.ncbi.nlm.nih.gov/pmc/articles/PM
C6645485). PMID 31329615 (https://pubmed.ncbi.nlm.nih.gov/31329615).
19. Cicchetti DV (1994). "Guidelines, criteria, and rules of thumb for evaluating normed and
standardized assessment instruments in psychology". Psychological Assessment. 6 (4):
284–290. doi:10.1037/1040-3590.6.4.284 (https://doi.org/10.1037%2F1040-3590.6.4.284).
20. Koo TK, Li MY (June 2016). "A Guideline of Selecting and Reporting Intraclass Correlation
Coefficients for Reliability Research" (https://www.ncbi.nlm.nih.gov/pmc/articles/PMC49131
18). Journal of Chiropractic Medicine. 15 (2): 155–63. doi:10.1016/j.jcm.2016.02.012 (https://
doi.org/10.1016%2Fj.jcm.2016.02.012). PMC 4913118 (https://www.ncbi.nlm.nih.gov/pmc/ar
ticles/PMC4913118). PMID 27330520 (https://pubmed.ncbi.nlm.nih.gov/27330520).

Others
A comparison of two indices for the intraclass correlation coefficient (https://pubmed.ncbi.nl
m.nih.gov/22396135/)

External links
AgreeStat 360: cloud-based inter-rater reliability analysis, Cohen's kappa, Gwet's AC1/AC2,
Krippendorff's alpha, Brennan-Prediger, Fleiss generalized kappa, intraclass correlation
coefficients (https://agreestat360.com/)
A useful online tool that allows calculation of the different types of ICC (http://department.ob
g.cuhk.edu.hk/researchsupport/IntraClass_correlation.asp)

Retrieved from "https://en.wikipedia.org/w/index.php?title=Intraclass_correlation&oldid=1153458997"