What Is the Design?

Longitudinal Designs
• Cohort: Individuals or groups with a common starting point.
– Birth cohorts--those who share a common period of birth
– Seniority cohorts--those who have worked at the same
place for about 5 years, about 10 years, and so on.
– Event cohort--people who have shared an event, for
instance, all the victims of Hurricane Sandy that hit the
Northeast coast of the United States in 2012.
• Types of cohorts:
– Birth cohorts
– Event cohorts

1
People are living longer

2
Lecture 2

Conceptualization and
Measurement

Yifan Shen
What Do We Have in Mind?

• Concepts: A mental image that

summarizes a set of similar observations,
feelings, or ideas.

7
What Do We Have in Mind?
• Variables: A characteristic that does not
have a fixed value.
– Constant: a special variable that has a fixed
value in a given situation; a characteristic or
value that does not change.
– School affiliation

8
Units of analysis
• The entities being studied, whose behavior is
to be understood.
– individual people, families, groups, colleges,
governments, cohorts, or nations
• Variable is a quality, attribute, or
characteristic of the unit of analysis
• The distribution of a variable describes the
variation across units of analysis in the
attribute associated with that variable.
9
Variation
• Descriptive Research
– Describe
variation(s)

10
Variation
• Explanatory Research
– Relationship between two variations

11
Example
• Do taller students perform better in their
final exam than shorter students?
• Unit of analysis?
• Variable
– Independent Variable
– Dependent Variable

13
Grade on
Height
Average

Tall students
3.8
(height>=175cm)

Short students
3.6
(height<175cm)

Variation in height Variation in exam result

14
Example
• Rampage: The Social Roots of School
Shootings by Katherine S. Newman et al.
(2004) studied the case histories of 27
different teenagers who had gone into
their schools and killed (mostly random)
fellow students.

15
Example
• Newman and colleagues concluded that there
were five “necessary but not sufficient” factors in
school shootings:
– a self-perception of shooters as socially marginal,
– psychosocial problems,
– cultured scripts linking masculinity and violence,
– failure of surveillance systems (so troubled kids are
“under the radar”), and
– availability of guns.

• Virtually all school shooters fit this description;

they have all these characteristics.
16
Example
• Were there other students who perceived
themselves as socially marginal or who
had psychosocial problems, and so on?
Why didn’t these other students turn into
school shooters?
• Their sampling frame (a list of school
shooters) allowed them to generalize to
other school shooters but not to tell you
how shooters differ from other teenagers. 17
What Is the Design?
• Units exist at different levels of collectivity
– Ecological fallacy: An error in reasoning in which
conclusions about individual-level processes are
drawn from group-level data.

18
Ecological fallacy
• Richer countries have higher rates of heart
disease; therefore, richer people have
higher rates of heart disease.
– In rich countries, yes, there is more heart
disease, but actually, it’s among the poor
individuals within those countries.

19
• In the 2012 US presidential election,
Republicans won the House of
Representatives, but Democrats held onto
the Senate, and President Obama was
reelected; therefore, Americans want a
divided government.
– America as a whole may “want a divided
government,” but relatively few Americans do.

20
How Will We Know When We’ve
Found It?
• Operation: A procedure for identifying or
indicating the value of cases on a variable.
• Operationalization
– The process of specifying the operations that
will indicate the value of cases on a variable.
– Should be consistent with research question.
– Consider time and resource limitations.

21
Chambliss, Making Sense of the Social World, 6e. © SAGE Publishing, 2020 22
• Public Opinion
– Satisfaction with the Government
– Indicator?

23
How Will We Know When We’ve
Found It?
Constructing Questions
• Single Questions
– Closed-ended (fixed-choice) question
• Response choices should be mutually exclusive and
exhaustive.
– Mutually exclusive: A variable’s attributes (or values) are mutually
exclusive when every case can be classified as having only one
attribute (or value).
– Exhaustive: Every case can be classified as having at least one
attribute (or value) for the variable.
– Open-ended question: A survey question to which
respondents reply in their own words, either by writing or
by talking.
• Mainly used when studying small groups
• Preferable when responses cannot be predicted
24
How Will We Know When We’ve
Found It?
Constructing Questions
• Indexes and Scales
– Index: A composite measure based on summing, averaging, or
otherwise combining the responses to multiple questions that are
intended to measure the same concept.
– Scale: A composite measure based on combining the responses
to multiple questions pertaining to a common concept after these
questions are differentially weighted, such that questions judged
on some basis to be more important for the underlying concept
contribute more to the composite score.
– Must demonstrate consistency of responses
– Many valid indexes already exist
25
How Will We Know When We’ve
Found It?
Making Observations
• Observations can be . . .
– Primary form of measurement
– Supplementary measures to questions
• Direct observations are often method of choice for
measuring behavior in natural settings
– Can allow measurement in context, but observations can
also be distorted
– If you set up a video camera in an obvious spot on campus
to monitor traffic flows, you may well change the flow—just
because people will see the camera and avoid it (or come
over to make faces).

27
How Will We Know When We’ve
Found It?
Combining Measurement Operations
• Triangulation: The use of multiple methods to
study one research question.
– Can strengthen measurement
– Divergence could indicate measures operationalize
different concepts.
– In surveys, for instance, people may say that they
would return a lost wallet they found on the street. But
field observation may prove that in practice, many
succumb to the temptation to keep the wallet.
– Explicit vs implicit attitude

28
How Much Information Do We
Really Have?
Level of measurement
The mathematical precision with which the values of a
variable can be expressed. The nominal level of
measurement, which is qualitative, has no mathematical
interpretation; the quantitative levels of measurement--
ordinal, interval, and ratio--are progressively more precise
mathematically.

Nominal Level of Measurement: Variables whose values have

no mathematical interpretation; they vary in kind or quality but
not in amount.

• Examples: State, religious affiliation, and so on.

• Categories must be mutually exclusive and exhaustive.
29
How Much Information Do We
Really Have?
Ordinal Level of Measurement: A
measurement of a variable in which the
numbers indicating a variable’s values specify
only the order of the cases, permitting greater
than and less than distinctions.

• Focused on greater than and less than

• As with nominal variables, different values
must be mutually exclusive and exhaustive.
• People’s satisfaction with the HKSAR government
30
How Much Information Do We
Really Have?
Interval Level of Measurement: A
measurement of a variable in which the
numbers indicating a variable’s values
represent fixed measurement units but have
no absolute, or fixed, zero point.

• No absolute zero point

31
How Much Information Do We
Really Have?
Ratio Level of Measurement
• A measurement of a variable in which the
numbers indicating the variable’s values
represent fixed measuring units and an
absolute zero point.
• Values are fixed.
• There is an absolute zero point.
• Ratio numbers can be added, subtracted,
multiplied, and divided.
• Can be used in more complex data analyses.
32
33
How Much Information Do We
Really Have?
Comparison of Levels of Measurement
• Researchers choose levels of
measurement when operationalizing
variables.
• Usually a good idea to measure at highest
possible level.

35
Did We Measure What We Wanted
to Measure?
Measurement Validity
• Criterion Validity
– The type of validity that is established by
comparing the scores obtained on the measure
being validated to those obtained with a more
direct or already validated measure of the same
phenomenon (the criterion).
– Relies on an already validated measure of a
phenomenon.
– For many concepts, it is hard to find a well-
established criterion. 36
Did We Measure What We Wanted
to Measure?
Measurement Validity

• Construct Validity: The type of validity that

is established by showing that a measure is
related to other measures as specified in a
theory.
– Must be sure that two measures are truly
independently produced.

37
Did We Measure What We Wanted
to Measure?
Reliability
• Reliability means that a measurement
yields consistent scores (so scores
change only when the phenomenon
changes).
• A prerequisite for measurement validity.

38
Did We Measure What We Wanted
to Measure?
Test–retest reliability: A measurement showing
that measures of a phenomenon at two points
in time are highly correlated, if the
phenomenon has not changed or has
changed only as much as the phenomenon
itself.

39
Did We Measure What We Wanted
to Measure?
Can We Achieve Both Reliability and
Validity?
• Reliability and validity of measures must
be tested after the fact.
– If they are not, little can be done to save the
study.
• Usually, it is best to use established
measures.

40
• The Dow Jones Industrials Index is a perfectly
reliable measure of the state of the U.S.
economy—any two observers of it will see the
same numbers—but its validity is shaky: There’s
more to the economy than the rise and fall of stock
prices.
• A good therapist’s interview of a married couple
may produce a valid understanding of their
relationship, but such interviews are often not
reliable because another interviewer could easily
reach different conclusions.

41
Conclusion
• Measurement validity is a necessary
foundation for social research.
• Achieving validity varies with the concept and
circumstances of the study.
• Plan ahead.
• Carefully evaluate others’ research and
remain skeptical.
• Use statistical tests when appropriate.

43