Quant.

Methods &
Measurement Design
“If we have data, let’s look at data. If all
we have are opinions, let’s go with
mine.”
What questions do you have as
we get started?
 Reading for this week: Kettler Chapter 3
(measurement design/development)
 We’ll discuss the reading briefly, but NOT
everything!

 Planning Guide 1 (P1) – feedback posted on

Reminders Blackboard

& To-Do  P2 – in class today

 To-do for next week:

 Merriam & Tisdell chapters 1, 3, & 4
 P3, P4, & P5 (in class)
 Reflective journal posting 1 (after classes next week)
 Quick revisit of validity – we’ll discuss more in
week 4

 The quantitative approach – very broadly

 Correlations, experimentation, and beyond!

Plan for  Assessment & measurement - briefly

Today  Direct & indirect measures
 Tests vs. rating scales
 Process & instrument development as research

 Planning Guide 2 – brainstorming and initial

research question drafts
 Kettler (2019) discusses four types of validity:
 Statistical conclusion validity

 Internal validity

Some quick  Construct validity

notes about  External validity

Validity
 We’ll discuss validity in more depth in ~3 week,
but…
 I don’t think he does a great job of making it
accessible right off the bat!
 Validity & reliability are often used to describe
measures that we use in research
 E.g., is the Wechsler Intelligence Scale for Children a
valid measure of intelligence in childhood

 Validity can be thought of as accuracy, and reliability

is then consistency/precision
So, Validity
in General  A measure having good reliability does NOT mean
that it has good validity
 We can be consistently wrong!

 However, we do need to have reliability in order to

have validity
 If a measure isn’t reliable, it can’t be valid
Thinking about it Visually
 Are the conclusions you make from your analysis (about
the relationships among variables) reasonable based on
the data & your question?
 You didn’t make a Type I error (false positive)
 You didn’t make a Type II error (false negative)
Statistical
 Depends on using the correct analysis/analyses
 For example, let’s say we gathered data on whether Conclusio
or not study subjects believe that humans are the
product of evolution by natural selection: n Validity
Yes/No/Unsure
 If we use something like a t-test to analyze these
data, we’re violating the core assumptions or
requirements of that test – that leads to problems
with this type of validity
 Does the evidence actually support the claims made
about a causal relationship between two variables?

 A study finds that the correlation between ice cream

sales and motorcycle sales is +0.89 – a very strong

Internal
positive correlation
 As a result, the researchers claim that buying

Validity
motorcycles makes people buy ice cream due to the
excess heat and wind as compared to driving a car.

 Does the evidence support their claim?

 NO!

 Lots of things impact internal validity – not just

experimenter bias/mistakes!
 Does your measure actually measure what it’s supposed
to?

 For instance, does my new intelligence test actually

measure intelligence, or does it just measure current
knowledge?
Construct
 This is based on a number of things:
 Characteristics of the construct (what is intelligence?)
Validity
 Alignment with previous measures (do results for my
new test correlate with the WAIS?)
 Lack of alignment with measures of other constructs
(my intelligence test shouldn’t be as strongly
correlated with a test of neuroticism)
 How generalizable or transferrable are your results?

 My dissertation research looked at open learning about car

safety systems among college students
 How well might those results generalize to…
 …all college students learning about car safety
systems? External
Validity
 …new drivers learning about car safety systems?
 …adults learning about car safety systems?
 …children learning about car safety systems?

 Depends on (among other elements):

 The context of the study
 The context you hope to generalize to
 Sampling/selection procedures, other potential biases
Questions about
validity?
Quantitative
Research

What are the defining characteristics?

 Focus on quantities
 Test scores, ratings, etc.

 Most often large, random samples

 Single-subject designs can be an exception!

Quantitative  Evaluating theories and hypotheses using

Research objective (as much as possible) measures
 Validity/reliability are crucial

 Deductive process
 “Top-down”
Deductive Reasoning in Quant. Research

• “Top-down” reasoning
• Move from more general
(theory) to more specific (case)
• Drawing a specific conclusion
based on a general premise
• Start with theory:
• Make a hypothesis
• Test
• Confirm or refute
CAUSATION
 X caused (is responsible for) the change in Y

 Experimentation allows us to find this

 Controlling all other variables

Types of
Relationships CORRELATIONS/ASSOCIATIONS
 X and Y typically change together
 -1 to 1
 0 is no relationship; Closer to 1 or -1, stronger
relationship
 Correlation is NOT causation
 May be other variables involved
 There are other types of relationships
that we can find with quant. research,
such as:
Brief Aside  Moderating effects
 X impacts Y, but its impact depends on (is
on moderated by) Z
Relationship  Mediating effects

s  X impacts Z, and then Z impacts Y

 X may or may not impact Y directly

 For now, we’ll just worry about

causation & association
 Experimentation
 “Gold standard”

 2+ groups*

Quantitative  Experimental plus control

Methods -  Recruit -> randomize -> intervention (manipulate IV)

Causation -> test DV

 Causal relationships

 True experiments vs. quasi-experiments

Quant. Methods –Survey
• Strengths:
• Easy to gather a lot of information
• Can ask about subjects that are difficult
to test
• Weaknesses:
• Susceptible to subject bias/expectations
• Susceptible to poor wording of
questions
• Typically correlational
• Causation cannot be found in a survey
 Surveys are a form of observational work

 We can also use quantitative observations in a

Quant.
more traditional sense
 Naturalistic or lab/other setting
 Direct or participant
Methods -
 Similar strengths/weaknesses to surveys:
 Can gather a lot of data on difficult variables
Observati
to manipulate
 Can’t establish causation – other things at
on
play
 **Very susceptible to researcher bias
without a clear protocol
Questions before
we move on?
Let’s talk
measurement
Let’s talk Measurement

Research is empirical – it’s based on observations and measurements

That is, the data for a study are gathered by observing and measuring changes
in variables

In order to measure those variables correctly, we need systematic, accurate

procedures for doing that

Sometimes we can measure things

When we can’t, we need to figure out indirect
directly (e.g., height or weight), but measures
often not!
Let’s talk Measurement
Research is empirical – it’s based on observations and measurements

That is, the data for a study (especially quant.) are gathered by observing and
measuring changes in variables

In order to measure those variables correctly, we need systematic, accurate

procedures for doing that

Sometimes we can measure things

When we can’t, we need to figure out indirect
directly (e.g., height or weight), but measures
often not!
CONSTRUCT
 Internal characteristics of an individual that we
can’t directly measure
 Personality, intelligence, etc.

About those  These help us describe/explain behavior

indirect  They seem introverted, they seem to be very open to

new ideas, etc.

measures… OPERATIONAL DEFINITIONS

 Gives us a unified, shared way we can think about and measure a
construct

 Gives us specific rules for how to define, measure, and test

hypotheses about that construct

 Two people with the same op. def. should consistently agree on
the measure of that construct
Let’s create an operational definition
What is “learning?”
Another Way to Think About This Issue

Are snakes,
How does the
sharks, and
answer to this
crocodiles more
question depend
dangerous than
on operational
cattle, horses,
definitions?
and deer?
Another Way to Think About This Issue

Are snakes,
How does the
sharks, and
answer to this
crocodiles more
question depend
dangerous than
on operational
cattle, horses,
definitions?
and deer?
 Kettler (2019) distinguished between two
major approaches to
measurement/assessment:
 Tests
 “Direct” measure of the construct of interest
OK, so we have a  If we’re interested in intelligence, we can
implement a (valid!) intelligence test
definition – how  Seen as more objective than rating scales,
do we measure because they are more direct
that variable?
 Rating scales
 Rather than scoring directly, a rater indirectly
evaluates the behavior of the individual
 The rater sometimes is the individual being rated
 Many use Likert-type scales, but not all
 Kettler (2019) provides a five-phase process:
1. Planning – establishing operational definitions, etc.

Actually 2.
3.
Development – creating items, reviewing, etc.
Testing (piloting) – piloting the measure with users
developing 4. Completing forms – designing final versions, etc.

measures
5. Sharing results – scoring, documentation, etc.

 We won’t talk through all five of these in detail for the

sake of time, but let me know if you have questions!
 Creswell & Clark (2018) provide a mixed methods
approach to instrument design as research
 Goal is to create a psychometrically-sound
measure of the construct of interest Instrument
 Exploratory sequential design developme
 Collection of initial qualitative data nt as
 Analysis of qualitative data
 Initial instrument design based on qual. sample
research
 Administered/tested with wider sample*
 Update instrument & repeat testing as needed
 Publish/disseminate/etc.
 Planning Guide 2 (P2) is all about getting started
on a research question

 You are welcome to use any of the perspectives

we discussed last week to approach a topic of
Planning interest

Guide 2  Keep in mind that each approach might ask

different questions
 E.g., dropping out of school:
 Positivist: to what extent does dropping out of school
impact income level?
 Critical: what systems make particular groups more
likely to drop out of school, and how can we
address/fix those systems?
 Planning Guide 2 (P2) is all about getting started on a
research question

 Coming up with questions:

 Two major approaches: start with interesting questions

Planning and find methods that fit OR choose a method and find
a question that works with it

Guide 2  Your questions could have a clear/unequivocal answer

OR they can be more interpretive – quant. vs. qual.!
 Be mindful about specific populations or sites where
you could potentially gather data
 Your question should involve some kind of connection
or relationship, but not necessarily correlational or
causal (doesn’t have to be quant.)
Based on your P2
discussions…

What would you like to discuss? What questions do

you have?
 As a group, choose a construct to discuss
 E.g., personality, intelligence, fear, etc. – it can be
connected to school psych or not!

 Create an operational definition for your construct

If we have  What are you really trying to get at? What would tell

time: item you about that construct?

developmen  Try writing some (3-5) questions that could be

t used on either a test OR a rating scale in order to
assess that construct
 Test or rating scale?
 What questions will you ask, and how will they be
answered?
 How does each question address the construct?