Program: RIP™ -Based
Inquiry and Statistics for Grades 6-12 Teachers at the University
of Hawaii, Graduate Professional
Development Series Seminar for Science Teachers 2002, ANOVA Science Education
Corporation, Honolulu, HI.
Program evaluation
submitted: 9-18-2002
The overall purposes of this graduate seminar were to review
and strengthen the participants’ understanding of and ability and confidence to
use the research investigation process (RIP) used in scientific inquiry, and to
introduce 6-12 grade teachers to research design, data handling and summary,
and data analyses techniques and procedures as they are used in the RIP. Specifically, this three-day seminar was
designed for teachers to explore the research investigation process; to use the
inquiry process to learn how to design and conduct scientific research studies;
to become familiar with techniques to assist in guiding students through the
scientific inquiry process; understand and be able to obtain random samples;
develop research investigations using independent and dependent designs; to
learn how to represent and present data obtained from research investigations;
to examine, practice, understand, and become competent in the ability to apply
data analysis techniques, including the quantification of error and statistical
tests for significance, to decision-making in science; and to increase
confidence in using scientific research in their approach to instructing
students in science and in addressing the scientific inquiry benchmarks and
science inquiry content standards. The
research investigation process (RIP) was reviewed and teachers were provided
the opportunity to further develop their understanding of each of the elements
of the RIP through their participation in and development of actual research
investigations. Techniques in data
summary, analysis and presentation were explored in the context of hypothesis
testing and decision-making in science.
Finally, numerous strategies for the teaching of science through true
scientific inquiry were also emphasized.
All aspects of this seminar were aligned with the State of Hawaii Science Content
and Performance Standards.
The data for this graduate seminar evaluation were obtained
from assessments of the 14 teacher-participants at the beginning of
(Pre-Assessment) and again at the end (Post-Assessment) of the 3-day
seminar. Items on the assessments
required demonstration of knowledge and understanding about the scientific
inquiry process, data analyses procedures, and decision-making in science. A number of these items required participants
to demonstrate their knowledge and understanding through application. Self-report items measured teacher confidence
levels in understanding and using data summary, representation, and analysis
techniques in scientific inquiry conducted in the classroom, and in comprehending
and applying the scientific inquiry content standards to their
instruction. A concept inventory
determined teachers’ familiarity with and ability to teach elements of
scientific inquiry, research design, and data summary and statistical analysis
techniques. The pre-seminar and
post-seminar assessment items were the same except for three additional
self-report items included on the post-assessment. These additional items assessed the teachers’
perceptions of how much their understanding of the research investigation
process and scientific inquiry process improved, and understanding of
application of statistics to data increased, as a result of participation in
the seminar. The data were statistically
analyzed using dependent t-tests to determine significant differences
(indicating change) between pre- and post-assessment mean values. For comparisons in which the assumption of
normality was violated, the Wilcoxon Signed Rank Test (nonparametric t-test)
was used to compare central tendencies.
The Pearson Product Moment Correlation Coefficient was used to test for
linear relationships. The criterion for
statistical significance (a) was set at
0.05.
Scientific Inquiry, Scientific Research, and the Research
Investigation Process (RIP)
This seminar focused on review and strengthening of teacher
understanding, abilities, and confidence related to instruction of science
through scientific inquiry, conducting scientific research, and engaging their
students in scientific research.
Although it was generally expected that teacher-participants would be
familiar with these aspects of scientific inquiry, a pre-seminar phone survey
of the participants’ perceived knowledge and abilities indicated that it would
be important to address the RIP before examining research design and analysis
strategies and procedures. Specific
emphases were placed on teacher demonstration of understanding the logic of
organization of the RIP elements, as well as the components and concepts
involved in each element.
Although participant self-reported familiarity
with/understanding of the RIP appeared to slightly increase from its
pre-seminar level by the end of the three-day seminar (Figure 1, below), the
difference was not statistically significant.
However, the power of the t-test was low and so the negative
statistical finding should be interpreted cautiously (see Figure 1 caption for
more details).
Mean (+SEM) RIP Concept Inventory Score
|
|
Figure 1. Familiarity with and
understanding of concepts related to methodology in the RIP.
Mean post-assessment value
did not differ from the mean pre-assessment value [t (13) =
-1.99, p=0.068]. Note: Because the power of the performed
statistical test (0.345) was below the desired power of 0.800, the results
should be interpreted cautiously.
At first glance, the lack of a significant change in
self-reported familiarity/understanding of the RIP concept appears surprising
because participant-demonstrated knowledge and understanding, and self-reported
confidence (discussed below) increased consistently for all of the RIP
assessment items. However, both the pre-
and post-assessment concept inventory mean values were relatively high and
indicated that the participants felt that they were already very familiar with
and had a good understanding of (could teach the concept to others) the RIP
before participating in the seminar.
Post-seminar teacher-participant demonstrated knowledge and
understanding of the RIP components and their contents (Figure 2, below) and of
the organization of the RIP elements (Figure 3, below) significantly increased
compared to pre-seminar levels.
Mean (+SEM) Level of Demonstrated Knowledge and
Understanding
of the RIP
|
|
Figure 2. Knowledge and
understanding of the research investigation process (RIP) components and
content. (Highest possible value for
knowledge and understanding was six.)
*Mean post-assessment value differs from the mean
pre-assessment value [t (13) =
-3.29, p=0.006].
Mean (+SEM) Level of Demonstrated Knowledge of
Organization of the RIP Elements
|
|
Figure 3. Knowledge and
understanding of the organization of the elements of the research investigation
process (RIP). (Highest possible level
of knowledge was seven.)
*Mean post-assessment value differs from the mean
pre-assessment value [t (13) = -2.49, p=0.027].
By the end of the three-day seminar compared to
pre-seminar levels, participants’ self-reported confidence levels for both
ability to use scientific inquiry in their instruction and their ability to
teach and engage students in scientific research activities increased from
“confident” to midway between “confident” and “very confident.” Although these increases were only slight to
moderate, they were both statistically significant (Figures 4 and 5, below).
Mean (+SEM) Confidence
Score
|
|
Figure 4. Self-reported
confidence levels for ability to use scientific inquiry in instruction.
Mean post-assessment value is significantly greater
than mean pre-assessment value [t (13) = -2.60, p=0.022].
Figure 5. Self-reported
confidence levels for ability to teach and engage students in scientific
research activities.
* Mean post-assessment value
is significantly greater than mean pre-assessment value [W (N=14)
= -54.0, p=0.034].
Design and Analysis
The seminar participants exhibited increases in their
knowledge and understanding as well as in their confidence related to the
elements involved in designing research investigations so that they can be
analyzed using the appropriate statistical tests. By the end of the 3-day seminar, participants
demonstrated a statistically significant increase in their knowledge and
understanding of research designs, statistical concepts including, but not
limited to, random sampling, error (variability), and statistical tests (Figure
6, below).
Mean (+SEM) Research Design and Statistics Score
|
|
Figure 6. Demonstration of knowledge and understanding
of research design and statistical concepts.
(Scores were based on the number of items correct
out of 10.)
* Mean post-assessment value
is significantly greater than mean pre-
assessment value [t (13) = -4.70, p<0.001].
The post-seminar increase in teacher-participant knowledge
and understanding of the research process was accompanied by a significant increase in teacher’ self-reported
familiarity and understanding of concepts related to research design and
methodology (Figure 7, below) and data analysis including descriptive and
inferential statistics (Figure 8, below).
Mean (+SEM) Research Design/Methodology Concept
Inventory Score
|
|
Figure 7. Familiarity with and
understanding of concepts related to research design and methodology.
* Mean post-assessment value is significantly greater
than mean pre-assessment value [t (13) = 6.38, p<0.001].
Mean (+SEM) Data Analysis Concept Inventory Score
|
|
Figure 8. Familiarity with and
understanding of concepts related to data analysis, including descriptive and
inferential statistics.
* Mean post-assessment value is significantly greater
than mean pre-assessment value [t (13) = -5.57, p<0.001].
By the end of the seminar, the average participant’ response
to the concept of research design and methodology rose from between “somewhat
familiar, but do not really understanding what it means” and “familiarity with
fair understanding of the concept” to “very familiar with concept and could
teach it to others.” Likewise, the
average participant’ response to the concept of data analysis rose from
“somewhat familiar, but do not really understanding what it means” to between
“familiarity with fair understanding of the concept” and “very familiar with
concept and could teach it to others.”
These findings indicated that teachers recognized their increased
knowledge and understanding related to research design and statistical
concepts.
With the increase in knowledge and understanding, the
teacher-participants also demonstrated an increase in self-reported confidence
in their understanding of the application of basic statistics to analyze research
data (Figure 9, below).
Mean (+SEM) Confidence
Score
|
|
Figure 9. Self-reported
confidence levels for understanding of the application of basic statistics to
analyze research data.
* Mean post-assessment value
is significantly greater than mean pre-assessment value [t (13) =
6.07, p<0.001].
By the end of the three-day seminar, the mean participant
confidence score more than doubled, significantly rising from between “not at
all confident” and “somewhat confident” to near “confident.”
Teacher Understanding of and Ability to Apply Data
Summary, Presentation, and Analysis techniques to Decision-Making in Science
By the end of the three-day workshop, seminar participants
demonstrated a significant 33% increase in their ability to determine the
appropriate statistic for describing the center of a group of data (Figure 10,
below).
|
|
|
|
|
Mean (+SEM) Score for Ability to Use Appropriate
Measure of Central Tendency
|
|
|
|
|
Figure 10.
Demonstration of the ability to determine the appropriate measure of
central tendency for a group of data.
(Highest possible ability score was four.)
* Mean post-assessment value
is significantly greater than mean pre-
assessment value [t (13) = 2.79, p=0.015].
In addition to gains in understanding of the application of
descriptive statistics, participants also demonstrated a post-seminar increase
in knowledge of the procedures used to calculate the three measures of central
tendency (Figure 11, below).
Mean (+SEM) Score for Ability to Calculate the
Three Measures of Central Tendency
|
|
Figure 11.
Demonstration of the ability to calculate the three measures of central
tendency. (Highest possible ability
score was six.)
* Mean post-assessment value
is significantly greater than mean pre-assessment value [t (13) =
3.16, p=0.008].
The seminar participants’ self-reported familiarity with,
and understanding of the concepts of, the three measures of central tendency
also significantly increased by the end of the three-day seminar (Figure 12,
below). This increase coincided with the
demonstrated increases in understanding and ability to calculate the measures
of central tendency (both discussed above), indicating that the participants
were able to accurately perceive and report their relative knowledge of and
ability to summarizing groups of data.
Mean (+SEM) Measures of Central Tendency Concept
Inventory Score
|
|
Figure 12. Familiarity with and
understanding of concepts related to the three measures of central tendency.
* Mean post-assessment value is significantly greater
than mean pre-assessment value [t (13) = 3.60, p=0.003].
Figure 13 (below) presents a scatterplot showing the pre- to
post-seminar change in participants’ familiarity/understanding of the concept
of measures of central tendency plotted as a function of change in demonstrated
participant knowledge of descriptive statistics. Pearson Product Moment Correlation
Coefficient revealed a significant moderate positive relationship between these
two variables (Figure 13).
|
|
|
|
|
|
|
|
 |
|
Amount of
Change in Self-Reported Familiarity/ Understanding
|
|
|
|
|
|
Amount of Change in Demonstrated Knowledge
|
|
Figure
13. Scatterplot of pre- to post-seminar
change in familiarity/understanding as a function of change in demonstrated
knowledge of measures of central tendency.
By the end of the three-day seminar, participant-reported
familiarity/understanding to the concept of error variance (standard deviation)
steeply increased compared to the pre-seminar value (Figure 14, below). Before taking the seminar, the
teacher-participants reported that the
concept “standard deviation” was between “somewhat familiar” and “familiar,” but have less than “a fair understanding of
what it means.” After participating in
the seminar, teachers reported that they were “very familiar with this concept,
but would have some difficulty teaching it to others.”
Mean (+SEM) Standard Deviation Concept Inventory
Score
|
|
Figure 14. Familiarity with and
understanding of the concept of standard deviation.
* Mean post-assessment value is significantly greater
than mean pre-assessment value [t (13) = -7.78, p<0.001].
The participants confidence in their understanding of how to
use statistics to describe the center and dispersion (variability) of a group
of data also substantially increased from less than “somewhat confident” to
slightly greater than “confident” by the end of the seminar (Figure 15,
below).
Mean (+ SEM) Confidence
Score
|
|
Figure 15. Self-reported confidence
levels for ability to use statistics to describe the center of and variability
for a group of data.
* Mean post-assessment value
is significantly greater than mean pre-assessment value [t (13) =
-4.52, p<0.001].
Seminar participants exhibited a statistically significant
increase in their ability to correctly identify the purpose of, and accurately
interpret data presented in graphs displaying central tendencies for two groups
(Figure 16, below) and scatterplots (Figure 17, below).
Mean (+SEM) Score for Ability to Recognize and
Interpret Bar Graphs
|
|
Figure 16.
Demonstration of the ability to correctly recognize the purpose and
accurately interpret data presented in graphs displaying the central tendencies
of two groups. (Highest possible ability
score was five.)
·
Mean post-assessment value is significantly greater than mean pre-assessment
value [t (13) = -2.81, p=0.015].
|
|
|
|
|
|
|
|
Mean (+SEM) Score for Ability to Recognize and
Interpret Scatterplots
|
|
|
Figure 17.
Demonstration of the ability to correctly recognize the purpose and
accurately interpret data presented in scatterplots. (Highest possible ability score was five.)
* Mean post-assessment value
is significantly greater than mean pre-assessment value [t (13) =
-3.83, p=0.002].
Increases in the ability of participants to construct graphs
for displaying of data and comparison of central tendencies also significantly
increased by the end of the seminar (Figure 18, below).
Mean (+SEM) Score for Ability to Construct Graphs
|
|
Figure 18.
Demonstration of the ability to construct graphs for display of data and
the comparison of central tendencies.
(Highest possible ability score was five.)
* Mean post-assessment value
is significantly greater than mean pre-assessment value [t (13) =
-3.52, p=0.004].
Consistent with demonstrated increases in abilities to
summarize data, identify, accurately interpret, and correctly construct tables
and graphs, significant increases in participant self-reports of familiarity
and understanding for concepts related to representations of data, including
tables and graphs, were exhibited at the end of the three-day seminar (Figure
19). At the start of the seminar,
teachers reported being “familiar with these concepts, with a fair
understanding of what they mean.” By the
end of the seminar, the teachers reported being “very familiar with these
concepts, but would have some difficulty teaching them to others.”
Mean (+SEM) Data Representation Concept Inventory
Score
|
|
Figure 19. Familiarity with and
understanding of concepts related to representing data in tables and graphs.
* Mean post-assessment value is significantly greater
than mean pre-assessment value [t (13) = -4.97, p<0.001].
Hawaii
State Science Content Standards
Teacher confidence in
ability to address science content standards relating to collecting, organizing,
displaying and analyzing data in their classroom was also affected by the end
of the seminar. Self-reported confidence
in ability to address content standards in the classroom rose significantly
from between “confident” and “somewhat confident” to above “confident” by the
end of the seminar (Figure 20, below).
Mean (+SEM) Confidence
Score
|
|
Figure 20. Self-reported
confidence levels for ability to address data summary and analysis science
content standards in the classroom.
* Mean
post-assessment score is significantly greater than mean pre-assessment score [t
(13) = -3.45, p=0.004].
Teacher
Perceptions of Impact from their Participation in this Graduate Seminar
The post-assessment contained three additional
self-report items designed to assess how much teacher-participants believed
their understanding about the scientific inquiry process, data analysis, and
the science inquiry standards improved as a direct consequence of their
participation in the seminar. The
results from these items are presented in Figures 21-23 below.
An overwhelming majority of the seminar-participants
(10 of 14) claimed that their understanding of the research investigation
process improved a large amount to completely as a result of their
participation in this seminar, while three participants claimed it changed a
moderate to a large amount (Figure 21, below).
One teacher reported that his understanding improved slightly less than
a moderate amount.
Figure
21. Pie chart representing
teacher-participants’ responses to “what extent, if any, did your understanding
of the research investigation process improve as a result of your participation
in this graduate seminar?” The scale for
responses included none, a small amount, a moderate amount, a large amount, and
completely.
A majority of the seminar-participants (9 of 14)
claimed that their understanding of the application of statistics to analyze
data obtained in a research investigation improved a large amount to completely
as a result of their participation in the 3-day seminar (Figure 22,
below). Four of the participants claimed
it improved a moderate to a large amount as a result of their participation. One teacher reported only a small to
moderate improvement on this item.
Figure
22. Pie chart representing
teacher-participants’ responses to “what extent, if any, did your understanding
of the application of statistics for understanding the data obtained from a
scientific research investigation increase as a result of your participation in
this graduate seminar?” The scale for
responses included none, a small amount, a moderate amount, a large amount, and
completely.
Fifty percent of the seminar-participants (7 of 14)
claimed that their understanding of the science inquiry standards improved a
large amount to completely as a result of their participation in this seminar,
five (about 36%) claimed it improved a moderate to a large amount, and two
reported that it improved a small to moderate amount (Figure 23, below).
Figure
23. Pie chart representing
teacher-participants’ responses to “what extent, if any, did your understanding
of the scientific inquiry standards improve as a result of your participation
in this graduate seminar?” The scale for
responses included none, a small amount, a moderate amount, a large amount, and
completely.