Program Evaluation: Scientific Research Through the RIP and Data
Analysis for K-5 Teachers- Professional Development Program, University of
Hawaii, Program conducted by ANOVA Science Education Corporation, 2002. Evaluation submitted August 22, 2002.
The purpose of this graduate seminar was to introduce K-5
teachers to the teaching of science through true scientific inquiry. Specifically, it was designed for teachers to
explore the research investigation process; to use the inquiry process to learn
how to design and conduct scientific research studies; to become familiar with techniques
to assist in guiding students through the scientific inquiry process; to
examine, practice, understand, and become competent in the ability to apply
data analysis techniques to decision-making in science; and to increase
confidence in using scientific research in their approach to instructing
students in science and in addressing the scientific inquiry benchmarks and
science inquiry content standards. The
research investigation process (RIP) was introduced and teachers were provided
the opportunity to further develop their understanding of each of the elements
of the RIP through their participation in and development of actual research
investigations. Techniques in data summary,
analysis and presentation were explored in the context of hypothesis testing
and decision-making in science. All
aspects of this seminar were aligned with the State of Hawaii Science Content and Performance
Standards.
The data for this graduate seminar evaluation were obtained
from assessments of the 14 teacher-participants at the beginning of
(Pre-Assessment) and again at the end (Post-Assessment) of the 3-day
seminar. Items on the assessments
required demonstration of knowledge about the scientific inquiry process, data
analyses procedures, and decision-making in science. A number of these items required teachers to
demonstrate their knowledge through application. Self-report items measured teacher confidence
levels in understanding and using scientific inquiry in the classroom and in
comprehending and applying the scientific inquiry content standards to their
instruction. A concept inventory
determined teachers’ familiarity with and ability to teach elements of
scientific inquiry and data summary and analysis techniques. The pre-seminar and post-seminar assessment
items were the same except for four additional self-report items included on
the post-assessment. These additional
items assessed the teachers’ perceptions of how much their understanding of
scientific inquiry changed and improved as a result of participation in the
seminar. The data were statistically
analyzed using dependent t-tests to determine significant differences
(indicating change) between pre- and post-assessment mean values. The criterion for statistical significance (a) was set at 0.05.
Teacher Knowledge and Understanding of the Scientific
Research Investigation Process (RIP), and Confidence in Teaching Scientific
Inquiry
Seminar participants demonstrated a large, statistically
significant increase in their knowledge and understanding of the individual
elements of the RIP by the end of the 3-day seminar (Figure 1, below). This included the logical order of the RIP
elements, understanding of components involved in each element, and
demonstration of the ability to construct testable hypotheses.
Figure 1. Demonstration of knowledge
and understanding of the elements of the RIP.
* Mean post-assessment score
is significantly greater than mean pre-assessment score [t (13) =
9.67, p<0.001].
The post-seminar increase in teacher-participant knowledge
and understanding of the research process was accompanied by a significant
increase in teacher’ self-reported familiarity and understanding of concepts
related to the scientific research process in the concepts inventory (Figure 2,
below). By the end of the seminar, the average participant’
response rose from “familiarity with fair understanding of concept” to “very
familiar with concept and could teach it to others.” This showed that teachers recognized their
increased knowledge and understanding.
Mean (+SEM) RIP Concept Inventory Score
|
|
Figure 2.
Familiarity and understanding of concepts related to elements of the
RIP.
* Mean post-assessment score is significantly greater
than mean pre-assessment score [t (13) = 6.48, p<0.001].
By the end of the 3-day seminar, participants’
self-reported confidence levels for their ability to use scientific inquiry,
their understanding of teaching science through inquiry, and their ability to
teach and engage students in scientific research activities all increased
significantly compared to pre-seminar (Figures 3, 4 and 5, below).
Mean (+SEM) Confidence
Score
|
|
Figure 3.
Self-reported confidence levels for ability to use scientific inquiry.
* Mean post-assessment score
is significantly greater than mean pre- assessment score [t (13)
= 11.08, p<0.001].
Mean (+SEM) Confidence
Score
|
|
Figure 4.
Self-reported confidence levels for understanding of teaching science
through inquiry.
* Mean post-assessment score
is significantly greater than mean pre-assessment score [t (13) =
9.25, p<0.001].
Mean (+SEM) Confidence
Score
|
|
Figure 5.
Self-reported confidence levels for ability to teach and engage students
in scientific research activities.
* Mean post-assessment score
is significantly greater than mean pre-
assessment score [t (13) = 7.46, p<0.001].
Teacher Understanding of and Ability to Apply Data
Summary, Presentation, and Analysis techniques to Decision-Making in Science
By the end of the seminar, participants demonstrated a
large, statistically significant increase in their knowledge and ability to
correctly organize data into a summary table and to construct a bar graph for
comparing the central tendency of two groups of data (Figure
6, below).
Mean (+SEM) Data Summary & Presentation Score
|
|
Figure 6. Demonstration of understanding and ability to apply data
organization and presentation techniques to data.
* Mean post-assessment score
is significantly greater than mean pre-assessment score [t (13) =
4.07, p<0.001].
Seminar participants also demonstrated a dramatic change in
their knowledge and ability to apply data analysis techniques to research
data. Comparison of the pre-and post-assessments
revealed that by the end of the seminar, participants significantly increased
their understanding of how to calculate descriptive statistics and their
ability to determine which measure of central tendency is most appropriate for
a group of data (Figure 7, below).
Mean (+SEM) Data Analysis Score
|
|
Figure 7. Demonstration of
understanding the calculations for descriptive statistics and ability to
determine the most appropriate statistic to represent central tendency for a
group of data.
* Mean post-assessment score
is significantly greater than mean pre-assessment score [t (13) =
5.99, p<0.001].
Teacher-participants demonstrated a statistically
significant increase in their ability to interpret data presented in
scatterplots and summarized in bar graphs by the end of the seminar (Figure 8,
below).
Mean (+SEM) Graph Interpretation Score
|
|
Figure 8.
Demonstration of ability to interpret scatterplots and
bar graphs.
* Mean post-assessment score
is significantly greater than mean pre-assessment score [t (13) =
4.01, p<0.001].
The post-seminar increase in teacher-participant knowledge
of and ability to apply data presentation and analyses were accompanied by a
significant increase in teacher’ self-reported familiarity and understanding of
concepts related to data presentation and analysis in the concepts inventory
(Figures 9 and 10, below). By the end of
the seminar, the average participant’ response for the three measures of
central tendency rose significantly from “somewhat familiar with concept, but
do not really understand what it means” to “very familiar with concept and
could teach it to others” (Figure 9).
Mean (+SEM) Central Tendency Concept Inventory
Score
|
|
Figure 9. Familiarity
and understanding of concepts related to measuring central tendency.
* Mean post-assessment score is significantly greater
than mean pre-assessment score [t (13) = 7.95, p<0.001].
Similarly, the average participant’ response for tables and
graphs rose significantly from “familiar with concept with a fair understanding
of what it means” to “very familiar with concept, but would have some
difficulty to teach it to others”
(Figure 10).
Mean (+SEM) Tables & Graphs Concept Inventory
Score
|
|
Figure 10.
Familiarity and understanding of concepts related to measuring central
tendency.
*Mean post-assessment score
is significantly greater than mean pre-assessment score [t (13) =
4.08, p<0.001].
Benchmarks and Standards
General teacher
confidence in and awareness of ability in understanding and applying scientific
inquiry to the teaching of science, and in ability to successfully address the
scientific inquiry standards were also affected by the end of the seminar. Teacher-participant self-reported confidence
in ability to address content standards in the classroom rose significantly by
the end of the seminar (Figure 11, below).
Mean (+SEM) Confidence
Score
|
|
Figure 11. Self-reported confidence levels for ability
to address content standards in the classroom.
* Mean
post-assessment score is significantly greater than mean pre-assessment score [t
(13) = 6.12, p<0.001].
Similarly, by the end of the seminar, participant
confidence about ability to accurately and completely address the scientific
inquiry standards dramatically increased from “somewhat confident” to near
“very confident” (Figure 12, below).
Mean (+SEM) Confidence
Score
|
|
Figure 12.
Self-reported confidence levels for ability to accurately and completely
address the scientific inquiry benchmarks.
* Mean
post-assessment score is significantly greater than mean pre-assessment score [t
(13) = 7.30, p<0.001].
Finally, by the end of the 3-day seminar, teachers
significantly increased their familiarity and understanding of inquiry
standards from being “somewhat familiar with this concept, but not really
understanding what it means” to being “very familiar with this concept, but
with some difficulty teaching it to others.”
This increase was statistically significant and was consistent with the
increase in teacher-participant confidence regarding scientific inquiry and
addressing the inquiry standards (Figure 13, below).
Mean (+SEM) Inquiry Standards Concept Inventory
Score
|
|
Figure 13. Familiarity and understanding of concept of inquiry standards.
*Mean
post-assessment score is significantly greater than mean pre-assessment score [t
(13) = 5.14, p<0.001].
Teacher
Perceptions of Impact from their Participation in this Graduate Seminar
The post-assessment contained four additional
self-report items designed to assess how much teacher-participants
believed their knowledge and abilities regarding the scientific research
investigation process and scientific inquiry were impacted by their
participation in this seminar. The
results from these items are presented in Figures 14-19 below.
An overwhelming majority of the seminar-participants
(12 of 14) claimed that their understanding of the research investigation
process was changed a large amount to completely as a result of their
participation in this seminar, while two participants claimed it changed a
moderate to a large amount (Figure 14, below).
Figure 14. Pie chart
representing teacher-participants’ responses to “what extent, if any, did your
understanding of the research investigation process change as a result of your
participation in this seminar?” The
scale for responses included none, a small amount, a moderate amount, a large
amount, and completely.
More than two-thirds of the seminar-participants (11
of 14) claimed that their understanding of the research investigation process
improved a large amount to completely as a result of
their participation in the 3-day seminar (Figure 15, below). Three of the participants claimed it improved
a moderate to a large amount as a result of their participation.
Figure 15. Pie chart
representing teacher-participants’ responses to “what extent, if any, did your
understanding of the research investigation process improve as a result of your
participation in this seminar?” The
scale for responses included none, a small amount, a moderate amount, a large
amount, and completely.
Figure 16 presents a scatterplot of the
teacher-reported increase in understanding of the research investigation
process plotted as a function of change in 
understanding of the research investigation process, both as a result
of participation in the seminar.
|
|
|
|
|
|
|
|
 |
|
Amount of
Increase in Understanding
|
|
|
|
|
|
Amount of Change in Understanding
|
|
Figure 16. Scatterplot of increase in understanding as a function of change in
understanding of the research investigation process, both resulting from
participation in the seminar.
As can be seen in Figure 16 above, there was a high
positive statistically significant relationship between the amount of change
and the amount of increase in understanding of the scientific research
investigation process. Approximately 79%
of the change in understanding is associated with the increase in
understanding.
Almost 79 percent of the seminar-participants (11 of
14) claimed that their understanding of the inquiry standards was changed a
large amount to completely as a result of their participation in this seminar,
while the remaining three participants claimed it changed a moderate to a large
amount (Figure 17, below).
Figure 17. Pie chart representing
teacher-participants’ responses to “what extent, if any, did your understanding
of the inquiry standards change as a result of your participation in this
seminar?” The scale for responses
included none, a small amount, a moderate amount, a large amount, and
completely.
All but one of the seminar-participants (13 of 14)
claimed that their understanding of the inquiry standards improved a large
amount to completely as a result of their
participation in the 3-day seminar (Figure 18, below). One of the participants claimed that her
understanding improved a moderate amount as a result of her participation.
Figure 18. Pie chart
representing teacher-participants’ responses to “what extent, if any, did your
understanding of the inquiry standards improve as a result of your
participation in this seminar?” The
scale for responses included none, a small amount, a moderate amount, a large
amount, and completely.
Figure 19 presents a scatterplot of the teacher-reported
increase in understanding of the inquiry standards plotted as a function of
change in understanding of the inquiry standards, both as a result of
participation in the seminar.
|
|
|
|
 |
|
|
|
Amount of Change in Understanding
|
|
Figure 19. Scatterplot of increase in understanding as a function of change in
understanding of the inquiry standards, both resulting from participation in
the seminar.
As can be seen in Figure 19 above, there was a high
positive statistically significant relationship between the amount of change
and the amount of increase in understanding of the science inquiry
standards. Approximately 61% of the
change in understanding of the inquiry standards is associated with the
increase in understanding of the inquiry standards.