Program Evaluation:  Scientific Research Through the RIP and Data Analysis for K-5 Teachers- Professional Development Program, University of Hawaii, Program conducted by ANOVA Science Education Corporation, 2002.  Evaluation submitted August 22, 2002.

 

The purpose of this graduate seminar was to introduce K-5 teachers to the teaching of science through true scientific inquiry.  Specifically, it was designed for teachers to explore the research investigation process; to use the inquiry process to learn how to design and conduct scientific research studies; to become familiar with techniques to assist in guiding students through the scientific inquiry process; to examine, practice, understand, and become competent in the ability to apply data analysis techniques to decision-making in science; and to increase confidence in using scientific research in their approach to instructing students in science and in addressing the scientific inquiry benchmarks and science inquiry content standards.  The research investigation process (RIP) was introduced and teachers were provided the opportunity to further develop their understanding of each of the elements of the RIP through their participation in and development of actual research investigations.  Techniques in data summary, analysis and presentation were explored in the context of hypothesis testing and decision-making in science.  All aspects of this seminar were aligned with the State of Hawaii Science Content and Performance Standards.

 

The data for this graduate seminar evaluation were obtained from assessments of the 14 teacher-participants at the beginning of (Pre-Assessment) and again at the end (Post-Assessment) of the 3-day seminar.  Items on the assessments required demonstration of knowledge about the scientific inquiry process, data analyses procedures, and decision-making in science.  A number of these items required teachers to demonstrate their knowledge through application.  Self-report items measured teacher confidence levels in understanding and using scientific inquiry in the classroom and in comprehending and applying the scientific inquiry content standards to their instruction.  A concept inventory determined teachers’ familiarity with and ability to teach elements of scientific inquiry and data summary and analysis techniques.  The pre-seminar and post-seminar assessment items were the same except for four additional self-report items included on the post-assessment.  These additional items assessed the teachers’ perceptions of how much their understanding of scientific inquiry changed and improved as a result of participation in the seminar.  The data were statistically analyzed using dependent t-tests to determine significant differences (indicating change) between pre- and post-assessment mean values.  The criterion for statistical significance (a) was set at 0.05.

 

 

Teacher Knowledge and Understanding of the Scientific Research Investigation Process (RIP), and Confidence in Teaching Scientific Inquiry

 

Seminar participants demonstrated a large, statistically significant increase in their knowledge and understanding of the individual elements of the RIP by the end of the 3-day seminar (Figure 1, below).  This included the logical order of the RIP elements, understanding of components involved in each element, and demonstration of the ability to construct testable hypotheses.

 

      Seminar Assessment

 

 

 

Mean (+SEM) RIP Score

 

 *

 

 

 

                                                       Figure 1.  Demonstration of knowledge and understanding of the elements of the RIP.

 

* Mean post-assessment score is significantly greater than mean pre-assessment score [t (13) = 9.67, p<0.001].

 

The post-seminar increase in teacher-participant knowledge and understanding of the research process was accompanied by a significant increase in teacher’ self-reported familiarity and understanding of concepts related to the scientific research process in the concepts inventory (Figure 2, below). By the end of the seminar, the average participant’ response rose from “familiarity with fair understanding of concept” to “very familiar with concept and could teach it to others.”  This showed that teachers recognized their increased knowledge and understanding.

 

 

 

 

 

Mean (+SEM) RIP Concept Inventory Score

 

      Seminar Assessment

 

 *

 

 

 

                                                      Figure 2.  Familiarity and understanding of concepts related to elements of the RIP.

 

 * Mean post-assessment score is significantly greater than mean pre-assessment score [t (13) = 6.48, p<0.001].

By the end of the 3-day seminar, participants’ self-reported confidence levels for their ability to use scientific inquiry, their understanding of teaching science through inquiry, and their ability to teach and engage students in scientific research activities all increased significantly compared to pre-seminar (Figures 3, 4 and 5, below).

 

 

 

 

  *

 
 


      Seminar Assessment

 

            Somewhat Confident

 

       Confident

 

     Completely Confident

 

      Very Confident

 

      Not at all Confident

 

Mean (+SEM) Confidence

Score

 

 

 

                                             Figure 3.  Self-reported confidence levels for ability to use scientific inquiry.

 

* Mean post-assessment score is significantly greater than mean pre- assessment score [t (13) = 11.08, p<0.001].

 

 

 

 

 

  *

 
 

 


      Seminar Assessment

 

            Somewhat Confident

 

      Very Confident

 

     Completely Confident

 

       Confident

 

      Not at all Confident

 

Mean (+SEM) Confidence

Score

 

 

 

                               Figure 4.  Self-reported confidence levels for understanding of teaching science through inquiry.

 

 

 

  *

 
* Mean post-assessment score is significantly greater than mean pre-assessment score [t (13) = 9.25, p<0.001].

           Seminar Assessment

 

Mean (+SEM) Confidence

Score

 

 

Figure 5.  Self-reported confidence levels for ability to teach and engage students in scientific research activities.

 

* Mean post-assessment score is significantly greater than mean pre-     assessment score [t (13) = 7.46, p<0.001].

 

 

 

Teacher Understanding of and Ability to Apply Data Summary, Presentation, and Analysis techniques to Decision-Making in Science

 

 

By the end of the seminar, participants demonstrated a large, statistically significant increase in their knowledge and ability to correctly organize data into a summary table and to construct a bar graph for comparing the central tendency of two groups of data   (Figure 6, below).

 

 

  *

 
 


 

 

Mean (+SEM) Data Summary & Presentation Score

 

      Seminar Assessment

 

 

                                         Figure 6.  Demonstration of understanding and ability to apply data organization and presentation techniques to data.

 

* Mean post-assessment score is significantly greater than mean pre-assessment score [t (13) = 4.07, p<0.001].

 

 

 

  *

 
Seminar participants also demonstrated a dramatic change in their knowledge and ability to apply data analysis techniques to research data.  Comparison of the pre-and post-assessments revealed that by the end of the seminar, participants significantly increased their understanding of how to calculate descriptive statistics and their ability to determine which measure of central tendency is most appropriate for a group of data (Figure 7, below).

 

 

Mean (+SEM) Data Analysis Score

 

      Seminar Assessment

 

 

 

 

Figure 7.  Demonstration of understanding the calculations for descriptive statistics and ability to determine the most appropriate statistic to represent central tendency for a group of data.

 

* Mean post-assessment score is significantly greater than mean pre-assessment score [t (13) = 5.99, p<0.001].

 

 

 

Teacher-participants demonstrated a statistically significant increase in their ability to interpret data presented in scatterplots and summarized in bar graphs by the end of the seminar (Figure 8, below).

 

 

 

Mean (+SEM) Graph Interpretation Score

 

            Seminar Assessment

 

  *

 

 

                                                  Figure 8.  Demonstration of ability to interpret scatterplots and bar graphs.

 

* Mean post-assessment score is significantly greater than mean pre-assessment score [t (13) = 4.01, p<0.001].

 

 

 

 

The post-seminar increase in teacher-participant knowledge of and ability to apply data presentation and analyses were accompanied by a significant increase in teacher’ self-reported familiarity and understanding of concepts related to data presentation and analysis in the concepts inventory (Figures 9 and 10, below).  By the end of the seminar, the average participant’ response for the three measures of central tendency rose significantly from “somewhat familiar with concept, but do not really understand what it means” to “very familiar with concept and could teach it to others” (Figure 9). 

 

 

 

 

 

 

Mean (+SEM) Central Tendency Concept Inventory Score

 

       Seminar Assessment

 

 *

 

 

 

 

                                            Figure 9.  Familiarity and understanding of concepts related to measuring central tendency.

 

 * Mean post-assessment score is significantly greater than mean pre-assessment score [t (13) = 7.95, p<0.001].

 

 

 

 

Similarly, the average participant’ response for tables and graphs rose significantly from “familiar with concept with a fair understanding of what it means” to “very familiar with concept, but would have some difficulty to teach it to others”  (Figure 10). 

 

 

 

Mean (+SEM) Tables & Graphs Concept Inventory Score

 

  *

 

       Seminar Assessment

 

 

                                        Figure 10.  Familiarity and understanding of concepts related to measuring central tendency.

 

*Mean post-assessment score is significantly greater than mean pre-assessment score [t (13) = 4.08, p<0.001].

 

 

 

 

Benchmarks and Standards

 

General teacher confidence in and awareness of ability in understanding and applying scientific inquiry to the teaching of science, and in ability to successfully address the scientific inquiry standards were also affected by the end of the seminar.  Teacher-participant self-reported confidence in ability to address content standards in the classroom rose significantly by the end of the seminar (Figure 11, below).

 

 

Mean (+SEM) Confidence

Score

 

      Seminar Assessment

 

 

  *

 

 

                            Figure 11.  Self-reported confidence levels for ability to address content standards in the classroom.

 

* Mean post-assessment score is significantly greater than mean pre-assessment score [t (13) = 6.12, p<0.001].

 

 

Similarly, by the end of the seminar, participant confidence about ability to accurately and completely address the scientific inquiry standards dramatically increased from “somewhat confident” to near “very confident” (Figure 12, below).

 

 

 

    *

 
 


Mean (+SEM) Confidence

Score

 

      Seminar Assessment

 

 

                       Figure 12.  Self-reported confidence levels for ability to accurately and completely address the scientific inquiry benchmarks.

 

* Mean post-assessment score is significantly greater than mean pre-assessment score [t (13) = 7.30, p<0.001].

 

 

Finally, by the end of the 3-day seminar, teachers significantly increased their familiarity and understanding of inquiry standards from being “somewhat familiar with this concept, but not really understanding what it means” to being “very familiar with this concept, but with some difficulty teaching it to others.”  This increase was statistically significant and was consistent with the increase in teacher-participant confidence regarding scientific inquiry and addressing the inquiry standards (Figure 13, below).

 

 

 

 

Mean (+SEM) Inquiry Standards Concept Inventory Score

 

       Seminar Assessment

 

  *

 

 

Figure 13.  Familiarity and understanding of concept of inquiry standards.

 

*Mean post-assessment score is significantly greater than mean pre-assessment score [t (13) = 5.14, p<0.001].

Teacher Perceptions of Impact from their Participation in this Graduate Seminar

 

The post-assessment contained four additional self-report items designed to assess how much teacher-participants believed their knowledge and abilities regarding the scientific research investigation process and scientific inquiry were impacted by their participation in this seminar.  The results from these items are presented in Figures 14-19 below.

 

An overwhelming majority of the seminar-participants (12 of 14) claimed that their understanding of the research investigation process was changed a large amount to completely as a result of their participation in this seminar, while two participants claimed it changed a moderate to a large amount (Figure 14, below).

 

 

n=12

85.7%

 

         n=2

         14.3%

 

 

Figure 14.  Pie chart representing teacher-participants’ responses to “what extent, if any, did your understanding of the research investigation process change as a result of your participation in this seminar?”  The scale for responses included none, a small amount, a moderate amount, a large amount, and completely.

 

 

More than two-thirds of the seminar-participants (11 of 14) claimed that their understanding of the research investigation process improved a large amount to completely as a result of their participation in the 3-day seminar (Figure 15, below).  Three of the participants claimed it improved a moderate to a large amount as a result of their participation.

 

n=11

78.6%

 

n=3 21.4%

 

 

Figure 15.  Pie chart representing teacher-participants’ responses to “what extent, if any, did your understanding of the research investigation process improve as a result of your participation in this seminar?”  The scale for responses included none, a small amount, a moderate amount, a large amount, and completely.

 

 

 

 

Figure 16 presents a scatterplot of the teacher-reported increase in understanding of the research investigation process plotted as a function of change in Text Box:  understanding of the research investigation process, both as a result of participation in the seminar.

 

 

Amount of

Increase in Understanding

 

Amount of Change in Understanding

 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 


Figure 16.  Scatterplot of increase in understanding as a function of change in understanding of the research investigation process, both resulting from participation in the seminar.

 

 

 

As can be seen in Figure 16 above, there was a high positive statistically significant relationship between the amount of change and the amount of increase in understanding of the scientific research investigation process.  Approximately 79% of the change in understanding is associated with the increase in understanding.

 

Almost 79 percent of the seminar-participants (11 of 14) claimed that their understanding of the inquiry standards was changed a large amount to completely as a result of their participation in this seminar, while the remaining three participants claimed it changed a moderate to a large amount (Figure 17, below).

 

 

n=11

78.6%

 

        n=3

        21.4%

 

 

 

Figure 17.  Pie chart representing teacher-participants’ responses to “what extent, if any, did your understanding of the inquiry standards change as a result of your participation in this seminar?”  The scale for responses included none, a small amount, a moderate amount, a large amount, and completely.

 

 

 

 

 

 

 

All but one of the seminar-participants (13 of 14) claimed that their understanding of the inquiry standards improved a large amount to completely as a result of their participation in the 3-day seminar (Figure 18, below).  One of the participants claimed that her understanding improved a moderate amount as a result of her participation.

n=1   7%

 

n=13

  93%

 

 

Figure 18.  Pie chart representing teacher-participants’ responses to “what extent, if any, did your understanding of the inquiry standards improve as a result of your participation in this seminar?”  The scale for responses included none, a small amount, a moderate amount, a large amount, and completely.

 

 

Figure 19 presents a scatterplot of the teacher-reported increase in understanding of the inquiry standards plotted as a function of change in understanding of the inquiry standards, both as a result of participation in the seminar.

 

Text Box:

 

Amount of Change in Understanding

 
 

 

 

 

 

 

 

 

 

 

 

 

 

 


Figure 19.  Scatterplot of increase in understanding as a function of change in understanding of the inquiry standards, both resulting from participation in the seminar.

 

 

As can be seen in Figure 19 above, there was a high positive statistically significant relationship between the amount of change and the amount of increase in understanding of the science inquiry standards.  Approximately 61% of the change in understanding of the inquiry standards is associated with the increase in understanding of the inquiry standards.