In order to evaluate a program for children with severe disabilities, an evaluator may choose to plan his/her evaluation upon the tenets of one of many theoretical models of program evaluation. The model that is chosen must reflect the theoretical undertones of the program goals and objectives. In the case of the Student Services Department of the Medicine Hat Catholic Separate Regional Division, the primary program objective must be focused upon the educational, behavioral, and emotional well-being of the students as they are the major stakeholders in the program. In theory, a model or approach like that of Provus' that involves examining the content, processes, and products (outcomes) will be most effective to evaluate the efficacy of the means of bringing about results while considering the perspectives of all the major stakeholders of the program. However, if there is limited time, and/or financial resources and/or human resources. then the evaluation team may need to focus upon one segment of the program to carry out the evaluation in the most efficient way possible. For the purpose of this assignment, however, the financial and human resources will be unlimited.
A model of program evaluation that focuses on program content, process, and product (outcome) is the most appropriate approach to evaluate this specific program because the content is an inherent component of the process, which cannot be separated from the product. As the program evaluator, I am also concerned with outcomes, such as whether the education system was putting forth the right program to bring about benefits to the students who are in the program. However, I am also concerned with the content and processes, because these must be in place in order to bring about the desired outcomes.
It appears that there are many indicators or processes that can have an effect on student success. A component of the evaluation must look at the specifics of this program to determine whether or not the current practices are the most effective ways to help them to meet objectives outlined on their Individual Program Plans (IPPs). There are many questions that could be answered when addressing the specific processes of the program. In particular: (a) What qualifications and/or previous experience must the teachers have to develop IPPs, (b) What are the qualifications of the individuals delivering services within the preschools, kindergartens, and daycares, (c) Will the documentation of the child's current level of functioning in the learning environment be completed by one individual or by everyone who works with the child, and (d) What is the nature of the child's support system--at home, at school, and/or in daycare? These are important questions to answer in order to properly assess whether or not the program is being implemented by those properly trained to do so, and that some standards for service delivery are being met.
Another component of the evaluation would be one that focuses on the products of program implementation (outcomes). It is through an evaluation of these outcomes that everyone involved with the planning and implementation of the program becomes more aware of the current practices that are highly effective and those that are only moderately effective and/or ineffective at bringing about change for the students. Three questions that could be addressed include: (a) Are the students who are placed in the within-center and/or the in-home based program learning new information, including skills to help them meet their learning and behavioral objectives as outlined on their IPPs, (b) How will these differences be measured, and by whom, and (c) Are these methods of bringing about behavioral change (outcomes) the most cost-effective ways of doing so? When it is discovered that the students are demonstrating some type of change and are meeting and/or exceeding the objectives, but the school division is facing budget restraints, I would like to know what it is about the in-home program that is beneficial for students, and furthermore if there is some aspect of the in-home program that could be implemented more efficiently without compromising any parts of the program for the students.
In conclusion, it is evident that a thorough evaluation of this program for students with disabilities must involve the examination of its content; the processes involved in its delivery, such as the reactions of teachers, parents, students, and administrators; and the outcome of the program. The effective delivery of this program involves the implementation of appropriate methods by appropriately trained individuals to individuals who are deemed to most need the services. It is for this reason that an evaluation of the program must not ignore any of the core components of its effective delivery.
Saturday, September 19, 2009
Saturday, September 12, 2009
Assignment One
Evaluation of the Comprehensive Guidance and Counseling Program in Utah
The Utah State Office of Education regularly conducts a formal evaluation of its Comprehensive Counseling and Guidance Program (CCGP). Specifically, a program evaluation occurs during the self-review, and the subsequent enhancement of the program occurs continuously as counsellors compare the existing program with new program goals. This model of school guidance and counselling bares a marked resemblance to a program of a similar name used in elementary and secondary schools in Alberta. The use of this model has also been promoted in rural and urban schools in Saskatchewan, as teachers and counsellors are formally instructed about the nature of this model and the means of implementing it.
At the outset, it is challenging to discern the formal model of evaluation that was utilized. However, much of the aspects of Provus' Discrepancy Model are seen in the methods used to conduct this review, yet the theoretical foundations of the evaluation appear to be similar to those of Scriven's model. In particular, the rationale for an evaluation of the program is based upon results, or student performance outcomes. However, unlike Scriven's model in which an 'outsider' conducts the evaluation, the summative evaluation of the counselling and guidance program is Utah focuses upon the perspectives and outcomes of the major stakeholders--students, teachers, parents/guardians, and administrators.
In respect to Scriven's model, the "D" or the Design of the evaluation includes a Performance Self-Evaluation completed on-line, annually by each member of the CCGP team. The "P" or evaluation Process includes an on-site review, conducted every three years. This review utilizes student performance outcomes in all components of the model (School Guidance Curriculum, Individual Student Planning, Responsive Services, and System Support) to determine the degree to which the standards or objectives of the model were being met. The on-site team at each school consists of a district counselling director, two or three school counsellors, and a building administrator. However, elementary school counsellors were exempt from participating in some of the review process because it was thought that not all students in these grades may be receiving guidance services or be involved with programs that are components of the CCGP. Instead, guidance and counselling personnel who work with students in the elementary grades were asked to complete the Self-Evaluation as a means of self-assessment that was meeting the standards of the CCGP model. However, they were not required to discuss the results of their Self-Assessment during the formal review process. The utility of the Product, "P", may be assessed by examining the ways in which students' academic, social, and psychological well-being has changed/improved since after receiving guidance services.
It is somewhat ironic that school counsellors working with elementary-aged students would be exempt from some of this process. One of the most fundamental aspects of the CCGP is that all students, from Kindergarten to grade twelve receive some guidance services. In addition, this guidance model advocates the importance of accountability. Therefore, the decision on the part of the program evaluation design team to excuse elementary school counsellors from some of the review process may serve to undermine the importance of providing guidance services to these students, and the subsequent accountability of teachers and counsellors to the integrity of the program. In addition, it was requested to make "student, parent, and teacher involvement brief" during the on-site review process. However, this practice appears to be detrimental to the sustainability of the program because the focus of the review, student performance outcome, cannot be accurately assessed without addressing the views of all stakeholders. Although the Student Outcomes Needs Assessment is completed by students, parents/guardians, teachers, and administrators, it can be questioned as to whether this means of data collection is sufficient. Participants were asked to respond to statements regarding their perspectives in the domains of academics, life/career development, multicultural/global citizen development, and personal/social development. Respondents used a Likert-type scale, where "1" meant "being not important/not effective," and "5" meant "being very important/very effective." Admittedly, this was a very large-scale evaluation, and using a variety of data collection methods of obtaining information must not be underestimated. In particular, it may have been valuable for the researchers to conduct on-line follow-up interviews, requesting written responses to questions that emerged from the analysis of the quantitative study.
The Utah State Office of Education regularly conducts a formal evaluation of its Comprehensive Counseling and Guidance Program (CCGP). Specifically, a program evaluation occurs during the self-review, and the subsequent enhancement of the program occurs continuously as counsellors compare the existing program with new program goals. This model of school guidance and counselling bares a marked resemblance to a program of a similar name used in elementary and secondary schools in Alberta. The use of this model has also been promoted in rural and urban schools in Saskatchewan, as teachers and counsellors are formally instructed about the nature of this model and the means of implementing it.
At the outset, it is challenging to discern the formal model of evaluation that was utilized. However, much of the aspects of Provus' Discrepancy Model are seen in the methods used to conduct this review, yet the theoretical foundations of the evaluation appear to be similar to those of Scriven's model. In particular, the rationale for an evaluation of the program is based upon results, or student performance outcomes. However, unlike Scriven's model in which an 'outsider' conducts the evaluation, the summative evaluation of the counselling and guidance program is Utah focuses upon the perspectives and outcomes of the major stakeholders--students, teachers, parents/guardians, and administrators.
In respect to Scriven's model, the "D" or the Design of the evaluation includes a Performance Self-Evaluation completed on-line, annually by each member of the CCGP team. The "P" or evaluation Process includes an on-site review, conducted every three years. This review utilizes student performance outcomes in all components of the model (School Guidance Curriculum, Individual Student Planning, Responsive Services, and System Support) to determine the degree to which the standards or objectives of the model were being met. The on-site team at each school consists of a district counselling director, two or three school counsellors, and a building administrator. However, elementary school counsellors were exempt from participating in some of the review process because it was thought that not all students in these grades may be receiving guidance services or be involved with programs that are components of the CCGP. Instead, guidance and counselling personnel who work with students in the elementary grades were asked to complete the Self-Evaluation as a means of self-assessment that was meeting the standards of the CCGP model. However, they were not required to discuss the results of their Self-Assessment during the formal review process. The utility of the Product, "P", may be assessed by examining the ways in which students' academic, social, and psychological well-being has changed/improved since after receiving guidance services.
It is somewhat ironic that school counsellors working with elementary-aged students would be exempt from some of this process. One of the most fundamental aspects of the CCGP is that all students, from Kindergarten to grade twelve receive some guidance services. In addition, this guidance model advocates the importance of accountability. Therefore, the decision on the part of the program evaluation design team to excuse elementary school counsellors from some of the review process may serve to undermine the importance of providing guidance services to these students, and the subsequent accountability of teachers and counsellors to the integrity of the program. In addition, it was requested to make "student, parent, and teacher involvement brief" during the on-site review process. However, this practice appears to be detrimental to the sustainability of the program because the focus of the review, student performance outcome, cannot be accurately assessed without addressing the views of all stakeholders. Although the Student Outcomes Needs Assessment is completed by students, parents/guardians, teachers, and administrators, it can be questioned as to whether this means of data collection is sufficient. Participants were asked to respond to statements regarding their perspectives in the domains of academics, life/career development, multicultural/global citizen development, and personal/social development. Respondents used a Likert-type scale, where "1" meant "being not important/not effective," and "5" meant "being very important/very effective." Admittedly, this was a very large-scale evaluation, and using a variety of data collection methods of obtaining information must not be underestimated. In particular, it may have been valuable for the researchers to conduct on-line follow-up interviews, requesting written responses to questions that emerged from the analysis of the quantitative study.
Resource
Utah State Office of Education. January, 2008. Comprehensive Counseling and Guidance Program. As found at: http://www.schools.utah.gov/cte/guidance_review.html.
Subscribe to:
Posts (Atom)