Assignment Five--Part One

Assignment Five--Part One

Short Survey of the Weight-Loss Support Group in Rural Saskatchewan

Original Version

November 20, 2009

Group Membership Survey



Janelle R. Christensen
University of Saskatchewan







In Partial Fulfilment of the Course:

ECUR 809.3 (01): Introduction to Program Evaluation

November 12, 2009

Introduction:

On the following four pages, there are several questions that ask you about you and your membership in the #### group to which you belong. The purpose of the survey is to obtain an overview of your perceptions of what it is like to belong to the group that you do. Your opinions will be used to help your group as a whole as you assist each other on your weight-loss journey. Please answer all questions as best you can. Only the program evaluator, namely Janelle Christensen, will see the results of the individual surveys. However, a summary of the results will be provided to your group once the course for which Janelle is completing this assignment has finished in December 2009.


How long have you been a member of ####? ______years and/or ______months



What was your main reason for joining #### when you initially joined the group?




What did you know about #### before becoming a member?




What kind of roles have you played in the last 3 months and before then in ####? (Circle YES or NO in each column for each item).


Past 3 Months
Before Then


a) Attend weekly meetings regularly

YES
NO
YES
NO
b) Talk at meetings (make comments, express ideas, etc.)

YES
NO
YES
NO
c) Help organize activities (other than meetings)

YES
NO
YES
NO
d) Participate in ####-related activities outside of meetings

YES
NO
YES
NO
e) Planned and lead one or more programs for a meeting

YES
NO
YES
NO

Group Functioning

1. What a group accomplishes is often dependent upon how the group functions. Think about the way your #### group works and how effective you think the group is in the following ways. Please rate each item from 1 (Low) to 5 (High) by circling the number that best describes your position. Circle U if you are uncertain or can’t answer.

Effectiveness of the group in:

Low Moderate High Uncertain

a) Listening to everyone’s perspective 1 2 3 4 5 U

b) Creating mutual respect, 1 2 3 4 5 U
understanding, and trust within the
group

c) Building a clear purpose (members 1 2 3 4 5 U
know what the purpose of the group is)

d) Providing effective leadership 1 2 3 4 5 U

e) Conducting meetings that accomplish 1 2 3 4 5 U
what is necessary

f) Carrying out planned actions 1 2 3 4 5 U

g) Orienting new members 1 2 3 4 5 U

h) Celebrating progress 1 2 3 4 5 U

i) Identifying and using members’ 1 2 3 4 5 U
skills and talents

j) Making decisions accepted by all 1 2 3 4 5 U
members

2. Please indicate your perceptions of the group using a 5-point scale from 1 (Infrequently) to 5 (All the time). Circle the number which best describes your position.

Infrequently Sometimes All the Time

a) My viewpoint is heard 1 2 3 4 5

b) I am viewed as a valued 1 2 3 4 5
member

c) I feel comfortable in the 1 2 3 4 5
group

d) I am satisfied with the 1 2 3 4 5
group’s progress




3. What is the most significant impact that membership in this group has had for you?










4. In your opinion, what could be done to improve the group’s effectiveness?
















Impact/Influence on Group Members

1. Your membership and participation in #### may have influenced your personal knowledge, beliefs, or skills. Please indicate this group’s IMPACT INFLUENCE ON YOU by rating each item below on a scale from 1 (low impact/influence) to 5 (high impact/influence. Circle the number which best describes your position. If you are uncertain/unsure about any of these effects, please circle U.

To what extent does the #### group to which you belong have an IMPACT/INFLUENCE on you with respect to:

Low Moderate High Uncertain

a) Your knowledge of portion control 1 2 3 4 5 U

b) Your understanding of serving sizes 1 2 3 4 5 U

c) Your understanding of others’ 1 2 3 4 5 U
perspectives

d) Your understanding of the 1 2 3 4 5 U
importance of support in the group

e) Your ability to read and understand 1 2 3 4 5 U
food labels

f) Your ability to monitor your own 1 2 3 4 5 U
behaviour, including food choices
and physical activity

g) Your knowledge of stress 1 2 3 4 5 U
management strategies and techniques



2. What is the greatest impact that being part of this specific #### has had upon YOU as an individual?







Thank you for your participation.

Assignment Three

Program Evaluation Assessment
Of a
Non-Profit Weight-Loss Group
In
Rural Saskatchewan
Janelle R. Christensen
University of Saskatchewan

As a member of a non-profit weight-loss group in rural Saskatchewan, it has been brought to this program evaluator’s attention that the current leader of this program has some concerns regarding the efficacy of her work with respect to assisting all members in their weight-loss journey. In order to determine the overall scope and design of an evaluation plan, one must first decide whether an evaluation needs to be conducted. In order to make this decision, an evaluation assessment, or a needs assessment must be carried out. The purpose of this assignment is to carefully examine the components of a specific weight-loss group in rural Saskatchewan in order to best decide whether or not an evaluation should occur.

Nature of the Program

This program “is a non-profit, non-commercial weight-loss support organization” (About #### Club Inc.) based in the United States. Its objective is to “encourage healthy lifestyles through weight management support groups and to sponsor obesity research” (About #### Club Inc.). This non-profit weight-loss group meets weekly during the evening. Like all other chapters of this group, “a private weigh-in is followed by a program on a wide variety of topics pertaining to the weight-loss journey and healthy lifestyle (About #### Club Inc.). The goal of the meetings is to provide “members with positive reinforcement and motivation in adhering to their food and exercise plans” (About #### Club Inc.). There are many resources provided to members, such as *** ***** ** **** that contains information about the exchange system for meal planning and motivational tips and practical advice and the companion workbook. There are some strategies used in order to attempt to motivate members, such as contests in which members compete with each other and the winner(s) are awarded non-monetary prizes. Following numerous consultations with the major stakeholders of this group, specifically the group members, including the Leader and the Co-Leader who are also members, it was determined that there were many perceived objectives of this group that were not being met. In addition, it was also questioned as to whether or not the mandate of the overall weight-loss group was being met. Specifically, it was questioned whether or not the program was doing what it has set out to do. Therefore, a needs assessment will be carried out in order to determine if a formal evaluation of this program needs to occur.

Needs Assessment

Step One

In order to conduct a needs assessment of this group, this internal evaluator has chosen to follow some steps outlined by Rouda and Kusy (1995). In particular, it is suggested to do an initial “gap analysis” (Rouda and Kusy, 1995). Specifically, the current skills, abilities, and knowledge of the major stakeholders can be assessed, and compared to the desired skill levels and knowledge base of all stakeholders. It will be these differences, or the “gap” between the current and the desired that will determine the nature and direction of this evaluator’s future evaluation. Therefore, in respect to the specific weight-loss group, the current skills and abilities of the members can be assessed by inquiring about their daily practices and efforts to eat healthier and become more physically active. These abilities have been assessed by asking participants to indicate their knowledge of various issues related to weight loss and changes in eating habits, exercise levels, and stress management. In order to obtain some evidence of these abilities, a Needs Assessment Questionnaire, as found in Appendix A, was designed by this internal evaluator and was administered to the members during a regular meeting. Members worked in small groups and discussed the questions and provided responses to them. Prior to administering this survey, the internal evaluator provided instructions to the group. In addition, members were assured of anonymity and confidentiality as they were asked not to identify themselves in any way on the questionnaire, including omitting their names from the page, and that the information they provided on the questionnaires would only be seen by the internal evaluator. It was initially desired to ask members to verbally respond to several questions in the context of our regular meeting. The questions were: (a) Do you leave each weekly meeting with the tools that you think you need to be successful, (b) In what ways are these tools provided , (c) In what way are members encouraged to succeed, (d) What occurs during each meeting that helps each person to become motivated to succeed, (e) Are members encouraged to use the various resources that are available to them on the website, (f) Are members made aware of these specific resources during the meetings, and (g) Are any of these resources integrated into the ‘program’ component of the meetings? However, due to time constraints during the meeting time, these questions were not asked.

The desired skill levels of all major stakeholders were also addressed by questions raised in the group sessions. For example, the group’s Leader desires to know what she can do to be of assistance to all members. In addition, members were also asked what information they would like to have brought forward that they believe would help to enhance their knowledge base that would assist them upon their weight-loss journey. A variety of responses were provided, thus warranting a need for further investigation. Admittedly, one cannot ignore the fact that weight loss and improvements to one’s own overall health and well-being are individual efforts. However, the perceived role of the group is to provide support and encouragement through the various activities so that each individual is inspired to set out on their own weight-loss journey each week. Therefore, it must be determined if this role is being fulfilled, and if so, if it is being done so in the most effective way possible.

Step Two

The next step in a needs analysis, as identified by Rouda and Kusy (1995), is to identify priorities based upon the issues that were raised in the first step. The authors identified numerous ways of doing so, such as conducting a cost-benefit analysis. Specifically, this evaluator can assess the ways in which the cost of the problem compares to the cost of implementing a solution. In this case, the cost of implementing a solution is much less than the cost of the problem. It must be noted that the word “cost” means much more than dollars and cents. Specifically, the implementation of a solution may mean that the group’s leader will engage in activities that will enhance her knowledge base of ways in which she can best foster motivation amongst the members, as well as provide tools that will educate members with new strategies for weight-loss and different options for becoming more physically active. The cost of the problem extends far beyond this particular small group. Furthermore, the cost to society in general of being overweight or obese is multi-dimensional, and is immeasurable. In particular, if the Leader does not have the knowledge base and the ability to provide members with information pertaining to healthy life-style choices, and to assist members when they feel helpless and/or have exhausted all options pertaining to healthy food choices and means of becoming physically active, then some individuals may become discouraged, leave the group because they do not receive the support they need, and the group process may eventually disintegrate.

However, one another approach of identifying priorities that the authors did not address, is to simply visually examine the responses provided by the group members on the Needs Assessment Questionnaire, thematically categorize them, and prioritize those general themes. The content of the program evaluation, specifically the survey or focus group questions, may be based upon these general themes.

Step Three

In order to determine some actual causes of performance problems and some possible opportunities for change (Rouda & Kusy, 1995), the internal evaluator looked at the responses of the final two questions on the Needs Assessment Questionnaire. Prior to administering the survey, it was anticipated that the questions posed to the group members would determine whether they believed that the leader and co-leader are effectively fulfilling the duties of their roles and meeting the goals and objectives of the organization. Interestingly, it was the leader who first expressed concern about this issue. Thus, one area of interest of the future evaluation will be to determine if the Leader and/or Co-leader are fulfilling the duties of their respective roles and what do members believe are some specific areas of improvement.

Step Four

The final step in Rouda and Kusy’s (1995) model is to identify possible solutions and growth opportunities. There were very several suggestions for improvement for change that were brought forth. It appears that members felt very comfortable sharing with one another in their small groups, and providing written responses to questions on the questionnaire. It is anticipated that the verbal responses to the additional questions, had they been posed, would have been much more parsimonious than the written responses on the questionnaire. One possible reason for this is that members would not feel comfortable sharing this information with the group when the Leader and Co-Leader are present. However, one way for the Leader and Co-Leader to enhance their knowledge base is to attend general meetings and conferences, and to engage in as many professional development opportunities as possible. If a program evaluation is conducted, it will address whether this is a practice that the Leader and Co-Leader engage in and the degree to which they deem that it is effective.

Conclusion

In September, 2009, the Leader of the non-profit weight-loss group in rural Saskatchewan approached this evaluator, who is also a member of the group, and expressed concerns about the efficacy of her leadership and the degree to which the meetings were beneficial to members. At that same time, this internal evaluator was cognizant of the fact that an upcoming assignment for her graduate class involved conducting an evaluation assessment of a program, and discussed the possibility of performing such an assessment and a possible evaluation plan. The Leader was delighted at this possibility, as she has many concerns, such as whether the practices that have been implemented to assist and encourage individuals as they learn to make healthier food choices, get more physical activity, and learn how to achieve a healthier lifestyle and an improved quality of life are actually meeting the objectives of the program? Following the evaluation assessment that occurred within the context of a regular meeting of this non-profit weight-loss group in rural Saskatchewan, it has been determined that a formal evaluation of this program must be conducted. Therefore, a formal evaluation plan will be designed within the next six weeks.


References

About #### Club Inc. (n.d.). Retrieved 09 29, 2009, from http://www.####.org/About ####.aspx
Rouda, R. a. (1995). Needs Assessment--THe First Step. Retrieved October 12, 2009, from http://alumnus.calech.edu/~rouda/T2_NA.html

List of Appendixes
Appendix A: Needs Assessment Questionnaire

Appendix A: Needs Assessment Questionnaire
******####—$$$$$$$ !!!!!

1. What are the three aspects of the group that you like the most?



2. What are the three aspects of the group that you dislike the most?


3. What are some things that you know NOW about weight loss and healthy lifestyle habits that you did not know when you joined ####? (example: food choices, types of physical activity, stress management techniques) Please be as specific as you can.





4. What are some things about weight loss and healthy lifestyle habits that you would like to know? Please be as specific as you can. (Examples: see above!)





5. What activities would you like to see incorporated into our meetings that you feel would help you achieve your weight-loss goals? Examples: discussion about specific topics (foods, behaviour changes, types of physical activity); cooking demonstrations; exercise demonstrations?

Assignment Two: Simulated Program Case Study--ECS Programming For Children With Severe Disabilities

In order to evaluate a program for children with severe disabilities, an evaluator may choose to plan his/her evaluation upon the tenets of one of many theoretical models of program evaluation. The model that is chosen must reflect the theoretical undertones of the program goals and objectives. In the case of the Student Services Department of the Medicine Hat Catholic Separate Regional Division, the primary program objective must be focused upon the educational, behavioral, and emotional well-being of the students as they are the major stakeholders in the program. In theory, a model or approach like that of Provus' that involves examining the content, processes, and products (outcomes) will be most effective to evaluate the efficacy of the means of bringing about results while considering the perspectives of all the major stakeholders of the program. However, if there is limited time, and/or financial resources and/or human resources. then the evaluation team may need to focus upon one segment of the program to carry out the evaluation in the most efficient way possible. For the purpose of this assignment, however, the financial and human resources will be unlimited.

A model of program evaluation that focuses on program content, process, and product (outcome) is the most appropriate approach to evaluate this specific program because the content is an inherent component of the process, which cannot be separated from the product. As the program evaluator, I am also concerned with outcomes, such as whether the education system was putting forth the right program to bring about benefits to the students who are in the program. However, I am also concerned with the content and processes, because these must be in place in order to bring about the desired outcomes.

It appears that there are many indicators or processes that can have an effect on student success. A component of the evaluation must look at the specifics of this program to determine whether or not the current practices are the most effective ways to help them to meet objectives outlined on their Individual Program Plans (IPPs). There are many questions that could be answered when addressing the specific processes of the program. In particular: (a) What qualifications and/or previous experience must the teachers have to develop IPPs, (b) What are the qualifications of the individuals delivering services within the preschools, kindergartens, and daycares, (c) Will the documentation of the child's current level of functioning in the learning environment be completed by one individual or by everyone who works with the child, and (d) What is the nature of the child's support system--at home, at school, and/or in daycare? These are important questions to answer in order to properly assess whether or not the program is being implemented by those properly trained to do so, and that some standards for service delivery are being met.

Another component of the evaluation would be one that focuses on the products of program implementation (outcomes). It is through an evaluation of these outcomes that everyone involved with the planning and implementation of the program becomes more aware of the current practices that are highly effective and those that are only moderately effective and/or ineffective at bringing about change for the students. Three questions that could be addressed include: (a) Are the students who are placed in the within-center and/or the in-home based program learning new information, including skills to help them meet their learning and behavioral objectives as outlined on their IPPs, (b) How will these differences be measured, and by whom, and (c) Are these methods of bringing about behavioral change (outcomes) the most cost-effective ways of doing so? When it is discovered that the students are demonstrating some type of change and are meeting and/or exceeding the objectives, but the school division is facing budget restraints, I would like to know what it is about the in-home program that is beneficial for students, and furthermore if there is some aspect of the in-home program that could be implemented more efficiently without compromising any parts of the program for the students.

In conclusion, it is evident that a thorough evaluation of this program for students with disabilities must involve the examination of its content; the processes involved in its delivery, such as the reactions of teachers, parents, students, and administrators; and the outcome of the program. The effective delivery of this program involves the implementation of appropriate methods by appropriately trained individuals to individuals who are deemed to most need the services. It is for this reason that an evaluation of the program must not ignore any of the core components of its effective delivery.

Assignment One

Evaluation of the Comprehensive Guidance and Counseling Program in Utah

The Utah State Office of Education regularly conducts a formal evaluation of its Comprehensive Counseling and Guidance Program (CCGP). Specifically, a program evaluation occurs during the self-review, and the subsequent enhancement of the program occurs continuously as counsellors compare the existing program with new program goals. This model of school guidance and counselling bares a marked resemblance to a program of a similar name used in elementary and secondary schools in Alberta. The use of this model has also been promoted in rural and urban schools in Saskatchewan, as teachers and counsellors are formally instructed about the nature of this model and the means of implementing it.

At the outset, it is challenging to discern the formal model of evaluation that was utilized. However, much of the aspects of Provus' Discrepancy Model are seen in the methods used to conduct this review, yet the theoretical foundations of the evaluation appear to be similar to those of Scriven's model. In particular, the rationale for an evaluation of the program is based upon results, or student performance outcomes. However, unlike Scriven's model in which an 'outsider' conducts the evaluation, the summative evaluation of the counselling and guidance program is Utah focuses upon the perspectives and outcomes of the major stakeholders--students, teachers, parents/guardians, and administrators.

In respect to Scriven's model, the "D" or the Design of the evaluation includes a Performance Self-Evaluation completed on-line, annually by each member of the CCGP team. The "P" or evaluation Process includes an on-site review, conducted every three years. This review utilizes student performance outcomes in all components of the model (School Guidance Curriculum, Individual Student Planning, Responsive Services, and System Support) to determine the degree to which the standards or objectives of the model were being met. The on-site team at each school consists of a district counselling director, two or three school counsellors, and a building administrator. However, elementary school counsellors were exempt from participating in some of the review process because it was thought that not all students in these grades may be receiving guidance services or be involved with programs that are components of the CCGP. Instead, guidance and counselling personnel who work with students in the elementary grades were asked to complete the Self-Evaluation as a means of self-assessment that was meeting the standards of the CCGP model. However, they were not required to discuss the results of their Self-Assessment during the formal review process. The utility of the Product, "P", may be assessed by examining the ways in which students' academic, social, and psychological well-being has changed/improved since after receiving guidance services.

It is somewhat ironic that school counsellors working with elementary-aged students would be exempt from some of this process. One of the most fundamental aspects of the CCGP is that all students, from Kindergarten to grade twelve receive some guidance services. In addition, this guidance model advocates the importance of accountability. Therefore, the decision on the part of the program evaluation design team to excuse elementary school counsellors from some of the review process may serve to undermine the importance of providing guidance services to these students, and the subsequent accountability of teachers and counsellors to the integrity of the program. In addition, it was requested to make "student, parent, and teacher involvement brief" during the on-site review process. However, this practice appears to be detrimental to the sustainability of the program because the focus of the review, student performance outcome, cannot be accurately assessed without addressing the views of all stakeholders. Although the Student Outcomes Needs Assessment is completed by students, parents/guardians, teachers, and administrators, it can be questioned as to whether this means of data collection is sufficient. Participants were asked to respond to statements regarding their perspectives in the domains of academics, life/career development, multicultural/global citizen development, and personal/social development. Respondents used a Likert-type scale, where "1" meant "being not important/not effective," and "5" meant "being very important/very effective." Admittedly, this was a very large-scale evaluation, and using a variety of data collection methods of obtaining information must not be underestimated. In particular, it may have been valuable for the researchers to conduct on-line follow-up interviews, requesting written responses to questions that emerged from the analysis of the quantitative study.

Resource

Utah State Office of Education. January, 2008. Comprehensive Counseling and Guidance Program. As found at: http://www.schools.utah.gov/cte/guidance_review.html.

Hi everyone!!

It was great to meet you today--looking forward to an exciting term!!!

Janelle