Psychology Textbook Part III Chapter 6 Lessons from Conducting an Equity-Focused, Participatory Needs Assessment

PART III PART THREE EVALUATION RESEARCH ?

refers to evaluation research , also known as program evaluation , as research purpose instead of a specific method . Evaluation research is the systematic assessment of the worth or merit of time , money , effort and resources spent in order to achieve a goal . Additionally , evaluation research is a type of applied research , and so it is intended to have some effect . Many methods like surveys and experiments can be used to do evaluation research . The process of evaluation research consisting of data analysis and reporting is a rigorous , systematic process that involves collecting data about organizations , processes , projects , services , or resources . Evaluation research enhances knowledge and , and leads to practical applications . The three case studies found in this section provide evaluation research for you to see how community psychology practitioners conduct this work in community settings . In Lessons from Conducting an , Participatory Needs Assessment , Brown et al . describe their process of engaging in collaborative work with the community in North Texas with a partnership that consisted of a community psychology practitioner , an academic partner , and other stakeholders . Program Evaluation A Fundamental Component in Effective Community Practice contributed by Patricia Connor , expands the traditional study format to include multiple studies from which lessons learned are highlighted through practice of community psychology ( In this study , practitioners and relevant stakeholders who work together to design and implement needed programs . Our third case story , Showing up and Standing with An Intersectional Approach to a Participatory Evaluation of a Housing First Program on , contributed by Anna takes us to the lovely ahu and captures the work conducted in an ongoing participatory evaluation partnership between Housing First program participants , staff , and community psychologist in the multicultural context of the Island of in Hawaii . Using an intersectional lens ( Crenshaw , 1989 Weber , 2009 ) this case study explores the challenges and successes of building this partnership among individuals from diverse racial and ethnic backgrounds with varying degrees of power , housing experiences , and mental and physical health issues .

PROGRAM EVALUATION A FUNDAMENTAL COMPONENT IN EFFECTIVE COMMUNITY PRACTICE Patricia . This case story expands the traditional study format to include multiple studies from which lessons learned are extracted through community psychology practice . The Big Picture . In ' Image IS the word strategy .

90 SEE CONTRIBUTORS PAGE FOR LIST OF AUTHORS ( EDITED BY PALMER , TODD ROGERS , JUDAH VIOLA , AND ) In this chapter I modify and expand the traditional study format to include multiple case studies from which I extract lessons learned through my practice of community psychology ( Program evaluation plays an important , structural role in its contributions to the assessment of the intervention work of change agents , here practitioners , and relevant stakeholders who work together to design and implement needed programs . My aim here and in all of my work is to encourage an evaluation mentality in agents . If these change agents develop interventions with an evaluation mentality , that is , with program evaluation as a core part of planning , design , and implementation , the resulting evaluative feedback can provide validation of the effectiveness of programmatic efforts , and thus , of change agents , or illustrate the need for substantive changes in aspects of the intervention efforts . The inclusion of evaluation strategies can assist program agents in identifying the critical elements to ensure meaningful interventions and to provide evidence of the viability of replication . Additionally , we must recognize both the values ( social justice , sense of community , empowerment , etc . which underlie the development of community practice interventions and the critically important role of a change agent who incorporates a program evaluation mentality into the design of those interventions . Program evaluation thus becomes an essential tool in the practice of . The overall aim of community psychologists work is the improvement of participants quality of life some examples include et al . 2020 ) Lin et al . 2020 ) Connor ( 2013 ) and Greenwood ( 2020 ) Stewart and ( 2020 ) and ( 2020 ) Improving quality may range from enhancing individuals sense of to ensuring needed supports some examples include et al . 2020 ) et al . 2020 ) et al . 2020 ) et al . 2017 ) and et al . 2020 ) However , confirming the value of such work or appropriately modifying it can only be accomplished through the inclusion of program evaluations . The essential questions for program or change agents are whether a proposed program is appropriate , whether the implemented program is as planned or how it has changed , and whether the program outcomes are as hoped for or as expected . Thus , developing and implementing interventions must be paired with evaluating the initial designs , or outcomes of those programmatic interventions , all of which can improve participants quality of life . My work and my career focus have been as a professor in a small college in upstate New York teaching a program evaluation course in a masters program , and as an evaluation consultant , engaging in numerous small , primarily local and large , and national program evaluations . From that work , I have identified seven lessons regarding the use of program evaluation strategies that are offered as guides to those in the practice of evaluating programs . I also provide three principles that serve as guides for program . The seven lessons , with illustrative studies , are based on two kinds of evaluation projects through my graduate program evaluation course and through my practice . The former evaluations emerge from a course requirement for students to participate in the design and implementation of a group evaluation project and the latter projects include my evaluations of specific programs or organizations . These studies , with their lessons learned immediately following , document that some efforts were successful and , not surprisingly , some were not .

CASE STUDIES IN COMMUNITY PSYCHOLOGY PRACTICE A GLOBAL LENS Study One The Executive Director ( ED ) of a human services agency that provides residential treatment for adolescents was interested in employees perceptions of their work environments . The ED , a manager , and an assistant met with the graduate students and me to discuss the ED purpose for the evaluation to learn how to make the organization a best place to work . We agreed that interviews with workers would be the most appropriate way to collect data as Wage by there could be with . questions . The manager would provide access to workers . The meeting ended quite satisfactorily , with a potential schedule for the next steps , and the ED , manager , and assistant left . Students walking down the hallway after the meeting overheard the manager say to the assistant , They should just do their jobs ! illustrating that the manager had little interest in soliciting feedback from employees . This attitude was in the difficulty students had throughout the project , first in getting access to employees , and second , in having employees agree to be interviewed . Some employees expressed concern that their interview information might not remain anonymous the manager was not considered trustworthy . Students completed interviews but fewer than expected and with less useful information than planned or expected . Seven overarching lessons learned are depicted in tables below . Success of an evaluation project relies on the effective participation of all levels of stakeholders , not just the leader or person seeking the evaluation . it . from . Image shows people ( clip art ) sitting Study Two A local County Commissioner of Mental Health was interested in whether children visits to an Emergency Department ( ED ) could have been avoided , particularly among those children receiving assistance from the County . The question was whether the children were in contact with any service agencies and whether that contact should have resulted in interventions that would have precluded the ED visit . Students in the graduate Program Evaluation class and I met with the Commissioner and formulated a plan for record reviews of intake information at the ED of the local hospital . Although students were reviewing unredacted records , Institutional Review Board ( approval was not considered necessary unpublished program evaluations do not require review . Students presented the results to the Commissioner who was able to work with the Department of Social Services to develop preventive interventions to reduce children unnecessary use of the ED . The success of this project resulted in the Commissioner taking the methodology to the relevant State Offices and the data collection strategy was replicated in nine counties . Three factors led to

92 SEE CONTRIBUTORS PAGE FOR LIST OF AUTHORS ( EDITED BY PALMER , TODD ROGERS , JUDAH VIOLA , AND ) this the quality of the project , the Commissioner appreciation of , and reliance on relevant data , and the Commissioner interest in expanding the use of relevant and usable information as a foundation for . The of methodology and the usability of results rely on the involvement of appropriate . A local prevention program focused on specific issues related to illegal substance use among younger people as required by their funding sources . Through my ongoing relationship with the program director as the programs evaluation consultant , we conducted multiple evaluations , including focus with key leaders in the community , an assessment of a specific neighborhood , and surveys with a summer leadership program for high school students , among several other projects over a period of approximately six years . Below are examples of successful and of those evaluations . Study Three In one particularly effective evaluation , students conducted an observational assessment of a neighborhood to identify both assets ( open stores , schools , churches , shops , etc . and liabilities ( closed stores , vacant houses , empty lots with trash ) Although the student evaluation groups typically included only five or six participants , this group comprised 14 very dedicated students , divided into seven pairs for the observations and interviews . The paired students divided the neighborhood into approximately equal manageable areas , did the observations in pairs , and conducted interviews ( with a structured interview developed during class time ) to obtain residents perceptions of the neighborhood . The collected information enabled the program director to develop strategies to advocate for neighborhood improvements and to identify specific locations for program development . The degree of determined and dedicated student involvement led to the clear success of this evaluation effort . Study Four In another evaluation , the program director of the same substance abuse prevention program requested that students conduct interviews with people presumed to be key stakeholders to obtain their feedback on the program . Working with the program director , students identified approximately 40 locally based , potential stakeholders , including religious leaders , politicians , educators , local business owners , and others . The project itself was built on the expectation that people in the community would be familiar with , if not involved in , the work of the project . However , these , all of whom the students contacted directly , were not sufficiently knowledgeable about , or in some cases , invested in the work of the program to participate in the interview process , resulting in inadequate numbers of completed interviews and thus , inadequate feedback regarding program implementation . Here the lack of success seems tied to the lack of interest or commitment on the part of the external stakeholders , most of who did not view themselves as stakeholders at all .

CASE STUDIES IN COMMUNITY PSYCHOLOGY PRACTICE A GLOBAL LENS 93 Study Five To evaluate a summer leadership program for high school students offered by the same substance abuse prevention program , the graduate students in the program evaluation course and I met with the program coordinator to identify the aims and activities of the program which would enable the students to develop . The coordinator who reported to the program director did not seem particularly interested in any kind of evaluation . After the initial meeting , the students were virtually unable to connect with the program coordinator who simply did not respond to emails or phone calls . The students , under my direction , finally developed a draft survey to enable some completion of the project before the end of the semester . The lack of success here the lack of commitment on the part of the internal stakeholder . Study Six In working with a program director in an agency that provides support to underserved , generally homeless , people , I suggested conducting a focus group with people who were receiving services to solicit their input in developing strategies to address their needs , which could result in modifications of existing programs . The program director asked approximately six or seven people to participate and four arrived at the designated time . Transportation costs in the form of bus passes and given gift certificates to a local chain were offered to encourage participation and to compensate for their time . However , the focus group did not achieve the expected outcome in that all participants had extensive experiences with such programs and were familiar with the kinds of questions that might be asked and with the range of what they perceived that agencies might consider acceptable responses . Thus , the circumstances under which the focus group was conducted , that is , in the agency itself with a peer as a , led to repetitions of stories and statements which only affirmed what was already happening , rather than suggestions for novel approaches to addressing the needs of program participants . Here the previous experiences of the participants framed and even limited the range of their contributions . The successes of projects rely on the community psychologists and community stakeholder commitment or interest in the program . Study Seven As director of a graduate program in Community Psychology , have consistently encouraged designed and implemented process and outcome evaluations of the program itself and of other offices at the College , for example , access to registration and financial aid offices , availability of library , and separately , food services .

94 SEE CONTRIBUTORS PAGE FOR LIST OF AUTHORS ( EDITED BY PALMER , TODD ROGERS , JUDAH VIOLA , AND ) evaluations provided useful and useable information , students preferences regarding the timing of classes evenings the development of a student organization organized , then dissolved the availability of advisors more needed the helpfulness of field placement coordinators helpful the employment outcomes of graduates particularly useful for current students for example , add , modify , or eliminate courses , increase field placement experiences , add electives , etc . and numerous other projects The above table provides information on the results of a evaluation including students preference , the availability of advisers , and suggestions for program improvements . suggestions for program improvement Participation in these kinds of evaluations provided the students with meaningful , experiences with the process of evaluation and with the programmatic commitment to assessing the usefulness and value of one work . Only one among numerous program evaluations yielded a particularly negative response when asked the reason for not continuing in the program , one person responded , I hate the program director who happens to be the author ! Engaging in the evaluation of one own program plays a critical role in establishing the legitimacy of program evaluation for internal and external audiences , including those in the program . Several of my evaluation experiences have reinforced the importance of effective process evaluations , particularly of observation . Three studies below illustrate that importance , two from one setting and the third from another setting which is described below . The first setting was a program that comprised artists collaborating with teachers in the delivery of primary school curricula . Storytellers emphasize the logical progression of a story ( beginning , middle , end ) for kindergartners and and math operations ( addition and subtraction ) for second and third graders dancers express the meaning of words in movement ( lean forward , then backward for wax and wane or ebb and ) This collaboration can result in improved grades for students , which can be documented over time through appropriate outcome measures , for example , quarterly grades compared with the previous year , or compared with another unit . However , the actual viability and of the program will depend on two factors first , the support of the classroom teachers through their involvement in the collaborative process , and second , the actual use of the arts by the artists . Study Eight An illustration of the first factor , support of the classroom teacher , was my effort to observe both the collaboration and the actual artist presence in the classroom with at least two observations of each teacher . One second grade teacher was determinedly not interested in

CASE STUDIES IN COMMUNITY PSYCHOLOGY PRACTICE A GLOBAL LENS 95 participating in any aspect of the process , though expected to do so by the principal the teacher even stated to me , You can do your little dance program here in the classroom but I am not going to be That teacher retired at the end of that school year . Most other Observations were conducted with the enthusiastic involvement of teachers and artists . One other significant observation was of the planning meetings to select the curriculum for the artistic mode of delivery . After the first year of the program , the planning meetings became more about setting up the calendar than about modifying or expanding the content and mode of the artists delivery of the curriculum . That focus , on the calendar rather than content , reflected the decreasing commitment of the teachers to effective participation in the process of collaboration . Study Nine An illustration of the second factor , actual use of the arts by the artist , an effective musician collaborated with a teacher in the delivery of a poetry unit with the expectation that the artist would use music to demonstrate the role of rhythm in poetry . In the observed classroom experience , the artist used her own skills in to deliver the lesson rather than her musical talent and musical instruments . The outcome of improved grades for the students was , in fact , related to the skill of the artist as a poet than to the artist as a musician in the delivery of the curriculum . This effectively precluded the presumed of the collaboration . Although such a conclusion would not have been drawn without the evaluator observation of the classroom exercise , there were also numerous observations of the effective and appropriate implementation of the collaboration as designed . Study Ten An entirely different example the importance of observation in an entirely SEE different setting , a national organization with a focus on a specific medical condition . The organization had developed an extensive image decorative only and says related to this study curriculum , a set Of nine chapters With Used an methodology accompanying slides , for medical professionals to bring current , information to those with the condition and to inform the general public about the condition . The aim of the evaluation was the assessment of the effectiveness and usefulness of this standardized curriculum . As the evaluator , I included observation of each of the three planned of the curriculum , one in a rural setting with people with the condition , one in a university with providers , caretakers , and people with the condition , and a third in an urban setting with providers and caretakers , primarily parents and family members of children with the condition . The observation revealed that the actual use of the curriculum varied widely across the three settings . The in the rural setting discussed the first several chapters the multiple presenters in the university setting each reviewed their own areas of expertise without reference to the curriculum , and the presenters in the primarily family setting focused on one chapter in the curriculum which did not overlap at all with the rural presentation . Participants in each setting completed surveys which demonstrated some increase in knowledge about , and understanding of , the condition across the three settings but clearly

96 SEE CONTRIBUTORS PAGE FOR LIST OF AUTHORS ( EDITED BY PALMER , TODD ROGERS , JUDAH VIOLA , AND ) those improvements were not related to the actual use of the curriculum . Again , the importance of observation is demonstrated in that the conclusion could only have emerged through my evaluator observation of each implementation . Program evaluations can identify successful programs which can be replicated however , such programs require careful analyses , typically through observation , to ensure that the implementation processes are . Study Eleven In the a local philanthropic foundation began to support collaborations through , and I applied for and received one of the first . Upon the completion of that grant , was subsequently approached to collaborate with a variety of programs and agencies over a period of years . These included focus groups with elderly residents of a public housing project to assess their satisfaction ( which impacted planned renovations of the housing project ) and observations of an advisory board for a child sexual abuse intervention program to identify strategies to enhance the Executive Directors success with the Advisory Board ( one obstreperous person resigned the Chair reorganized meeting structure ) The success of each led to my being contacted by subsequent community agencies and programs to participate in a joint submission to the funding source as the value and usefulness of engaging in evaluation activities became more evident . Here the overall success emerged out of my previous experiences and my local reputation . Study Twelve As part of an overall assessment , another national foundation with a focus on individuals was interested in whether programs which they funded were using strategies that matched the vision and mission of the national organization and whether the were resulting in the desired outcomes . Most local program directors were understandably proud of their own efforts , the extent of local participation , and the outcomes of the programs . As the evaluator for the national organization , I undertook the task of assessing six of the local programs ( somewhat randomly selected ) to identify both aspects that were congruent with the national organizations goals and objectives and those that needed modification to increase their rates of success . These program directors were willing to participate in the evaluation activities but were also accustomed to receiving only praise for their efforts in initiating and managing their programs . My evaluation reports for each program documented their successes but also included recommendations for improvement . The reports were not well received directors who had welcomed me , participated actively in the evaluation activities and seemed to accept and even welcome verbal recommendations at the end of each visit , did not appreciate having any of what they perceived as less than positive results in a written report . The outcome was the termination of the entire evaluation project .

CASE STUDIES IN COMMUNITY PSYCHOLOGY PRACTICE A GLOBAL LENS 97 The value of program evaluations is learned primarily through the experience of having results that easily lead to program improvements , which highlight the usefulness of conducting assessments . Study Thirteen In one New York evaluation , six counties were selected to participate in a public health intervention and were asked to different their own program designs in their efforts to achieve the desired public health outcome . At the end of the evaluation period , some strategies were clearly more effective than others which led to the adoption , or at least the encouragement of the adoption , of those strategies . As the evaluator I had assured each of the participating counties that their results would be anonymous , that is , the State as the funding source would not know which counties were successful and which were not . The need for that promise of anonymity was essential because the local staff was concerned that future funding could be affected by the State staff knowledge of specific outcomes . At the end of the project , with positive results clearly disseminated , the State staff requested rather strongly that the anonymity be unveiled so that the successful and not so successful counties be identified . I refused , based on the ethics of my adhering to my promise . That ethical decision led to the termination of that relationship ! Study Fourteen Another instance of ethical difficulties was in the final evaluation report of , project which required collaboration across multiple human service agencies . Funding for year two was based on the viability of the project and commitment of the agencies , both of which were to be documented in the evaluation report . A new project manager , who started just weeks before the report was due , requested changes in the report which would enhance the appearance of a positive outcome for year two , but which somewhat misrepresented the actual data . Discussion ensued resulting in the project manager asserting her position as manager and me asserting my role as evaluator with my intention to adhere to the independence of the evaluation process and outcome . The awkwardness of the situation for me resulted in my submitting only a hard copy ( in the days before electronic submissions ) on the day the report was submitted , the manager interest in , and possibility of making changes in the report . Adhering to ethical boundaries can be difficult in some circumstances . Conclusion This was a series of studies to illustrate factors that can affect the critical role that program evaluation plays in the practice of community psychology . Each of these factors emerges from my , experiences of working with

98 SEE CONTRIBUTORS PAGE FOR LIST OF AUTHORS ( EDITED BY PALMER , TODD ROGERS , JUDAH VIOLA , AND ) based agencies and programs that rely on county , state , or federal funding , that is , public monies , or on local , or national foundations for private money . Extracted from these seven lessons learned are three principles that have emerged as guides for my work in the field of program evaluation . They are ( ensuring that the evaluation results are useful , making sure the evaluation is simple and doable , and ( evaluation efforts are congruent with the program efforts . The figure below highlights these three principles Ensure that the evaluation results are useful to the program managers for whom they are being prepared and that the results are in a format that is usable forthe program managers . only understand charts , then provide charts if they only understand text , then provide text . Make the evaluation simple and , simple enough for the program managers to understand the what is being done and why is it being done which will ensure its usefulness and usability . Further , make sure all aspects ofa planned evaluation can be accomplished its implementation . There will always be changes that are needed but having a fully planned framework makes those changes easier and more straightforward . Evaluation efforts should be congruent with the program efforts on such factors as program complexity , length , and cost . Simple programs do not require complex evaluations , long require appropriately long evaluations , and inexpensive programs do not need expensive evaluations . The above highlights the three principles shown in text above and says the same thing . Finally , program evaluation serves as a critical part of the practice of community psychology , providing essential information for funding sources , and crucial feedback for those aiming to improve individuals quality of life and . Those of us who practice in the community most assuredly value the and , at the same time , the idiosyncrasies of that work , as in the seven takeaways and the illustrations of each . Those who are change agents or also intuitively or actually know the value of building assessments or measures into their change efforts from the beginning to identify both areas in need of improvement and areas of success . Using appropriate program evaluation strategies based on the three principles cited above will enhance the efficacy of interventions . From Theory to Practice Reflections and Questions

CASE STUDIES IN COMMUNITY PSYCHOLOGY PRACTICE A GLOBAL LENS 99 Program evaluation plays an important , structural role in its contributions to the assessment of the intervention work of change agents , where community psychology practitioners and relevant stakeholders work together to design and implement needed programs ( 2021 ) What lens might a community psychologist bring to the table in a program evaluation ?

What lens would another psychologist bring ( social psychologist or clinical psychologist ) Describe why it is important when conducting program evaluations to analyze the data collected from an ecological level . What conceptions did you hold prior to reading this case story about program evaluations References , 2020 ) Building communities in tense times Fostering connectedness between cultures and generations through community arts . American journal of Community Psychology , 65 , Roy , Fiesta , 2020 ) Safe spaces embedded in dangerous How Chicago youth navigate daily life and demonstrate resilience in neighborhoods . American journal of Community Psychology , 66 , Hess , Greene , 2020 ) Randomized controlled trial of a multilevel intervention to address social of refugee mental health . American journal of Community Psychology , 65 , Lin , 2020 ) The impact of comprehensive community initiatives on child , youth , and family outcomes A systematic review . American journal of Community Psychology , 65 , 2020 ) for depression and anxiety in women . Archives of Iranian Medicine , 23 ( Connor , 2013 ) From isolation to community Exploratory study of a of Community Psychology , 41 ( Greenwood , 2020 ) Empowering features and outcomes of homeless interventions A systematic review and narrative synthesis . American journal of Community Psychology , 66 , Ma , Law , Zhao , 2017 ) Evaluation of a positive youth development program or adolescents with greater psychosocial needs Views of the program participants . International journal and Human Development , 16 ( DOI Stewart , 2020 ) How far have we come ?

An integrative review of the current literature on sense of community and . American journal of Community Psychology , 66 , 2020 ) Meaningful engagement in research Community residents as of knowledge . American journal of Community Psychology , 66 , i ' ones , 2020 ) Reducing the biological and psychological toxicity of

100 SEE PAGE FOR OF AUTHORS ( BY PALMER , TODD ROGERS , JUDAH , AND ) related stress Initial efficacy of the intervention of early adolescents . American journal of Community Psychology , 65 ,