Minnesota Campus Compact is committed to deepening the evaluation and assessment of civic engagement’s outcomes for students, institutions, and communities. This section provides resources related to our collaborative work around assessing the effectiveness of civic engagement.
Upcoming civic learning rubric assessment tools. Created by the Massachusetts Department of Higher Education, these two assessment rubrics measure civic values and civic knowledge and are under consideration by the AAC&U for inclusion in its suite of assessment tools. The civic knowledge rubric will be available for testing on February 29th, and can be found at the link above.
The Global Learning Assessment Opportunity: the Global Engagement Survey (GES). The GES is looking for institutions to participate in the survey, which assess global learning outcomes by examining institutions and their summer and spring programming. Apply to participate by March 15, 2016.
MNCC Assessment Publications
From Activities to Benefits – This report reflects on the 2009 Civic Engagement Forums that Minnesota Campus Compact convened with the Minnesota Evaluation Association. We aimed to bring out multiple perspectives on what civic engagement outcomes matter, to identify models and resources, and to generate new ideas for implementing and learning from evaluation.
Guiding Principles for Evaluation
The Joint Committee on Standards for Educational Evaluation recommends four basic attributes (and numerous sub-standards) for any program evaluation:
- Utility, including identification of stakeholders, credibility of evaluators, pertinence of information, and clarity and timeliness of reporting.
- Feasibility, including practicality of procedures, political viability, and cost effectiveness.
- Propriety, including service to participants, community, and society, respect for the rights of those involved, and provisions for complete and fair assessment.
- Accuracy, including program documentation, use of valid and reliable procedures, appropriate analysis, impartial reporting, and justified conclusions.
Palomba, Catherine A., and Trudy W. Banta, Assessment Essentials: Planning, Implementing, and Improving Assessment in Higher Education (San Francisco: Jossey-Bass Publishers, 1999).
Eight additional principles that were developed in relation to student success, but are relevant in general:
- Context is everything.
- The whole is greater than the sum of the parts.
- Evidence is essential: the more, the better.
- Test prevailing assumptions.
- Cast a wide net.
- Use outsiders to ask hard questions.
- Focus on what matters to student success.
- Stay the course.
George D. Kuh et al., Assessing Conditions to Enhance Educational Effectiveness: The Inventory for Student Engagement and Success (San Francisco: Jossey-Bass, 2005), pp. 10-16.
Finally, a common acronym to help people remember what makes a good evaluation objective is SMART:
- Specific (how does the objective link the behavior and an anticipated numeric or descriptive result?)
- Measurable (is there a reliable system in place to illustrate achievement of the objective?)
- Achievable (can the objective be achieved?)
- Relevant (are the people within the objective in a position to make an impact on the situation?)
- Time-based (is there a clearly defined start and end for the objective?)
A Glossary of Terms
Evaluation: the systematic process of determining the merit, value, and worth of someone or something
Assessment: the process of measuring, quantifying, and/or describing results related to the attributes covered by the evaluation
Formative Evaluation: intended to help “form” (develop, change, or improve) a program, conducted before or during implementation of the program to allow feedback to be acted upon
Summative Evaluation: intended to judge a program’s effectiveness and demonstrate accountability, conducted after the program has ended or has operated for a significant period of time so the actual outputs and outcomes can be compared to the original objectives; may also be used to form the program if the program will continue or be offered again
Goal: intended results in general terms (e.g., an institutional culture that is supportive of civic engagement)
Objective: intended results in precise terms, identifying specific policies, behaviors, etc. (e.g., the institution’s strategic plan includes civic engagement as an explicit priority)
Measure/Indicator: a specific standard or criterion by which something can be judged (e.g., students graduate with a stronger commitment to ongoing civic engagement)
Benchmark: a piece of evidence to which later results/performance can be compared (e.g., the number of service-learning courses offered in a particular year)
Output: activities done to achieve intended results (e.g., a two-day orientation and ongoing mentoring were offered to first-generation college students)
Outcome: meaningful change, the intended results of activities/programs/partnerships (e.g., retention rates among first-generation college students went up)
Tool/Instrument: a device used to collect data, information, or evidence (e.g., tests, questionnaires, interview schedules, checklists, rating scales, observation records)
Audit/Inventory: documentation of what is already going on and where there might be additional interest (e.g., a survey of campus faculty and staff about their civic engagement activities)
Quantitative Methods: techniques that generate numerical scores, ratings, or findings (e.g., survey questions answered yes/no or on a scale)
Qualitative Methods: techniques that yield descriptive information, stories, and narrative analysis (e.g., open-ended questions on surveys, journals, ethnographic field studies, participant observation)
Direct Measures: techniques where knowledge, skills, or other desired outcomes are demonstrated in response to the instrument or in their own existence (e.g., tests in which students show what they have learned, official tenure and promotion policy guidelines that explicitly recognize engaged scholarship)
Indirect Measures: techniques where people reflect on outcomes (e.g., surveys asking students to report what they have learned, interviews asking faculty their perceptions of how engaged scholarship is recognized in tenure and promotion)
Levels of Analysis
Micro: changes in individuals or their circumstances
Macro: changes in broader structures that impact people’s lives
Meta: evaluation of the evaluation
For more on the language of research see the following useful article at Journalist’s Resource: Statistical terms used in research studies; a primer for media
Select Evaluation Resources
Service-Learning/Civic Engagement Related Tools
- American Democracy Project – Individual and Campus Assessment Tools Campus: http://www.aascu.org/WorkArea/DownloadAsset.aspx?id=7590 Individual:http://www.aascu.org/WorkArea/DownloadAsset.aspx?id=7589
- Assessing Civic Competency and Engagement in Higher Education: Research Background, Frameworks, and Directions for Next-Generation Assessment. By Judith Torney Purta, Julio C.Cabrera, Katrina Crotts Roohr, Ou Lydia Liu, and Joseph A.Rios. This ETS research report provides a detailed review of existing frameworks and programs for civic engagement in higher education.
- Assess Your Work: Essentials of Community Engagement – From: Nexus Community Partners
- Association of American Colleges and Universities – VALUE rubrics http://www.aacu.org/value
- CIRCLE Working Paper #78, Building an Evidence-Based Practice of Action Civics: The Current State of Assessments and Recommendations for the Future, by Jennifer Gingold – includes tools in the appendixes http://www.civicyouth.org/wp-content/uploads/2013/08/WP_78_Gingold.pdf
- Civic-Minded Graduate scale and rubric – Developed by the Center for Service & Learning at IUPUI, this model identifies specific knowledge, skills, and dispositions to describe how students develop civic-mindedness. http://csl.iupui.edu/teaching-research/opportunities/civic-learning/graduate.shtml (For an example of a MNCC member institution using this scale in its civic engagement assessment work, see http://www.smsu.edu/campuslife/civicengagement/?id=2989.)
- Community Campus Partnerships for Health – Methods and Strategies for Community Partner Assessment http://depts.washington.edu/ccph/pdf_files/tools-partner.pdf
- How to Read and Use Research to Make Your Job Easier – A Guide for Educators and Practitioners Who May Be New To Research, from CIRCLE – http://www.civicyouth.org/wp-content/uploads/2007/07/v4.i3.6.pdf
- Imagining America’s Assessing the Practices of Public Scholarship research team shares documents on its own work as well as links to other resources, including a few on story-based documentation and video documentation: http://imaginingamerica.org/research/assessment/
- IUPUI list of accountability tools for civic engagement: http://csl.iupui.edu/assessment/accountabilityToolsforCivicEngagement.cfm
- National Institute for Learning Outcomes Assessment – NILOA assists institutions and others in discovering and adopting promising practices in the assessment of college student learning outcomes: http://www.learningoutcomeassessment.org/
- National Service-learning Clearing House: Evaluation & Assessment, Links and Tools: https://gsn.nylc.org/clearinghouse/results?key=&type=&attrs=20
- Partnerships for Success Partnership Rubric – This rubric from Rhode Island Campus Compact, is a tool designed to support institutional, organizational, and programmatic partnership development and assessment. http://ricompact.org/wp-content/uploads/2014/10/PFS_PartnershipRubric.pdf For more information, use this link http://ricompact.org/programs/partnerships-for-success/
- Service-Learning Research Primer (Steinberg, Bringle, & Williams) – This 2010 publication covers service-learning research design, measurement, ethical issues, data analysis, and dissemination.
- University of Minnesota Post Service-Learning Experience Survey
- Minnesota Compass – data, trends, ideas and analysis for Minnesota organizations http://www.mncompass.org/
- Minnesota Council for Quality http://www.councilforquality.com/
- Minnesota Evaluation Association www.mneval.org/
- American Evaluation Association www.eval.org
- Council for the Advancement of Standards in Higher Education https://www.cas.edu/index.html
- Association for Institutional Research—Development of student service-learning course survey http://airweb.org/page.asp?page=731
- Community Campus Partnerships for Health, Tools & Resources https://ccph.memberclicks.net/resources
- Colorado State University – Writing Guides: Conducting Survey Researchhttp://writing.colostate.edu/guides/research/survey/
- Wisconsin Center for Education Research Field-Tested Learning Assessment Guide (FLAG) Attitude surveys http://www.wcer.wisc.edu/archive/cl1/flag/cat/catframe.htm
- Fanning, E. (2005). Formatting a paper-based survey questionnaire. Practical Assessment, Research & Evaluation, Vol.10(12). http://pareonline.net/pdf/v10n12.pdf
- Frary, Robert B. (1996). Hints for designing effective questionnaires. Practical Assessment, Research & Evaluation, 5(3). http://pareonline.net/getvn.asp?v=5&n=3
Free Online Survey Tools
- Google Docs Forms http://www.google.com/google-d-s/forms/ Free, easy to use – view and download results in a spreadsheet
- Lime Survey http://www.limesurvey.org/ A robust open source survey program for advanced users, which includes survey logic and analysis features; it requires some technology knowledge, but a growing support community is available to assist you.
Some popular sites with limited free basic services and some fee-based advanced features:
- Kiwi Surveys http://kwiksurveys.com/
- PollDaddy http://www.polldaddy.com
- SurveyGizmo http://www.surveygizmo.com
- SurveyMonkey http://www.surveymonkey.com
- Zoomerang http://www.zoomerang.com
Focus Group Resources
- Free Management Library – Basics of Conducting Focus Groups http://www.managementhelp.org/evaluatn/focusgrp.htm
- ERIC/AE Staff (1997). Designing structured interviews for educational research. Practical Assessment, Research & Evaluation,5(12). http://pareonline.net/getvn.asp?v=5&n=12
Other Resources and Articles
- Campus Climate Survey on Personal and Social Responsibility (AAC&U, in collaboration with the Research Institute for Studies in Education at Iowa State University) http://www.aacu.org/core_commitments/publications.cfm
- Center for Civic Partnerships – logic model tips and resources from Public Health Institute’s Center for Civic Partnerships http://www.civicpartnerships.org/docs/tools_resources/Logic%20Models%209.07.htm
- Center for What Works – benchmarking for nonprofits http://www.urban.org/policy-centers/cross-center-initiatives/performance-management-measurement/projects/nonprofit-organizations/projects-focused-nonprofit-organizations/outcome-indicators-project
- Checklist for Building Organizational Evaluation Capacity by Boris B. Volkov and Jean A. King (2007) https://www.wmich.edu/sites/default/files/attachments/u350/2014/organiziationevalcapacity.pdf
- Civic-Minded Graduate: A North Star, by Kathryn S. Steinberg, Julie A. Hatcher, and Robert G. Bringle, Michigan Journal of Community Service Learning, Fall 2011, pp. 19-33
- Collective Impact Forum – A place for those practicing collective impact to find the tools, resources, and advice they need, and to share experience and knowledge to accelerate the effectiveness and adoption of collective impact. http://www.collectiveimpactforum.org/
- Data + Design – A simple introduction to preparing and visualizing information from Infoactive and the Donald W. Reynolds Journalism Institute. https://infoactive.co/data-design/titlepage01.html
- Educators’ Guide to Service-Learning Program Evaluation – this toolkit walks readers step-by-step through a program assessment process http://www.livingknowledge.org/fileadmin/Dateien-Living-Knowledge/Dokumente_Dateien/Toolbox/LK_E_EvaluationToolkit.pdf, by Steven Teles and Mark Schmitt, Stanford Social Innovation Review
- Evaluation Plan Guidance – “A step-by-step guide to designing a rigorous evaluation” from Social Innovation Fund and the Corporation for National & Community Service http://www.nationalservice.gov/sites/default/files/documents/SIF%20Evaluation%20guidance%208%205%202014.pdf
- Evaluating Community Change – from Grantmakers for Effective Organizations, and written from a funder’s prospective
- Grantmakers for Effective Organizations Resource Library – A limited selection of GEO’s publications and other resources are available to the public.http://www.geofunders.org/resource-library
- Guide to Actionable Measurement, from the Gates Foundation http://www.gatesfoundation.org/learning/Pages/a-guide-to-actionable-measurement.aspx
- Impact, Influence, Leverage, and Learning (I2L2) Outcomes Framework – This working paper provides sample indicators for various social impact outcomes, and the ORSImpact.com website includes other resources, such as a guide to developing an outcome map. http://orsimpact.com/resource-download/?resource_id=756
- Information is Beautiful – see data through various visualizations http://www.informationisbeautiful.net
- Methods and Strategies for Institutional Assessment, From: Shinnamon A, Gelmon S and Holland B., Methods and Strategies for Assessing Service-Learning in the Health Professions. Available from CCPH.
- Michael Quinn Patton’s Utilization-Focused Evaluation Checklist: https://www.wmich.edu/sites/default/files/attachments/u350/2014/UFE_checklist_2013.pdf
- Monitoring & Evaluation News, including participatory techniques such as Most Significant Change (MSC) HERE
- National Science Foundation 2002 User-Friendly Handbook for Project Evaluation www.nsf.gov/pubs/2002/nsf02057/start.htm
- Online Evaluation Resource Library – supported by the Division of Research, Evaluation and Communication, Directorate for Education and Human Resources, National Science Foundation http://oerl.sri.com/
- Resources for Evaluating Community Organizing from Alliance for Justice http://bolderadvocacy.org/tools-for-effective-advocacy/overview-of-evaluating-community-organizing/reco
- Socrative; an app that allows for an instructor or supervisor to gain almost immediate feedback from students
- SROI Act II: A Call to Action for Next Generation SROI by Cynthia Gair – understanding social return on investment, from REDF: https://ccednet-rcdec.ca/files/ccednet/pdfs/2009-REDF_SROI_Act2.pdf
- State of Evaluation: Evaluation Practice and Capacity in the Nonprofit Sector (October 2010), a report from the Innovation Network: http://www.innonet.org/client_docs/innonet-state-of-evaluation-2010.pdf
- The State of Learning Analytics in 2012: A Review and Future Challenges, by Rebecca Ferguson: http://kmi.open.ac.uk/publications/pdf/kmi-12-01.pdf
- Statistics Help: nilesonline.com/stats/
- Success Metrics: interesting conversations on the Social Edge sitehttp://www.socialedge.org/discussions/success-metrics
- Technology and the Completion Agenda: brief essay on the intersection of education, assessment and the tech world
- W.K. Kellogg Foundation Evaluation Handbook: www.wkkf.org/default.aspx?tabid=75&CID=281&NID=61&LanguageID=0
- The Worth of What They Do – a study from Repair the World, examining the effects of alternative breaks and other short-term volunteer projects on communities in need in the U.S. and abroad: http://werepair.org/short-term-volunteering-can-have-long-term-positive-effects-on-communities/