Part 1: Learning Outcomes
Main Function
Learning outcomes should exist to communicate what a student is expected to know or do after successfully completing an activity, course, program, degree, or other academic credential.
Key Points to Know
- Outcomes should focus on the major learning that shows what students should be able to do or remember a few years after successfully completing an academic experience.
- Outcomes should communicate these learning expectations to students, faculty, and the public.
- Outcomes should describe actions that students can achieve in a way that faculty can measure.
Benefits
Good learning outcomes
- provide clear instructional goals for faculty members, helping them focus curriculum on content and concepts that are most important.
- present clear expectations for the students; and
- position institutions to be accountable for student learning and learning improvement.
Why Learning Outcomes are Necessary
An assessment model grounded in learning outcomes also supports the College’s mission to provide “student-centered teaching and learning experiences.” Learning outcomes keep the College focused on student learning and allow the College to remain in compliance with its regional accreditation responsibilities. This handbook includes information related to the focus on student learning.
Core Questions
Learning outcomes and their assessment answer two important questions: (1) What do we want students to learn? and (2) How do we know they have learned it? These questions are relevant at the institutional, program, and course levels. The answers may seem obvious to some, but the answers are not always apparent within the curriculum.
Terms such as goals, objectives, competencies, and proficiencies are too often used interchangeably with the term outcomes. These terms do not always have the same meaning. Specifically, the term ‘outcome’ focuses on what the student will do, while the term objective traditionally indicates what an instructor will do. So, we begin by defining learning outcomes. In short, well-written learning outcomes are statements that clearly articulate what students are expected to be able to do after they successfully complete an activity, course, program, or degree.
At the institution or program level, learning outcomes are usually expressed broadly or in general terms. Institutional learning outcomes and program learning outcomes are examples of broad learning outcomes. Learning outcomes at the course level are more specific. In all cases, learning outcomes communicate value to students and the public, and these expectations for student performance provide a framework that allows faculty to build the curriculum.
The declared, taught, and learned curriculum types should, in theory, be the same, but without outcomes and assessment, there is no way to know if they are the same or not. Well-written outcomes help align the curriculum we declare and teach, and well-written outcomes provide the foundation for both the expectations of student learning and good assessment. Outcomes and assessment with a focus on student learning assist instructors and the institution in demonstrating that learning has occurred measurably. This is a documented best practice that informs good teaching.
Faculty must know that Palm Beach State College has learning outcomes at three levels: institutional learning outcomes, program learning outcomes, and course learning outcomes. Outcomes are developed by faculty and are published online. Web links are provided with brief descriptions. All levels of student learning outcomes can be found at
www.pbsc.edu/LearningOutcomes
Program Learning Outcomes (PLOs)
Developing program learning outcomes from the existing objectives in the state curriculum frameworks at the program level (A.S., ATD, CCC, ATC, and CCP). Outcomes-based learning continues to be an ongoing practice that now includes bachelor’s degree programs. Outcomes at this level are reviewed annually during program review, and faculty go through the curriculum committee to have the changes approved when there is a need to revise PLOs.
www.pbsc.edu/ProgramLearningOutcomes
Course Learning Outcomes (CLOs)
Course learning outcomes are aligned to the PLOS and are included in each syllabus. Faculty must go through a curriculum process to revise learning outcomes, and revisions must be approved by the cluster and curriculum committee. Course learning outcomes are created by using the State of Florida‘s assigned frameworks at the onset of a program, as they do with the PLOs. The College’s assessment director is available to assist faculty with outcomes development when needed. www.palmbeachstate.edu/utilities/CourseOutlines/
Benefits of a Curriculum Driven by Learning Outcomes
Learning outcomes provide an opportunity for faculty to evaluate course and program offerings in terms of student learning. Faculty can make a difference in the learning experiences of students at Palm Beach State by collaborating with each other to define clear expectations for learning that can then be communicated to students. The implementation of learning outcomes has been and continues to be a transformative experience of how we examine students and learning as the focus has turned to what students learn and how we can improve student learning. The process is also evolutionary, and we are constantly learning. The key is that a partnership in learning develops – students know what they will be able to do as the result of the learning, and faculty will have the tools to ensure that students are learning the stated outcomes. By focusing on learning outcomes, this partnership has other benefits for both the faculty and students.
Outcomes-based Curriculum Allows Faculty to…
- Know exactly what students are expected to learn in each course and know the recommend outcomes for programs and courses.
- Provide focus for developing appropriate learning experiences for students so that they have the knowledge, skills, and abilities to succeed.
- Empower students to become more involved with their learning experiences.
- Assess students’ learning and use results as a tool for improvement.
- Grow professionally as they step away from traditional teaching formats and try innovative pedagogies to get students more involved in the learning process.
Outcomes-based Curriculum Allow Students to…
- Know exactly what is expected of them.
- Become more involved in their learning experiences.
- Apply knowledge, skills, and abilities from one class to the next or to the workplace.
Developing Learning Outcomes
Begin by asking, “What will the students be able to do upon successful completion (of the degree, the program, or the course)?” Consider in the answer the knowledge, skills, and abilities you might expect a student to have years after successfully completing your course or program. There may be several outcomes, and there is no magic number of outcomes for an institution, a program, or a course. What is important is that the essential components of learning are represented. Once a desired outcome is identified, it must be developed and written.
Author Ruth Stiehl (2017) provides an excellent four-part backward-design model for developing learning outcomes. Her model requires faculty to collaboratively consider and find consensus regarding the concepts and issues that students must understand and address, the skills students must master, the assessments that students must take to demonstrate mastery in class, and finally, what students can do after they finish the course or program. The result of this process is an outcome for the course, program, or institution.
Once the intended outcome is identified, it must be carefully written. Good outcomes clarify expectations of the faculty member to the student and ultimately the public, so ensuring the outcome is in fact “good” becomes important.
Several qualities characterize a good learning outcome, including these.
- Good learning outcomes include action verbs.
- Good learning outcomes clearly state who is to do the action.
- Good learning outcomes clearly state what action is to be done.
- Good learning outcomes are achievable.
- Good learning outcomes are observable.
Assessing the Quality of Learning Outcomes
Previously, it was suggested that all outcomes be achievable, observable, measurable, and that they are aligned to the curriculum. Additionally, each outcome should include action verbs, clearly state who is to do the action, and clearly state what action is to be done.
As a final check, it is important to assess the quality of learning outcomes at all levels by alignment and actual data obtained through the PBSC QLI Annual Assessment and Program Review initiative. The college employs Nuventive Improve as a data repository for activities, including the ISLO assessment. In addition, the CTLE Back Design Academy provides a plethora of tools and workshop materials that can assist in measuring the quality of outcomes.
CTLE SITE
Part 2: Aligning Classroom Teaching and Learning to Outcomes
Main Function
Every instructor should offer classroom activities that are directly aligned to learning outcomes so that students have opportunities to be introduced to related concepts and to practice or master related skills. This does not mean that all activities must be connected to a learning outcome, but it does mean there should be at least one activity aligned to every learning outcome.
Key Points to Know
- A classroom activity that is aligned to learning outcomes might be used to introduce students to content related to a given outcome, or the activity might be used to allow students to practice or master related skills.
- Students must be given opportunities to learn, practice, or master related content if there is an expectation for them to do so.
Benefits
Classroom activities that are aligned with learning outcomes
- help faculty members know where in the curriculum they will introduce related content to students or give students opportunities to practice or master related skills and abilities;
- provide opportunities for students to engage with related content, knowledge, skills, and abilities; and
- build in formative assessment opportunities which help faculty members know how well students are achieving a given outcome before it is time to formally assess.
Getting Started
You have good outcomes. Now what? Good outcomes mean nothing if several things do not happen. Students should know the outcomes. In fact, students should be explicitly told what the outcomes are, because it lets them know what they will be expected to demonstrate during or by the end of the course or program. Faculty should know where in the curriculum students have the opportunity to be introduced to related skills and when students can practice those skills. It should also be clear to the faculty member, students, and external evaluators what assessments allow students to demonstrate those skills.
Curriculum maps are excellent tools to articulate this information and should be considered before building classroom activities. Maps provide a “check and balance” for faculty members and the institution. Curriculum mapping is an exercise that can be done at all levels. Mapping allows faculty to document connections between program and institutional outcomes, between course and program outcomes, and between course activities, assignments, or assessments and outcomes at any level.
Mapping Learning Outcomes to the Curriculum
Begin by asking for any outcome: “In what program, course, or session will students be introduced to and master the necessary skills associated with this outcome?” How a map is created really depends on the level of outcomes being mapped, and specific terms used may vary among institutions that use curriculum mapping. Curriculum maps “front-load” a lot of work in planning a course or program, but once a curriculum map is complete, it provides clear direction for instruction and assessment.
Terminology of a Curriculum Map
Typically, a curriculum map includes three levels of activity related to outcomes. Many institutions use the terms “introduce” and “reinforce” to describe the first two levels. The highest degree may be described by what the faculty will do (emphasize, provide extra coverage) or what the students will do (master). In this section, the terms “introduced, reinforced,” and “proficient” will be used to describe activity levels.
- Introduced – students likely see content or learn a skill for the first time.
- Reinforced – students are typically allowed to practice related skills or apply related knowledge.
- Proficient– students are typically expected to demonstrate mastery.
Building Classroom Activities that Lead to Outcomes Achievement
After mapping the curriculum, a faculty member should be well acquainted with not only each learning outcome but also the extent to which related skills and knowledge are included in the course. Faculty should know at this point if students are simply introduced to the content in the course. Faculty should know if they will reinforce previously learned content, giving students the chance to practice related skills, or if the student is expected to demonstrate mastery of related skills in the course.
In other words, if a curriculum map verifies that only an introduction to knowledge or skills is necessary, a lecture may suffice, and instructors can often select activities that require only lower levels of thinking. However, if knowledge and skills are to be practiced by the student, instructors should seek activities that will require students to apply and analyze content. Finally, if students must demonstrate mastery of knowledge or skills, instructors should consider classroom activities that will promote the higher-level thinking skills of synthesis and evaluation.
Part 3: Assessment of Learning Outcomes
The Need to Know Information
Main Function
Assessment should inform teaching and improve student and institutional learning.
Key Points to Know
- Assessment should provide useful information to faculty and the institution.
- Assessment should measure the intended learning outcomes.
- Assessment should never be used punitively.
- Assessment is not perfect.
Benefits
- Good assessment is a tool for learning: it lets students demonstrate outcomes achievement with feedback to tell them where they are missing the mark and allows faculty to adjust teaching when necessary.
- Good assessment lets faculty know what progress is being made toward an outcome.
- Good assessment lets faculty test what they teach instead of teaching to a test.
- Good assessment lets faculty stay ahead of legislation.
- Good assessments provide a way for faculty and the institution to articulate student learning to internal and external constituents.
Why Assessment is Necessary
Why do faculty give grades to students instead of trusting that each one learned what was taught? One might presume grades are assigned to let students know they have learned (or not). Grading then becomes a tool for learning and growing because it helps students learn what they did and did not understand in the material.
In this same way, assessment is a tool. Just as the students can use grades to improve, assessment is about getting the same benefit for ourselves. Specifically, assessment allows gathering feedback we can use for our own benefit. The primary purpose of assessment is to improve learning, and good practice suggests we assess for three reasons: to assist students with learning, to advance students through a program of study, and to adjust our teaching or curriculum.
Assessment, while never perfect, is a tool to help faculty and an institution know and articulate how well students are achieving course and program learning outcomes. Before the learning outcomes model currently in place throughout colleges and university systems, assessment efforts were more focused on indirect measures and assessments of achievement, such as GPA and transfer rates. Although these measures are still monitored, the current model provides greater benefits for faculty and students by incorporating direct measures such as assignments and assessments that are embedded at the course level.
Getting Started
If you have not already, you are strongly encouraged to create a curriculum map so you know where in the curriculum the outcomes and related content are introduced, reinforced, and emphasized. A curriculum map can also be utilized to “map” assessment to the outcomes. After you know what you expect students to do or know and when, you can select the right tools and develop an assessment plan. An outcomes assessment plan helps an instructor, program faculty and staff, and even an institution know what will be assessed, when it will be assessed, and how it will be assessed.
Developing assessment plans for an outcomes-based curriculum is far from unique to Palm Beach State College. It is good practice and commonly accepted in higher education, and many good resources are available.
Selecting Assessments
Assessment should be both formative (along the way) and summative (near or at the end), and multiple measures should be built into an assessment plan. Assessment should measure student performance and capture specifically the skills required to achieve a learning outcome. Assignment grades should not be used to assess a learning outcome unless the grade equates to a score that specifically measures the skills associated with achievement of the outcome.
When choosing an assessment, be certain you know why you are using it, who will assess and be assessed, what outcome it will measure, and what the scores will help you know or understand about student learning. Include direct measures, that is, assessments that require student performance directly related to a learning outcome.
Direct Measures of Learning Outcomes
Each assumes a clear relationship to a given outcome.
- Ratings of student performance
- Rubric scores on projects, presentations, research, written work, or other assignments
- Portfolios
- Quiz or test scores
- Results of student performance observations
- Results of skills-based or lab-based performance
- Results from student response systems
Indirect Measures of Learning Outcomes
- Course grades (will not give any information related to achievement of individual outcomes)
- Grades on projects, presentations, research, written work, or other assignments unless a rubric or scoring criteria clearly relate the assignment to a specific outcome
- Scores on external exams for entry into other programs unless the exam has isolated content scores that can be directly related to a specific outcome
- Satisfaction surveys
- Course evaluation surveys
Scoring Guides, Rubrics, and Checklists
Scoring guides, rubrics, and checklists can be effective assessments for formative assessment because such instruments typically include multiple criteria that students must master before taking a summative assessment to show they have achieved an outcome. Scoring guides and rubrics can be effective as summative assessment instruments. These tools most often clearly state criteria and require an instructor to either “rate” the degree to which the students meet the criteria or simply indicate the student does or does not meet each expectation. Scoring guides and rubrics can be applied to (used to score) a variety of assignments, including projects, presentations, research, written essays or papers, case studies, skill demonstrations, and more, and – assuming multiple faculty use the same scoring instrument and calibrate their use of the instrument – trend analysis becomes possible. Trend analysis allows faculty to address the degree to which their students are learning in the classroom, and trend analysis allows an institution to discuss the quality education it provides.
Selecting Performance Standards
Standards are often referred to as achievement targets, benchmarks, or expectations. Standards help faculty and the institution know the degree to which students are learning and easily provide a way to articulate student learning to each other and to external stakeholders. Setting standards and recording whether students meet those standards (or do not) is an important part of closing the assessment loop each cycle. There are multiple considerations when selecting standards.
Minimum Scores
A standard might be a minimum score, say a score of “4” on a 5-point rubric. It might also be a comparison to an external score such as a national benchmark, for example equal to or greater than the national average for students at a community college. A standard might also be set against previous performance, for example, with an expectation of annual improvement or maybe a demonstrated gain by the end of a semester over a pretest taken during the first week.
Achievement Targets
After a standard is established, it is also important to know the expected percentage of students who should meet the standard. In some cases, for example a licensure exam, meeting a standard may be a requirement to complete the program. In other cases, faculty may expect that a minimum percentage of students, say 80%, will achieve a minimum score. In still other cases, faculty may be looking for improvement over an initial point in time or an initial score.
Sample achievement targets include the following:
- 95% of students will demonstrate the skills safely and accurately in a lab setting.
- 75% of student scores will score at least 20 points higher on the post-test than on the pre-test.
- 80% of students will achieve a B or higher on the exam.
- 80% of students who take the licensure exam will achieve a score equal to or greater than the national average.
Developing Assessment Plans
Assessment plans at the institutional, program, or unit level should be developed with content experts to ensure careful alignment to the outcomes. Assessment plans should include very specific components, and the plans should be documented. Faculty developing assessment plans for a course should consider documenting the plan.
Assessing the Assessment Plan
Assessment plans should be evaluated for clarity and connection to the curriculum. The following criteria should be met when choosing assessment instruments and benchmarks.
- Selected assessments purposefully measure an intended outcome.
- Selected assessments are affirmed by content experts (faculty, staff, or literature).
- Selected assessments provide information that is as accurate and valid as possible.
- The assessment plan includes multiple measures with at least one direct, authentic measure of student learning for each learning outcome.
- Achievement targets are clearly stated and justified by faculty who teach the related content.
- Data collection processes are explained and appropriate.
- The plan includes ways to share, discuss, and use the results to improve student or institutional learning (see Part 4 of this handbook).
Components of an Assessment Plan
Plans at all levels (institutional, program, course) should include at a minimum, the measures and achievement targets for each learning outcome. Every outcome should have at least one measure and a corresponding standard should be established for each measure. The standard may be called something else such as benchmark or achievement target. Currently at Palm Beach State, both terms are used in assessment plans.
Implementation and Data Collection
Good assessment plans also include details regarding implementation and data collection. For example, if a course-level assessment is selected as a program measure, the assessment plan should clarify for all instructors in that course what the assessment items are and exactly when in the course the assessment is to be administered. “When” may be inherent because it is stated as a specific unit or lab within the course. However, if an assessment is not specific to a unit test or assignment, it should be clear what elements of the course should be covered before students take the assessment or complete the assignment.
An assessment plan can succinctly clarify other important details. Consider including:
- Semesters or date ranges for the assessment cycle
- Who will be assessed (students in what class?)
- When in that cycle the data are to be collected (state any specific unit(s) that must be covered before the assessment is administered, or provide a time frame such as ‘during the last regular week of class)
- Directions for reporting results
All academic programs at Palm Beach State College have developed assessment plans, and faculty in many courses have done the same.
Authentic Assessment
The assessment plan should address the issue of the type and amount of credit students can earn for participating in the assessment. Specifically, faculty should discuss how much credit and whether credit earned is “extra” or “authentic,” meaning it is applied to the course grade. Authentic assessment is, of course, preferred. When students participate in assessment with results attached to their course grade, they are more likely to give their best effort, and faculty are more likely to find the results useful to improve learning.
In some programs, faculty will not have options regarding how to award credit, but in those cases where it is optional, ideally, faculty will try to agree on a percentage of the grade a student can earn on the assessment. For example, faculty may agree that the assessment will be worth between two and three percent of the total grade.
Part 4: Using the Results
Main Function
Assessment results should be used to improve student or institutional learning.
Key Points to Know
- Assessment can only become meaningful if results are shared, discussed, and used to improve learning.
- Assessment results should never be used punitively.
- Results of one cycle should inform improvement strategies for the next cycle.
Benefits
When assessment results are reviewed with the intent to improve learning
- Discussion and reporting are improved: Faculty can point to tangible evidence when they report their students are (or are not) learning.
- Teaching is improved: Faculty can know how to adjust teaching when they see where students have skill deficiencies. In many cases, faculty can know how their students perform compared to other students, helping those faculty members hone their teaching methods.
- Collaboration is enhanced: Regular conversations about results can lead to discussions of best instructional methods, wider benchmarking and comparisons, common challenges and effective solutions, and more.
Self-Assessment on Using the Results
It is critical to understand that assessment is not meaningful unless and until the results are used appropriately. Assessment becomes meaningful only when faculty and others review the results and use those results to inform instruction and improve learning, their own or that of their students.
Review at Palm Beach State College varies based on the type of outcomes assessed.
- Institutional learning outcomes and general education competencies – review is most often by open invitation, done in small forums, campus meetings, cluster reviews, during Development Day breakout sessions, or in committee meetings. Faculty review results and develop improvement strategies to be implemented in the next cycle.
- Program learning outcomes - annual review includes a review of learning outcomes assessment results. Program faculty review the results, select at least one outcome to target for improvement unless all have been met and other data points are selected, and develop action plans for the next cycle.
- Course learning outcomes - review is conducted individually by faculty during the annual appraisal or preparation for the continuing contractual portfolio. Faculty reflections on and use of assessment results are documented and shared with associate deans.
In a conversation about using the results, this is the time to think of it as a continuous loop. It truly never ends!
- We must develop outcomes, the important learning goals we set for students.
- We must ensure classroom activities will provide opportunities for students to learn what we want them to learn and achieve the outcomes.
- We must assess. How else can we know whether and to what degree they know and do what we expect them to know and do?
- Finally, we must use those results to figure out what to do next, but then it starts all over, completing yet continuing the cycle! Results may lead us to keep or revise the outcomes, but we must start all over again based on the learning goals.
In education, reviewing assessment results and planning for the next teaching, learning, and assessment cycle is often referred to as closing the loop. When the loop is closed on each cycle, the discussion typically includes a review of several components.
- Faculty should consider the continued appropriateness of the learning outcomes, measures, and targets, planning for improvements where needed.
- A review should include conversations about how assessments were implemented, and if that implementation still makes sense, assuming it did in the first place.
- Individual faculty should consider the results of their students and compare those results compared to students taught by colleagues.
- Together, faculty should consider ways to adjust teaching methods to address any noted learning deficiencies and ways to improve the process to ensure maximum consistency if that is a concern.
A review of the results should never be punitive! The process at PBSC continues to evolve, and so does student learning. Results should always be reported in aggregate without individual student or instructor names, and results should be used only to make learning better and to design improvement strategies. If assessment results are less than desirable, faculty should derive an action/improvement plan with measurable objectives without singling out any individual instructor.
Reporting Results
So, what information should be shared? While assessment reports will vary greatly depending on the audience and purpose, some basic components should always be included in a final report. The Executive summary, the actual analysis of results, any disaggregate findings, etc. are examples of results and findings.
Benefits of Sharing and Discussing Results
Results can be used to inform teaching. When faculty understand the deficiencies that exist, teaching methods can be adjusted to account for those shortfalls. With summative assessment, improvements can be planned for the next cycle. In the case of formative assessment, improvements can be made immediately.
Using results can lead to enhanced collaboration. Discussions about results should include conversations about pedagogy, successes, and challenges. As these topics are discussed, synergetic solutions can be developed to address unattained outcomes or unmet achievement targets.
When faculty participate in regular review of assessment results, they are more knowledgeable regarding the degree to which students are achieving learning outcomes in the program or course. Such knowledge becomes useful in conversations with supervisors and deans and on performance evaluations when some related conversation is required, or in similar conversations with academic leaders, community business partners, or other stakeholders.
Faculty who are well acquainted with details about student learning are further positioned to stay ahead of legislation. When lawmakers begin to discuss changes to impose on educators, faculty and staff can speak to issues of curriculum with the facts about learning at their college.