Usingthe Occupational Performance Model (Australia) to structure processand outcome measures for occupational therapists working with children

Jill Hummell, Diana Barnett and Sharon Doyle

Based on a performance indicators document produced by Westmead Hospital, November, 1994 and a paper presented at the 6th State Conference of OT Australia AAOT-NSW, Mudgee, NSW (1993, October)

JillHummell, BA,MADipOT, is a lecturer in the School of OccupationalTherapy, The University of Sydney. At the time of writing theoriginal paper and document, she was Deputy Manager, OccupationalTherapy Department, Westmead Hospital, Westmead, NSW.

DianaBarnett BAppSc(OT) is a therapist at The New Children’s Hospital,Westmead, NSW. At the time of writing the original paper anddocument, Diana was an occupational therapy Student Unit Supervisorat Westmead Hospital, Westmead, NSW.

SharonDoyle BAppSc(OT) is a therapist at The New Children’s Hospital,Westmead, NSW. At the time of writing the original document, Sharonwas an occupational therapist at Westmead Hospital, Westmead, NSW.

INTRODUCTION

It is essential for occupational therapy to be effective and efficient and optimize the use of scarce resources. A strategy which can be used within an overal plan to evaluate services, promote accountability and quality mangement is the use of process and outcome measures. These measures can be viewed as one method of collecting and analyzing quantifiable effects of occupational therapy intervention with the aim of using the data to maintain a continuous cycle of quality improvement. In addition, the process and outcome measures can form the basis from which empirical research can be developed and implemented.

Although evaluation has long been one aspect of the occupational therapy process (Hagedorn, 1992; Hopkins & Tiffany, 1978), in the current economic climate, it is critical that evaluation is based on quantifiable measures. The Australian Council of Health Care Standards (ACHS, 1992; 1995) maintains that a range of quality activities which address both processes and results must be utilized to evaluate and improve patient/client care.

Recent occupational therapy literature, including Austin and Clark (1993), Barnett and Hummell (1994), Hammell (1994), Noyce (1993) and Rogers and Holm (1994), has raised many issues relevant to measuring outcomes of occupational therapy services. In addition, these authors have identified strategies which can be used to measure the effects of intervention.

PURPOSEOF THIS PAPER

This paper outlines the evolution of process and outcome measures originally developed in the Paediatric Area of the Occupational Therapy Department at Westmead Hospital as well as our later revisions. Examples of the process and outcome measures used are provided. Information about documenting these measures, benefits encountered and future plans are included in addition to recommendations for occupational therapists who which to develop measures for their service.

The method of developing, implementing and evaluating process and outcome measures

described in this paper is one of a range of methods. Each occupational therapy service needs to determine the method/s and measures which are most meaningful to that service.

We anticipate that we will continue to evaluate and modify the information contained in this paper. Process and outcome measures consistent with other aspects of occupational therapy services need to be regularly evaluated and improved.

We have chosen to document process and outcome measures using a positive framework (Muhlenhaupt, 1991). An example of a positive outcome measure is “increased skill in self care activities”. Primary reasons for using a positive framework include:

*Relative to process measures there is consistency with the procedures involved in the occupational therapy process of assessment, goal setting, intervention and evaluation.

*Relative to outcome measures, there is consistency with individual and programme goals of direct intervention.

DEFINITIONS

The terms ‘process measures’ and ‘outcome measures’ have been used to identify the two different types of measures we believe are most relevant to occupational therapy direct intervention services. Both are measures “used to review performance” (Noyce, 1993.p.1).

Aprocess measure is a documented measure of an activity, for example,an assessment, which is in engaged in with/for a client (Noyce, 1993).

Barnett and Hummell (1994) suggest that process measures are developed after identifying the optimal procedures in the occupational therapy process of assessment, goal setting, intervention and evaluation of direct intervention with each client caseload/subgroup.

Anoutcome measure is a documented measure of the results ofintervention, for example client progress, as a direct or indirectresult of intervention provided (ACHS, 1995; Noyce, 1993)

Outcome measures are “a means of assessing the impact of intervention” (Hammell, 1995.p.46). They need to be sensitive (Hammell, 1994; Velozo, 1994), objective (Rogers & Holm 1994), meaningful (Atkins & Clark, 1993; Barnett & Hummell, 1994; Noyce, 1993) and reliable (Hammell, 1994). In addition, they need to provide information about the unique contribution of occupational therapy services to individual clients and client groups (Barnett & Hummell, 1994; Hammell, 1994; Velozo, 1994).

We recommend that process and outcome measures are developed for specific client caseloads/subgroups and services. Some measures may be useful across client caseloads and services and others may not. The examples we provide in this paper may be meaningful or may require partial or complete revision to obtain data appropriate to other settings.

Generating process and outcome measures is followed by data collection, collation and analysis. The results of the analysis and ensuing recommendations for changes, if these are required to improve the service, are documented. These changes are implemented and the cycle continues (Austin & Clark, 1993; Barnett & Hummell, 1994).

EVOLUTIONOF PROCESS AND OUTCOME MEASURES

The original process and outcome measures were developed for the client caseload for children with perceptual motor difficulties and their families (Barnett & Hummell, 1994). The primary reason for this was that the client management protocols for this population were being reviewed. Focussing on a single client caseload assisted in making the task achievable.

Two tools were employed to assist in developing these measures. First, existing occupational performance models were used to construct a framework for developing performance indicators (Chapparo & Ranka, 1993; Pedretti & Pasquinelli, 1990). Occupational performance was used because it offered the greatest scope for describing the focus of occupational therapy practice for children with perceptual motor difficulties at the time.

Second, principles of flow charting (McConnell, 1989) were used to facilitate our understanding of the optimal procedures followed in the occupational therapy process of assessment, goal setting, intervention and evaluation of direct intervention. The process measures developed were based on the outcome of the flow charting.

After successfully trialling the assessment and process measures, additional ones were developed for children with perceptual motor difficulties. Simultaneously, measures were developed for other client caseloads. Developing and trialling these later process and outcome measures was achieved much more efficiently than the original ones for a range of reasons including:

*practice

*many of the original measures were directly applicable to other client caseloads, or required minor modifications

Some were inappropriate and new ones were developed specific to the new client caseload. For example, children who had conditions/disorders which resulted in occupational therapy which targetted maintenance of skills with or without aids or equipment, outcome measures were modified to reflect this.

PROCESSPERFORMANCE INDICATORS

The procedures that constitute occupational therapy managment for children with perceptual motor difficulties from the receipt of referral to discharge are illustrated by the flow chart in Appendix 1 (Barnett & Hummell, 1994.pp 16-18). Charting the procedures allowed us to identify the essential and optional processes at each stage of occupational therapy intervention (for example, referral, gather data). The essential processes became ourprocess measures and are outlined as follows (Barnett & Hummell, 1994):

Assessment

*Initial interview is completed

*Relevant non-standardised assessments are completed

*Relevant standardised assessments are

completed

*Assessment findings are discussed with parents and child at the appointment following assessment or via telephone within two weeks of assessment, as negotiated with parents and children.

Aimsand Goal Setting

*Treatment aims/broad goals are determined collaboratively between the child, family members and therapist (including other team members when appropriate).

DocumentingGoals

*Goals are documented in the therapy file.

ReportWriting – Outpatients

*Initial report is written by the sixth appointment

*Letter/report is sent to referring doctor and parents.

*Initial report and/or review report/discharge report is written and sent to appropriate personnel with parent/guardian’s permission.

Documentation- Outpatients

*Relevant information is documented into therapy file immediately after each client contact.

Evaluation

*The child’s progress is formally evaluated at a time determined with the client and family, when the goals are set.

OUTCOMEMEASURES

The initial outcome measures were determined through group discussion about the primary presenting problems of children with perceptual motor difficulties and therefore the primary aims/goals of occupational therapy intervention (Barnett & Hummell, 1994). Later, outcomes

measures were developed for children from a range of other client caseloads. All outcome measures were developed which:

*retainedand promoted the individualised nature of occupational therapyintervention

*capturedthe essence of occupational therapy involvement with children andtheir families

These two characteristics were achieved by documenting individual goals and consequently outcome measures using the Occupational Performance Model (Australia) (Chapparo & Ranka, 1993). Consistent with this occupational performance model, the outcome measures were documented relative to those occupational roles, occupational performance areas and occupational performance components. that were most frequently identified as occupational therapy treatment goals with children from each specific client caseload. Examples are detailed below.

1.OccupationalRoles

*Satisfaction with occupational roles – self maintainer and/or player and/or school or pre-school student – relative to the current physical, cultural and social environments, as determined collaboratively with the child and family.

* Increased independence with occupational role performance

2.OccupationalPerformance Area Tasks

Productivity

*improved handwriting

A number of more detailed performance indicators were drafted in the occupational performance area of productivity for handwriting. These included:

ProductivityOutcome Measure

Improvedhandwriting

Examples of more specific outcome measures for handwriting tasks and subtasks

Increasedspeed of writing

Increasedquality of writing

-spacing

-size

-neatness

-formationof letters

Increasedquantity of writing

Increasedconsistency of writing

Increasedendurance when writing

Improvedposture when writing

Improvedgrip when writing

Play

*increased engagement in age appropriate play

SelfMaintenance

*increased skill in self care sub tasks, tasks and routines

A number of more detailed outcome measures were developed for these areas. For example, in the occupational performance area of self maintenance, detailed outcome measures were structured as follows.

SelfMaintenance Outcome Measure

Increasedskill in self maintenance subtasks, tasks and routines

Examples of more specific outcome measures within this performance area include:

Mealtime Routines

Increasedskill with utensils

Increasedskill with pouring a drink

Improvedsocial behaviour at mealtimes

Dressing Routines

Increasedskill with dressing

Increasingskill with undressing

Increasedskill with fastenings

Increasedskill with buttons

Increasedskill with shoelaces

Personal Hygiene Routines

Increasedskill with bathing

Increasedskill with cleaning teeth

Increasedskill with combing hair

Increasedskill with drying self

Increasedskill with washing self

The outcome measures listed here are not considered inclusive of all occupational therapy intervention aims/broad goals for children. They are examples which demonstrate the varying levels of specificity of outcome measures that can be developed.

Sleep/Rest

* Increasedbalance betweensleep/rest and activity routines

3.OccupationalPerformance Components

The following are examples of outcome measures relevant to each performance component.

Sensorymotor

*Improvedmotor control appropriate to occupational tasks, subtasks androutines

Biomechanical

*Increasedrange of motion required to perform occupational subtasks, tasks androutines

Cognitive

* Increasedsustained attention to an occupational task, subtask and routine

Interpersonal

*Improvedpeer interaction during occupational tasks, subtasks and routines

Intrapersonal

*Increasedperseverence when participating in occupational tasks, subtasks, androutines

4.ExternalEnvironment

The following are examples of outcome measures developed relevant to external environments.

PhysicalEnvironment

* Increasedaccessability to home/leisure/school environments

SocioculturalEnvironment

*Increasedability of carers to safely and effective care for their child’soccupational needs

*Parentscan identify positive qualities/skills in their child’s occupationalperformance

SensoryEnvironment

* Parentsconfidently handle and position their child for occupationalperformance tasks, subtasks and routines

*Parentprovide developmentally appropriate toys for their child’s leisureneeds

Consistent with the individualised nature of occupational therapy intervention with children who have perceptual motor difficulties, individual goals are initially developed collboratively with the child and his/her family prior to intervention and after assessment, as outlined in the flow process chart in Appendix 1. The content of short and long term goals are documented in a manner that is consistent with major constructs in the Occupational Performance Model (Australia), 1993.

Whether or not the goals are achieved is the measureable aspect of the outcome performance indicator.

DATACOLLECTION FORMAT

Data is collected for each child who received intervention and for the relevant caseload.

Data collection forms are used to document the extent to which process measures have been completed, and outcome measures achieved. One form is used to document process and outcome measures for each child, and another form, to collate information for each client caseload.

IndividualChild Data

The process and outcome measures for each child are documented onto the goal sheet in Figure 1 (Barnett & Hummell, 1994). On this sheet, the completion of process measures is indicated by a date in the relevant space. A blank space indicates non completion. The abbreviations on the top right hand side of this form are explained in the accompanying key (See Figure 1).

The achievement of outcome measures is indicated using a 1 – 4 rating scale and placing the relevant number in the ‘outcome measure’ column. Figure 1 provides an example of this documentation. It must be remembered that these goals are set with the child and parents and are considered high priorities by them at the time. The information obtained from each child is subsequently collated onto the caseload process and outcome measure forms.

Figure1: Goal Sheet and Key to Abbreviations

CaseloadData

The data collected for each client caseload is collated onto the caseload process and outcome measure forms prior to data analysis (Barnett & Hummell, 1994). Relevant comments are also transferred onto this form. The figures below provide examples of caseload process (Figure 2) and outcome measures (Figure 3) forms.

Figure2: Process Measures Form

(Adapted from form developed at

Westmead Hospital, September, 1994)

TimeFrame

A time frame for the collection and analysis of data needs to be determined relevant to each client caseload. For example, if initial contracts for therapy are approximately three to four months for children with perceptual motor

Figure3: Outcome Measures

Form(Adapted from form developed at Westmead Hospital, September, 1994) difficulties, process and outcome measures are collated and analyzed approximately every four months. Time frames for collation and analysis for process and outcome measures may vary for different caseloads.

DATAANALYSIS

Calculationof Percentages

Once the data has been collated for each client caseload onto the appropriate form, percentages are calculated (See Figures 2 and 3).

To calculate percentages for each process and outcome measure the following method is suggested (Barnett & Hummell, 1994):

Process Measures Numerator =The number of children for each completed process measure. Denominator =The number of children in the relevant caseload who attended occupational therapy.

Outcome Measures Numerator=All children who achieved their documented goal for the relevant outcome measure.

Denominator=All children who had a documented goal for the relevant outcome measure.

Analysis

The collated data, with the percentages of children who achieved their stated goals in relevant occupational roles, occupational performance area and occupational performance components, in addition to meaningul comments, is analysed (See Figure 4). Given that one child

Figure4: Outcome measure data analysis may have a number of goals documented in one performance role, area, or component category the collated data is no longer relevant to the number of children who have achieved these outcomes, but to programme outcomes.

Any comments noted on the child’s individual goal sheet (See Figure 1), are important for data analysis. For example, a child may not have achieved his/her stated goals, but a major event in the child’s life may have occurred which explained this outcome. Data is similarly analyzed for process measures.

As a group, all occupational therapists involved with the relevant caseload are given an opportunity to discuss the results of the documented process and outcome measures for the caseload. For each process measure, the percentage completed are discussed. For each outcome measure, the percentage of goals achieved or not achieved are discussed. Issues relevant for discussion with each client caseload when reviewing the data have been found to include:

*In terms of percentages, what are the outcomes of occupational therapy intervention programmes.

*Are occupational therapy intervention programmes more effective in achieving some goals/skills than others?

*For those goals/skills for which intervention is less effective, are different intervention strategies required or is there another service/profession to which referrals should be made?

*Are the documented goals appropriate, do they represent the current priorities; are they too difficult; too easy?

*Are the documented procedures (process measures) being followed? If not, do they need to be changed?

The following categories may be useful when analysing the data (QRB), 1989).

Client issues

These include the perceived level of motivation, and/or health, for example the child was ill during the period of intervention.

Therapist issues

These include the quality of the therapeutic relationship and the therapists expertise in intervention and goal setting.

Organisational issues

These include the available resources, for example, space and equipment to carry out intervention.

REVIEW

It is important to regularly review the process and outcome measures. When reviewing them, issues to consider can be summarised by the following questions.

Dothe process and outcome measures provide meaningful data?

Arethey the most appropriate measures to use?

Dothey achieve their stated purpose?

Dothey require modification or elimination?

In addition, it is important to regularly review the data collection format and data analysis process.

FUTUREPLANS

Future plans for process and outcome measures include the following:

*determining and developing additional specific outcome measures

*reviewing the frequency of data collection

*investigating the computerization of documentation and data collection

*integrating the measures with statistics collected

*reviewing relevant literature on an ongoing basis

*engaging in outcome research studies

BENEFITS

The benefits encountered by the occupational therapists involved in the development of these process and outcome measures have both direct and indirect. They have included the following (Barnett & Hummell, 1994):

*improved skills in goal setting and clinical reasoning

*clarification of the goals of occupational therapy intervention at both an individual and programme level

*discussions and clarification of the appropriate focus and priorities of occupational therapy intervention, evaluation and procedures

*philosophical and clinical discussions about occupational therapy including sharing information and networking within and between departments

*provided data on the effectiveness of direct occupational therapy intervention and a formal review of client mangement protocols and outcomes at a programme level

*improved skills in the development of data collection forms and data analysis

*increased confidence with clarification of the occupational therapy role, and data demonstrating the effectiveness of intervention

RECOMMENDATIONS

The following recommendations are made for people who wish to develop, implement and evaluate process and outcome measures in their services (Barnett & Hummell, 1994)

1) Review client management protocols

2) Link the development of process performance indicators with existing protocols

3) Use group discussion as a means of developing, analysing and reviewing measures

4) Develop a small number, 2 – 3, outcome measures which are meaningful and reflect the priority of the service. These may be caseload specific or across all caseloads.

5) Develop forms for recording process and outcome measures.

6) Trial the forms and analyse the data collected

7) Implement recommended changes

8) Review/evaluate forms used and the value of the measures developed

9) Develop additional process and outcome measures and continue the process outlined.

SUMMARY

This paper has outlined the evolution of process and outcome measures for children which were initially developed at Westmead Hospital, and our later revisions. Data collection and data analysis procedures have been discussed. The need to regularly review the measures developed and the data collection format was stressed. Future plans and the benefits of engaging in this process of developing performance indicators along with recommendations are provided for those interested in developing process and outcome measures for their service.

CONCLUSION

It is hoped that we have provided a strategy for the development, implementation and evaluation of process and outcome measures for occupational therapists. As stated earlier, this is one method of developing process and outcome

measures. Measures need to be relevant to the caseload of clients seen and the existing protocols within the service delivery system.

It is helpful to use a practice model to assist in the generation and revision of process and outcome measures. The Occupational Performance Model (Australia) (1993) contains the necessary scope to do this. The constructs within this model, particularly occupational roles, occupational performance areas and occupational performance components reflect the scope and diversity of occupational therapy intervention that occupational therapists offer to children with a range of presenting problems, and their families.

ACKNOWLEDGEMENTS

The authors acknowledge the paediatric occupational therapists from Westmead Hospital and The New Children’s Hospital (formerly Royal Alexandra Hospital for Children) who assisted in the development of the process and outcome measures. We acknowledge the encouragement provided by Sue Robinson for the original project.

References

Austin, C., & Clark, C.R. (1993) Measures of outcome: For whom? BritishJournal of Occupational Therapy, 56(1), 21-24

Barnett, D. & Hummell, J. (1994) Performance indicators for

occupationaltherapists working with children who have perceptual motordifficulties, and their families. (Available from the Occupational Therapy Department, Westmead Hospital, Westmead, NSW) November.

Chapparo, C., & Ranka, J. (1993, October) Occupational

performance:a practice model for occupational therapy. Paper presented at the OTAustralia AAOT-NSW 6th State Conference, Mudgee, NSW

Characteristics of clinical indicators (1989) Quality Review

Bulletin,November, 330-339

Caddow, P. (1986) Questions on quality. Nursing Times, July 16, 42-43

Hagedorn, R. (1992) Occupational therapy, foundations and

practice. London:Churchill Livingstone.

Hammell, K.R.W. (1994) Establishing objectives in occupational therapy practice. Part 2. British Journal of Occupational Therapy, 57(2) p 45-48

Hopkins, H.L. & Tiffany, E.G. (1978)Occupational therapy – a

problem solving process. In H.L. Hopkins and H.D. Smith (Eds.). Willardand Spackman’s Occupational Therapy (5th Ed.). Philadelphia: J.B. Lippincott

Lehmann, R. (1989) Forum on clinical indicator development; a discussion on the use and development of indicators. Quality Review Bulletin,July, 223-227

McConnell, J. (1989) The seven tools of TQC. (3rd Ed.). Delware Group: Dee Why, NSW

Muhlenhaupt, M. (1991) Components of the program planning

process. In W. Dunn (Ed.). Paediatric occupational therapy: Facilitatingeffective service provision (pp. 124-136), New Jersey: Slack Inc.

Noyce, J.A. (1993 April) Performance indicators – A summary for occupational therapists.Newsletter of the NSWAOT, Edition 307, 14-16

NSW Association of Occupational Therapists Performance

Indicators Working Party. (1993, October). Performance indicators foroccupational therapists – draft 2. (Available from OTAustralia AAOT-NSW, PO Box 142, Ryde, NSW, 2112)

Pedretti, L & Pasquinelli, S. (1990) A frame of reference for

occupational therapy in physical dysfunction. In L. Pedretti, and B Zoltan, (Eds.). Occupational therapy: Practice skills for physicaldysfunction (3rd ed.) (pp. 1-16). St. Louis: C.V. Mosby.

Rogers, J.C., & Holm, M.B. (1994) Accepting the challenge of

outcome research: Examining the effectiveness of occupational therapy practice. American Journal of Occupational Therapy, 48(10), 871-876

Schnieden, H. (1988) Management and performance indicators – is a backlash possible?Hospital and Health Services Review, February, 28-29

The Australian Council on Healthcare Standards (1995)Charter forchange: The framework. Sydney, ACHS.

The Australian Council on Healthcare Standards (1991) Clinicalindicators: a user’s manual – hospital wide medical indicators. Sydney: ACHS

Thomas, J.W. (1990) The issue is not the useability of claims data but the quality of the indicators.Quality Review Bulletin, December. 422-423

Velozo, C.A. (1994) Should occupational therapists choose a single functional outcome measure? American Journal of Occupational Therapy, 48(10) 946-947

APPENDIX