TRAINING MANUAL

FOR

AN ECOSYSTEM MODEL

Stage III

Developing the Assessment Technique

This ecosystem model employs a two-phase assessment technique to obtain the necessary information with which to design environmental conditions. The first phase elicits respondents' perceptions regarding the environment or behavior in the environment, and the second phase asks respondents to briefly describe why they have these perceptions. The condition, policy, program, or physical property discussed in the respondents' brief descriptions become environmental referents (ERs) for the model's design purposes. The team will need to develop the test instruments or methodology for this two-phase assessment.

The more usual testing approaches can be used in the assessment's first phase. The team can develop its own instrument, modify an existent instrument, or use a standardized commercial instrument. Since environments tend to have highly unique characteristics from one campus to another, the team must take care that the instrument adequately covers conditions particular to its own environment and does not cover conditions or situations which do not pertain to its environment. Many instruments are designed for and helpful in establishing comparisons among college environments but are not very helpful in studying an individual college environment. Therefore, it is highly likely that the team will develop its own instrument based on a composite of methods. However, a review of other instruments can be extremely helpful in doing this.

The assessment's second phase will have to be developed by the team. It involves the obtainment of environmental referents. A simple form may be devised on which respondents can write their answers. There are advantages in using a tandem approach in which respondents answer the phase one instrument and then immediately supply phase two environmental referent information on a form designed for this purpose. First, the team will have to conduct only one set of testing sessions. Second, the analysis of data from both phases can begin simultaneously. And from the respondents' viewpoint, they will be bothered only once. The team will need to develop its phase one instrument first so that its phase two method can match subsequent environmental referents to it. In the development of assessment procedures during on-campus applications of the ecosystem model, psychometric issues concerning the validity and reliability of instruments were not a prime consideration. However, planning teams can certainly develop their own indices of reliability through split-half or test-retest procedures.

The processes given in Stage III have been developed and used by previous ecosystem planning teams. These are given in recommended sequence but are not all inclusive. Each team will probably add to the state of the art and develop processes useful to its work and to the advancement of ecosystem technology. Since the team sets important parameters for its assessment in Stage II, it can begin Stage III by conducting a review of assessment instruments that address these parameters. Often, as teams work through Stages III and IV, they find they want to readjust some of the areas of inquiry developed in Stage II. Two other essential tasks in Stage III include tailoring an assessment instrument or writing an original assessment instrument for phase one and developing an environmental referent form and analysis for phase two. The concluding two tasks entail processes for the assessment technique's pilot test and subsequent refinements that produce the team's final assessment technique.

Work on this manual proceeded while model applications were in progress. The procedures for reviewing, tailoring, or writing assessment instruments, and developing environmental referent forms have been distilled from experiences gained through three model applications. The procedures suggested for the assessment technique's pilot test have been written on the basis of two model applications. At the time this manual went to press, one model application had completed Stages IV and V. Therefore, the processes developed and given in Stage III for the analysis of environmental referent (ER) information have been tested by only one team.

Review of Assessment Instruments

The team should assemble a copy of each assessment instrument it can locate that deals with the environment it is studying. Various offices on campus may well have a copy of applicable standardized commercial instruments. If the environment has been the subject of a previous survey, it is helpful to obtain a copy of the instrument that was used. A brief synopsis of commercial and other published Environmental Assessment Instruments is given in Technical Appendix D, p. 99.

The review of instruments might turn up one that the team feels can be used for its project. Short of this, the review will enable the team to become more aware and conversant with the ways an environment can be assessed and provide ideas for the development of its own instrument.

It is equally important for the team to familiarize itself with the capabilities and constraints that its available computer resources and programs have. It may be that some of the instruments' formats could not be accommodated by the team's resources. Again, this is a point in the process at which the team will need information from a member knowledgeable in computer technology or from a consultant.

Process

In reviewing the assessment instruments, the team should:

1. Determine which instrument covers all or the greatest number of the team's chosen areas of inquiry and data objectives.

2. Determine how closely the instrument matches the desired level of inquiry--are the items too general or do they cover important breakdowns the team wants to assess? For example, the team might consider an item such as, My room is satisfactory for studying, too general. They may want to know if facilities in the room are adequate, or they may even want to ask about such specifics as lighting or bookcases.

3. Determine if respondents could easily identify with the instrument's language. For example, on some campuses it is applicable to use the term "student union" in reference to the campus' facility for student activities, government, etc., whereas on other campuses it is more appropriate to call it a "student center." In some cases a campus will have even developed a jargon in connection with the environment. To stray from this could weaken the assessment's results by annoying or confusing respondents, i.e., calling the environment's eating place a cafeteria when it is known as the dining hall or coffee shop. In other instances, some words are taboo and cannot be used in an instrument. Sensitivities among campus constituents may dictate against the use of such words as hell and damn.

4. Determine if the instrument is a feasible length for administration on the campus.

5. Determine if resources or computer availabilities can accommodate scoring and analyzing the instrument.

6. Determine whether, given the above considerations, there is an instrument that the team could use, intact, with modification,and/or additional sections or scales.

Discussion

If the team locates an instrument that it can use intact, then it can move on to developing its device or method for obtaining environmental referents. However, this fortuitous circumstance is not likely to occur often. The team will have a head start if it is able to locate an instrument that it can adapt for its assessment needs. The adaptations could entail writing additional items to achieve the level of inquiry desired, developing additional sections of items to cover facets of the environment not originally included, and changing words to reflect usage more common to the campus.

Process

In tailoring an instrument for its ecosystem project, the team should:

1. Obtain permission from the instrument's author(s) to adapt it and instructions on how to handle copyrights if necessary.

2. Decide in what ways the instrument will need to be tailored for use on the team's campus.

3. Review and apply the specific considerations given in the following section on Developing an Instrument as guidelines when tailoring an instrument.

4. Set up a systematic approach for making adaptations or changes. For example, the team may want to begin by deleting unusable items, then identify and change words that might annoy or confuse respondents, and finally write additional items or sections to accommodate the team's data objectives not included in the instrument. Or the team may want to subdivide into subcommittees to accomplish the tasks that will be needed in order to tailor the instrument. Whatever method is used, it is helpful to keep each task separate. The team may experience difficulty with some of these tasks in the beginning. Repetition develops facility and this is hard to achieve if the team jumps from one task to another and back again.

Discussion

If the team cannot find an instrument applicable to its use from among those reviewed, then it will have to develop its own instrument to use for the assessment's first phase. Development of an instrument is a fairly complex and time-consuming task. It does pay off, however, in providing the team with the best instrument for assessing their campus environment. Experience has also indicated that instruments developed by the team have a high "face validity," or acceptance, with respondents.

One vital consideration in developing or tailoring an instrument is obtaining the degree of importance that each item has for the respondents. This is crucial information for the team because a negative perception of some aspect of the environment does not necessarily indicate that the aspect so rated is important to the respondent. In a time of very limited resources, services cannot afford to invest their time and energies in changes that are not seen as important by those who live in the environment. An importance to the respondent measure can be built into the phase one format, or it can be dealt with in the environmental referent section. Some teams have chosen to include some ratings of importance in the phase one instrument, as well as in the environmental referent form.

Process/Format Review

The point of departure for developing an instrument is to review and become familiar with the various testing formats that can be used (see Technical Appendix D, p. 112, for a brief survey on Environmental Assessment Techniques). For purposes of this manual, these formats are classified as perceptual, goal statement, behavioral, and demographic.

Perceptual formats use a rating scale to measure respondent perception.

a. Likert scale presents a rating with discrete points (often five) which respondents use in replying to an item. The rating could be:

Strongly Agree (SA-1), Agree(A-2), Neutral(N-3), Disagree(D-4), Strongly Disagree(SD-5))

This campus is friendly. 1 2 3 4 5

b. Semantic differential uses opposite adjectives to describe aspects of the environment; respondents place a check mark to indicate their perception somewhere along the continuum given between the two extremes.

This campus is friendly - - - - - - unfriendly.

Goal Statement format uses the magnitude of discrepancy between "is" and "should be" ratings that respondents give in reply to the goal statement to measure perceptions. For instance, a category or area of inquiry concerning student services' goals in an instrument could title the section Goal of Student Services and use the format to measure perceptions on a number of items under that title.

Goal of Student Services

Behavioral format poses items about the environment in action or behavioral terms and asks respondents to note how often or how seldom these apply in their case.

I have talked with my advisor. Once (1) 2 to 5 Times (2) Never (3)

Demographic format assigns each choice in a category a code number and asks respondents to indicate the one applicable to them. The number of codes which can be used will depend upon the available computer program.

Your age

O = 17 and under
1 = 18
2 = 19
3 = 20
4 = 20-25

Are you a transfer student?

7 = Yes--from a two-year college
8 = Yes--from a four-year college
9 = No

The perceptual, goal statement, and behavioral formats are most useful in connection with environmental referents. The demographic format is used in this model to give planners the capability to separate groups in terms of future redesign and programming. For example, it may be important to separate responses by the respondents' age or major to determine if needs and perceptions are similar to diverse age groups and academic pursuits.

The collection of demographic information should be limited to that which the team can reasonably use. Including a large number of categories produces more data than can often be dealt with and also requires a larger sample of respondents in order to assure that a sufficient number of respondents are included in each of the categories.

Process/Format Selection

The team may choose a single format for writing items or it may use different formats for different areas of inquiry or categories of items. Team members often find that they need to practice writing some items in order to get a feel for the process before they can make definite decisions regarding the selection of a format or formats. Thus, there can be a great deal of moving back and forth between this section and the following one on item writing.

A suggested process would include these steps:{Note: go back and fix this section}

As a result of its work with the basic assessment formats, the team may even wish to create its own format. In such instances, the team usually blends several aspects from standard assessment formats to obtain the information that will meet its specific desires.

The format selection process which interacts with the next section on item writing concludes when the team has:

1. Tentatively decided on a format or formats to use in writing items.

2. Decided how the assessment techniques will identify which items are most important to respondents.

Process/Item Writing

The team may decide to write all the phase one instrument items as a team or it may divide into committees and have each committee write items for individual sections or areas of inquiry. If the latter course of action is chosen, the planning team as a whole would then review each section for revisions and approval. Whatever approach the team chooses, it is advisable for all the team members to practice writing some sample item and critique these before undertaking the task in full.

The practice will readily demonstrate the variety of ways an item may be expressed, and team members will soon identify those formats which best convey the subject matter on which they desire information. After several practice rounds of writing items and giving them a critique as suggested under Process/Format Selection, members will become adept at producing items that best seek the information desired.

1. Important considerations to keep in mind while writing items for the instrument include:

a. Writing clear and concise items to which respondents can easily reply. Each change of format should be accompanied by a set of equally clear instructions so that the respondents will understand what to do.

b. Never asking about two things in an item or include more than one environmental element. If two points are included, the result will be an uncertainty on which point the respondent has replied when the data are analyzed. For example:

Vandalism occurs because the place is rundown and students don't respect others' property.

SA(1) A(2) N(3) D(4) SD(5)

should be stated:

Vandalism occurs because students don't respect others' property.

SA(1) A(2) N(3) D(4) SD(5)

and:

Vandalism occurs because no one cares about keeping up this place.

SA(1) A(2) N(3) D(4) SD(5)

c. Use language, even some jargon when necessary, that the respondents commonly use or associate with the environment.

d. Be aware when adjectives are used in an item. Such words as "most," "few," and "usually" can load an item, diminish specificity, and thus, can bias results. For example:

It is usually quiet in the dorm.

SA(1) A(2) N(3) D(4) SD(5)

versus:

It is quiet in the dorm when I need quiet.

SA(1) A(2) N(3) D(4) SD(5)

Most rules are too restrictive.

SA(1) A(2) N(3) D(4) SD(5)

versus:

Regulations on decorating rooms are unreasonable.

SA(1) A(2) N(3) D(4) SD(5)

and:

I would not like twenty-four hour visitation every day.

SA(1) A(2) N(3) D(4) SD(5)

At other times, adjectives are needed for clarity or to establish the extent or degree to which the statement applies. For example:

Bull sessions about serious topics are a frequent occurrence. Most students living here are respectful of others' rights. I've had very few conflicts with the people living in my dorm. A good rule of thumb is to put yourself in the respondent's shoes to determine whether an adjective helps you or causes you a problem in replying to the item.

e. Phrase items in a manner that will prompt the respondents to think of specific environmental referents when they answer the assessment's second phase. For example the phrase "facilities in my room" will help respondents think of such things as lighting, bookshelves, and desk space when they come to explain their reply on a statement such as, The facilities in my room are adequate for studying.

f. Attempt to use behavioral terms or terms on which there appears to be wide consensus. Avoid technical terms or those which describe emotional states which may be interpreted differently by various respondents. For example:

I experience stress in the classroom.

SA(1) A(2) N(3) D(4) SD(5)

versus:

There are factors or situations in the classroom which make me feel uncomfortable and interfere with my learning.

SA(1) A(2) N(3) D(4) SD(5)

I need assertion training.

SA(1) A(2) N(3) D(4) SD(5)

versus:

I need help in learning how to stand up for myself so people don't take advantage of me.

SA(1) A(2) N(3) D(4) SD(5)

2. When the team has established a level of satisfaction in writing items, it will need to:

a. Seek advice from a consultant or team member knowledgeable in assessment techniques to assist in reviewing its practice items.

b. Determine if the format or formats initially agreed upon are usable.

c. Make a final decision regarding the formats it wishes to use for which sections or areas of inquiry.

d. Set up systematic approach for developing the instrument and writing items. This might include how the team will divide to write items.

3. When the team has completed writing items for the phase one instrument, it should be given a final review and 20 to 30 copies prepared for a pilot test implementation.

Process/Answer Sheets

Once formats have been decided upon and items are being written, it is important for the team to decide how to set up an answer sheet for the phase one instrument. It is necessary to have a clear format for the answer sheet to aid the respondents in giving their replies. It is also essential that the sheet be set up with the constraints of the available computer programs in mind. At this point, the team or a subcommittee of team members will need to determine:

1. The number of spaces (codes) available in the program being used for responses to any one demographic item.

2. Whether they will use keypunch or use a method to have the data read directly from the answer sheets.

3. A general plan for setting up the answer sheets to accommodate the items in the instrument.

The final setting up of an answer sheet will have to await the completion of the phase one instrument. Often it is necessary to consult with someone outside the team for this task, as it tends to be a fairly technical process.

Process/Instrument Data Analysis

It is also very important at this point in the process for the team to decide what information they want from the computer analysis of the data. The team will most likely need to work with their computer consultant to determine the appropriate program(s) that will provide the information they desire. Specific questions which must be answered include:

1. How will data be reported? (Reporting in percentages is a common and useful procedure.)

2. Will data be reported only for the total population or will the responses be broken down in terms of key demographic variables? (This latter procedure can be used initially or can be accomplished at a later time if so desired.)

3. Will any tests of statistical significance be required?

Discussion

Phase two of the ecosystem model's assessment is fundamental to its originality and to its later design processes. It is the information obtained from this part of the assessment that will be most useful to the planning team in developing environmental designs or changes. A form needs to be developed on which the respondents can write descriptions of what is happening in the environment that produces their perceptions of it and thereby provide the necessary environmental referents. A method for analyzing this environmental referent information must also be devised.

Three examples of ER forms are given below. The process suggested for developing the assessment's second phase is to devise the form on which respondents will record their ER information, then concentrate on developing a method for analyzing the information. This sequence is recommended because the devise or form that is created usually sets up the order in which ER information will be given, and, thus, can affect the design for analyzing the data.

When using the ER form in tandem with the phase one instrument, it is recommended that the respondents not be given complete details about the form until they have finished the instrument. Details about the form could become confused with instructions for the instrument, and the respondents might be tempted to bias their replies on the instrument in order to avoid having to answer many items on the ER form. However, it is fair and often essential to briefly inform respondents that they will be requested to give additional information on items.

Process/ER Form

Guides to follow in creating an ER form for phase two of the assessment technique include:

1. If an importance measure (those items considered most important by respondents) is to be obtained through the ER process, a method will have to be established by which respondents indicate importance on the ER form or through the selection of items for which they give ER information. Thus, when the data are analyzed the team will know which parts of the environment are most beneficial to retain and to change.

2. When a tandem approach is used, the ER form will need to establish a way to show how the respondent replied to items on the instrument. This is necessary because it is important to know whether the subsequent ER information stems from a highly negative or positive response.

3. If it is important to study ER data according to demographic information, the ER form will need to establish a method by which the respondents can provide the needed demographic information on the form.

4. The ER form will need to ask why the respondents replied as they did.

5. The ER form will need to ask what the respondents suggest to correct or improve the situation and what should be retained.

6. Give clear instructions for the ER form's use, preferably with illustrations.

7. After reviewing the example ER forms below, the team can either adapt one for its use or develop its own ER format. The subsequent ER form should then be prepared for a pilot-test implementation on some 20 to 30 respondents.

An ER form designed by Leland Kaiser, James H. Banning, and LuAnne Aulep for use in tandem with a standardized, commercial instrument, The Institutional Goals Inventory,* could be adapted for any instrument employing the Goal Statement format see Figure 2, p. 43).

In the case of this ER form, respondents are requested to review the first 90 items in the instrument and provide ER information on each item for which they have indicated a discrepancy of more than one-column space (see IGI format as illustrated in Figure 2).

The team will thus obtain ERs on those items for which there is the greatest discrepancy between "is" and "should be." However, there is no research to date which proves it necessarily follows that these items are the most crucial to respondents. In other words, a respondent might indicate that a high discrepancy exists, but the fact that it does exist may not be at all important to the respondents.

An importance measure could be built into this ER form by changing the way the respondents are to select items on which to give further ER information. Respondents could be asked to circle the numbers of those items that have the most importance for them as they answer the instrument, then instructed to use the ER form to give further information about the items they circled.

*The Institutional Goals Inventory, copyright AD 1972 by Educational Testing Service, published and distributed by the Institutional Research Program for Higher Education, Princeton, N.J.

[Insert Figure 2. Kaiser, Banning, Aulepp ER Form]

Another ER form originally designed by Barbara Peavey, Ursula Delworth, and LuAnne Aulepp, and subsequently revised through model applications, has been used in tandem with instruments employing the Likert scale format (see Figure 3, p. 45). Respondents are asked to circle the numbers of items that hold significant importance for them as they answer the initial instrument. Then the ER form is given and the respondents are instructed on how to provide additional information about the items they have circled. In this manner the ER process is used to establish the importance measure. In this procedure, it is suggested that respondents be given a minimum number of items on which they are expected to provide ERs.

[Insert Figure 3. WICHE Ecosystem Model ER Form]

An ER format developed by John Corazzini, Lois Huebner, and Susan Wilson at Colorado State University, eliminates reference back to the phase one instrument in order to complete the ER form. Pages of the ER form are wider than the pages of the instrument and can be attached behind each page of the instrument so that only a column of boxes is visible while the respondents answer the phase one instrument. Respondents answer the instrument according to instructions and then write the answer they have given in the corresponding ER form box that is visible (see Figure 4, below).

When respondents have finished the instrument, it is detached. The remaining ER form repeats each of the phase one instrument's items. Respondents can then complete these forms (see Figure 5, p. 47) according to instructions without having to refer back to the first instrument. The instructions for this ER form ask respondents to give information for certain answers in response to phase one items. A similar direction could also be used to ask respondents to write information on items that were most important to them.

[Insert Figure 5. Corazzini, Huebner, Wilson ER Form]

Process/ER Analysis Method

Just as the team needed to consider computer programs and capabilities for scoring and analyzing data in its development of the phase one instrument, the team must devise some method of analysis for ER data so that it can consider the resources it will need to implement the analysis. Often, students who could receive credit for conducting the assessment process and analyzing the ER data are recruited by the team. There are times the environment will have staff personnel who can be recruited. Whoever is recruited to help the team implement the assessment and assist with the ER analysis will have to be trained. Training cannot proceed until an ER analysis method is developed.

An ER analysis method has been devised for the WICHE ecosystem model ER form illustrated above. A similar method could be developed for any ER form by readjusting it to accommodate the needed demographic breakdown and the response format used in the instrument. The experience with ER analysis is based on work with only one model application. Planning teams may well devise alternate methodology more appropriate to their own uses.

The following steps comprise this ER analysis method:

1. The ER form is retained by the team or its implementers for scoring and analysis while the phase one instrument is sent to the computer for scoring and analysis.

2. The first step in analyzing the ER data is to take an ER item count. The ER item count entails a simple tally of the number of ER responses received per item on the initial instrument. When the team has selected an important demographic breakdown, the item count should be taken according to demographic category. Tally forms might look as follows:

Total ER Count

ER Count by Demographic Breakdown

3. After an ER item count has been tallied, the tallies are assembled and an ER item chart is compiled. This chart provides a global view of the total number of ER responses per item and per demographic category, if desired. The team then has a quick and readable summary of those items receiving the highest number of ER responses system-wide. If broken down into key demographic variables, the summary also provides a list of those items receiving the highest number of ER responses from specific groups. The chart might look as follows:

ER Item Chart

4. Using the information displayed by the ER item chart, the team selects those items that should be given an ER content analysis. Items with a high ER response overall are good candidates for analysis. If the responses are broken down into demographic categories, the team may want to rank order those items on which a high number of ERs were given by a particular group.

5. When the decision has been reached on which items are to be analyzed, the items can be grouped according to their parent sections or scales used in the instrument for each area of inquiry. Those analyzing the items should be assigned items within a particular scale or section. At this point the ER forms will have to be cut apart and grouped according to item number and scale. If important demographic variables are being scored and analyzed by computer, the team will have had to code the instruments and ER forms. Thus it will be necessary to write this code number on each ER response before the ER forms are cut apart and grouped. The team may want to photocopy the complete set of ER responses before cutting them apart. Then, if some are lost, the original set can be referred to.

6. The first step for those who analyze ER responses is to sort each item's ER forms according to the "Agree," "Disagree," and "Neutral" responses. This is done by looking at which response the respondent has circled on the ER form.

7. The person then takes all the "Agree" responses and reads the respondents' "Why" comments several times until it is possible to group similar "Why" comments into a few categories. In some instances, a single response may have to constitute a category, but the fewer categories that have to be made the better.

8. Then the person should develop a heading or name for each of the categories and briefly describe it.

9. Then tally the number of "Why" comments in each category.

10. And finally, record information on "Agree/Why." (See Figure 6, p. 5 for an example of the ER Content Analysis Sheet that was developed for this purpose.)

11. Once the "Agree/Why" comments have been categorized, tallied, and recorded, the "What" comments contained in each "Agree/Why" category are read. Again, when possible, similar comments should be grouped and categorized.

12. A tally is taken on the number of comments that make up each ''What" category. When it is impossible to group a comment, it should be listed as a category.

13. Data concerning the "What" comments are recorded under their appropriate "Agree/Why" categories. (See point D in the ER Analysis Sheet.)

14. The same processes of reading, grouping, categorizing, tallying, and recording comments is applied to the "Disagree" and "Neutral" responses on the item.

15. The entire process is repeated for each item that has been assigned to be content analyzed.

[Insert Figure 6. Environmental Referent Content Analysis Sheet]

Discussion

The team is now ready to pilot test their assessment technique. By conducting a pilot test, the team can determine how long it will take respondents to complete the assessment procedures, to identify items that require revision or deletion, and otherwise to obtain a feeling for the assessment's usefulness.

Unless the respondent population is quite small and cohesive, it is best to pilot test the assessment on its intended respondents. Using respondents from the target population far outweighs the possibility that a full-scale assessment might be jeopardized by knowledge of the testing procedure becoming widespread. In past applications of the model, neither the pilot test nor sequential testing sessions has resulted in the respondents shortchanging ER information. Respondents have been aware that additional information would be requested on items, but completion of the ER form by one set of respondents has not reduced the number of answers given by later respondents. The only time this hazard might become a serious consideration in the selection of the pilot test population would be in the case of a small, cohesive group. In this event, the team might want to locate as similar a population as possible for purposes of the pilot test.

Having the assessment instrument answered by some 20 or 30 respondents should provide the team with sufficient information to refine and make final their assessment technique. The pilot test will also provide ER information for use in training people to conduct the team's ER analysis In this manner the information received in the pilot test can serve to test out the team's method of ER analysis.

Process

Important guides for the assessment technique's pilot test should include:

1. Selecting a small random sample from the target population or as similar a population as possible.

2. Timing how long it takes respondents to complete the assessment procedures.

3. Debriefing the respondents on their reactions to the assessment, including their comments on whether language used in the assessment was easily understood by them; what word substitutions they might suggest; what, if any, items seemed ambiguous to them and why; what, if any, items seemed irrelevant to them and why; and whether the instructions for the assessment procedures were easily understood and followed.

4. Arranging an expression of appreciation such as lunch or cookies and coffee for pilot test respondents.

Discussion

Based on results from the pilot test, the team can make final adjustments on its assessment technique. If the assessment procedure is too long, then the team must make modifications. It is recommended that testing time not exceed 90 minutes; if adequate information can be procured in an even shorter time span, this is preferable.

The team does not necessarily have to send the responses of the pilot test on its phase one instrument to the computer for scoring and analysis. It will conserve resources and be much better if the team personally reviews them because the important information at this point will be the respondents' reactions and comments on the instrument's items and whether these items are prompting good ER responses. The latter are determined by comparing the ER responses to their parent item on the instrument. This can be facilitated by retaining the instruments so that each team member will have a copy from which to work.

The cost and services needed to prepare the instrument and form for implementation can be given realistic estimates after the team has made final its assessment technique. Team members who have been informing those persons in a position to provide these resources about project activities and progress should now confirm that the final estimates are agreeable and able to be met.

Process

To complete its work on developing an assessment technique, the team will need to:

1. Review all results of the pilot test.

2. Rewrite ambiguous items.

3. Rewrite items that do not appear to elicit good ER information.

4. Cut items considered in the final analysis to be not very important

5. Change wording as may be indicated.

6. Rewrite any ambiguous instructions.

7. Re-do any part of the answer sheet that was not clear to respondents.

8. Readjust a format if several respondents had trouble using it.

(Another important indicator for the need to adjust a format is obtaining consistently unusable information in response to it.)

9. Determine how the instrument and/or procedures can be trimmed if the testing period exceeds 90 minutes or the time available for testing.

Suggestions for this might include:

• Asking half of the respondents to give ER information on the first half of the instrument, and asking the other half to give ER information on the remainder of the instrument.
• Deleting one format in the phase one instrument if the format contains few items that could be rewritten for another format.

10. Confirm with those providing project resources, such as money and/or services, that the estimates for those resources are agreeable and can be met.

When the final adjustments and/or modifications of the assessment technique have been made, clean copies of the corrected instrument, answer sheet, and ER form should be made for the team's final review and approval. If no further corrections, additions, or deletions are made, the instrument, answer sheet, and ER form can be proofread and sent out for printing. If changes are made, then the instrument and ER forms need to be corrected and proofread. The team is now ready to proceed to Stage IV processes; these involve administering the assessment technique and analyzing its results.