Follow

Evaluations from A to Z

Best Practices
Before you get started, these are some of the tips we recommend as you get your feet wet.  Few parts of Arbiter are more exciting to the nerds among us than evaluations.  Having the numbers come pouring in helps shed light on things that otherwise may have gone unnoticed.

  1. Start simple! 
    Evaluations can be fine tuned to match many needs.  By starting simple you can see what works and doesn't work and adjust it the following season.  This ensures you get value out of evaluations from the start.
  2. Create separate evaluations for contacts
    It takes more work to create separate evaluations, but creating a separate evaluation for contacts allows you to compare an official's performance as viewed by the schools and from their peers.
  3. Copy evaluations for following seasons
    Instead of having an evaluation run forever, put a start and end date on an evaluation.  When the next season rolls around, copy that evaluation and adjust the dates.  This allows you to ask the same questions but gather data from different time periods.  After a few seasons you will have a great perspective on how the group and individuals are progressing.
  4. Use categories
    Categories allow you to group criteria together.  If you have related items, you can see how the group and individuals are doing in a general category.  Are they amazing on the field but don't handle the coach, players, and crowd well?  Do they know the rules like the back of their hand but aren't always positioned to make the right calls?

Assigners

Officials

Contacts

Assigners

Getting Started

Settings

Parameters for Completing and Viewing Evaluations Reports

Building Criteria

Completing Reports

Viewing Responses

 

Getting Started

To set up the forms for the evaluations:

  • Click the USERS tab
  • Make sure to be on the OFFICIALS sub-tab
  • Click the Evaluations link on the left side of the screen

 

(back to top)

Settings

Creating an evaluation report:

  • Click on the green add button.  This will bring up the screen with evaluation report settings, reporting permissions and criteria to build a custom report. First fill out the settings. 

Name: Give a customized name to the specific report (e.g. Peer Review, Coaches' Evaluation).

Score Range: The score range you want your report to have.

Requires Comment: If the evaluator gives a specific score (ex. 2) they will be required to comment and give a reason for that score. If you want to require comments all the time, the number should be whatever the highest number in your score range is.

Publish Responses #: Once this number of responses has been entered, the evaluation report will be shown to those that are allowed to view it. At this point a “responses” link will appear to the officials and/or sub-administrators that have been given permission to view them. All responses will be shown to viewers anonymously.

Sports & Levels and Positions: Select the sports and levels that this report will apply to. This allows different reports games of different levels. It also distinguishes between reports for different sports. After you've chosen the sport(s) and level(s), choose the position(s) you want the report to apply to.

Use Sub-Categories: Each of the ratings criteria can be separated into sub-categories. This allows the assigner to group common criteria and thus determine weights better for specific areas.

Back to Assigners

(back to top)

Parameters for Completing and Viewing Evaluations Reports

Evaluation report can be completed by:

Set the parameters as to who is allowed to fill out an evaluations report. 

  • Admins
    Allows admins on the account to complete evaluations.  Because admins complete the evaluations from the games and assignments screen, they can only complete evaluations on games they can see (scope, global filter, etc. will limit those games).  Admins can evaluate multiple games that happen at the same time as well.
  • Evaluators
    An evaluator can be anyone that is put on the game for the purpose of evaluating only. Evaluators are created from the sports/levels screen. Edit the sport that you want an evaluator for. Create a new position or edit one of the existing ones and mark it “observer”. This allows you to rank certain officials to be evaluators. 
  • Lead Officials
    A lead official is defined by the preference you have set.  It can be one of two defaults:  The official in the first listed position or the official who is highest ranked in the first listed position.  Admins can choose the lead official on a game by game basis by editing the slot and marking it as the lead official.  Each game is limited to a single lead official.
  • All OfficialsThis allows officials to complete evaluations on each other.  Since officials can't evaluate themselves, if an evaluation only has the plate umpire selected to evaluate the plate umpire won't see an evaluation for that game (unless there are multiple evaluations). 
    There is no permission or preference for which officials can complete evaluations.  The permission to 'Select games to evaluate' is to allow evaluators to find officials to evaluate.
  • Team Contacts
    You can allow your team contacts to complete evaluations on officials.  If you don't want all team contacts to evaluate officials, go to Assigning and then the Teams tab.  Open the contacts for a team and only check the Eval checkbox for contacts that should complete evaluations.  By default, contacts can complete evaluations.

Note:  No one can evaluate an evaluator.  If you have an evaluator listed on a game they will not appear when completing evaluations.  If you receive a notice that an official can't be evaluated, check to make sure that the position didn't get marked as an evaluator (observer) by accident.  For example, occasionally we'll see a Referee marked as an evaluator so they don't appear.

Responses can be viewed by: Officials can be given permission to view responses that were made regarding them. You can also allow sub-admins to view the responses.

Scoring Description: Describe what you want each individual rating to count as.  This will explain to the evaluator what would be expected of the official to receive a five, four, etc. For example, 5=Excellent, 4=Above Average, 3=Average, 2=Poor, 1=Unacceptable

At this point click the Build Criteria button. This will automatically save the report up to this point. 

 

Back to Assigners

(back to top)

Building Criteria

Build Criteria: These criteria are the points of officiating that you want the officials to be evaluated on.

Entering criteria: Click on the green add button to add a criterion. This will pull up its fields. First enter the name. Then enter the standard, which is teh specific thing an evaluator should be looking for to rank the official. For example, the name of the criterion may be “Appearance” and the standard would be “Has proper, clean, and well fitted uniform, black shoes, etc.” The weight determines how much each criterion will affect the overall score. Make sure that the weights of all of your criteria add up to 100%.

Categories: When completing an evaluation report the criteria will appear grouped in the categories that they are attached to (This feature will only appear if the “Sub-categories” box was checked at top of page). 

 

Back to Assigners

(back to top)

Completing Reports

Where to find Evaluation Reports

Go to the games screen and click on the slot of the game to be evaluated. When the slot opens you will see an A+ on the right side of the slot information. Choose the appropriate report from the pull down menu and click go.

How to complete reports

Verify that all of the information is correct. Using the pull-down button, insert a number into each of the criteria boxes. To view the criterion’s standard click on the criteria name. Insert comments as needed to the right. When each of the score areas is filled write any additional comments at the bottom. To send the report to the assigner check the submit box. If you would like to return later to finish the report choose Save.

 

Back to Assigners

(back to top)

Viewing Responses

To view responses to any evaluation, simply return to the USERS tab and click the Evaluations link. On the right side of the screen is the "Responses" column. Click on the envelope of the desired evaluation report. This will display a listing of officials and their average scores from every submitted response. To change the sort order click on the column headers. To view the average score for each criterion click on the number under “overall score."

To view an individual's reports click on the numbers in the "# Responses" column. This will display all of the reports submitted for the official and the evaluator who submitted the report. To view an individual report click on the score on the right of the page. If there is an issue the assigner may unsubmit the report, which sends it back to the person who submitted the report for revision and resubmission. The official then will have the opportunity to either completely redo the report or just modify it. The assigner can change the scores or comments from the report view page.

You can also print out a report of all the responses that have been made for a certain evaluation. Simply click the Responses link on the left side under the "Reports" section. Choose your export format, then check which Report Type you want. It must be either "Individual Responses" or "Individual Summary."

Evaluators:To view the average ratings of an evaluator change the pull-down button on the left side of the screen from officials to evaluators. This shows the average score that this person has rated on others. By clicking on the number under "#Responses" the assigner can see all the reports that a person has submitted and the score associated with each. An individual report can be viewed by clicking on the score on the right of the page.

 

(back to top)

Officials

Completing Reports

Viewing Responses

Completing Reports

Where to complete reports:

  1. Once you log in click on the EVALUATIONS tab on the top of the page.
  2. Click the report that needs to be completed. 
  3. Select the date range. The system will look for games in that date range that need to be evaluated. By default, the date range will be set for the previous week. To change the dates type in the correct dates in MM/DD/YYYY format and then click GO.
  4. Check the Show All checkbox to view those that have been completed already. 
  5. To complete a new evaluation or to view one previously submitted click on the A+ on the right side of the evaluation.

How to complete reports:

  1. Verify that all of the information is correct. 
  2. Using the pull-down menu, insert a number into each of the criteria boxes. 
  3. To view the criteria’s standard (the standard is an explanation of each criteria) click on the criteria name. 
  4. Insert comments as needed to the right. When each of the score areas is filled write any additional comments at the bottom.
  5. Click Submit to complete the report and send it to the assigner. 
  6. Click Save to save the report to be completed at a later time. This report will be available under the Evaluate officials link later with the A+ in gold. If the report is submitted the A+ will be green.

 

(back to top)

Viewing Responses

  1. Click on the envelope under the "Response" column. This will bring up a list of the different reports that the official has been evaluated on. To view the details of individual responses click on the numbers under “# Responses” and then click on the magnifying glass to see details. All responses are listed anonymously. To see the average score for each criteria click on the number under “Overall Score” from the View Responses page. This will display the average score from all submitted responses for that official for the specified report. Clicking on the averages will display the comments made from that criteria. Clicking on the criteria name will display the standard for that criteria. A link is provided at the bottom right with the overall average score for the report which displays the overall comments.

 

(back to top)

Contacts

Where to Complete Evaluation Reports: Click on Evaluate officials from the start page. Select the report that is to be completed. A list of the forms will appear that fall within the specified date range (default is the previous week). To change the date range enter the beginning date and ending date and click GO. To evaluate an official click the red A+ on the right side of the page. 

If a report has been saved or returned by the assigner for revision the A+ will be gold. If the report has been submitted the A+ will be green.

Note: By default only unsubmitted reports will display. To see all reports check the Show All checkbox to the left.

How to complete reports:

  1. Verify that all of the information is correct. 
  2. Using the pull down arrow, insert a number into each of the criteria boxes. To view the criteria’s standard (the standard is an explanation of each criteria) click on the criteria name.
  3. Insert comments as needed to the right. When each of the score areas is filled write any additional comments at the bottom. 
  4. Click Submit to complete the report and send it to the assigner. 
  5. Click Save to save the report to be completed at a later time. This report will be available under the Evaluate officials link later with the A+ in gold instead of read indicating that it had been saved previously.

(back to top)

 

 

Was this article helpful?
0 out of 0 found this helpful
Have more questions? Submit a request

Comments