Region VI ADA Training Evaluation Report

September 2013
Prepared by Fredrick E Menz and Andy Winnegar

Background

This evaluation was conducted on training and technical assistance provided to federal contractors in Region VI during FY2012-2013. The training was designed from a regional training and needs assessment conducted in FY2011-2012 (Menz & Winnegar, 2012) in which it was determined that customized, face-to-face training be provided to staff in Community-Based Rehabilitation Organizations (CBROs) toward competencies indicated in the U.S. Department of Labor standards of compliance with Section 503 of the Rehabilitation Act (Section 503) 1 and the department's Office of Federal Contract Compliance Programs (OFCCP's) proposed rules to would strengthen the affirmative action requirements established in Section 503 of the Rehabilitation Act of 1973 obligating federal contractors and subcontractors to ensure equal employment opportunities for qualified workers with disabilities.

Training Methodology

Each customized training programs was based upon the identified needs for Region VI (Menz & Winnegar, 2012). Regional needs were broken down by state and participating CBRO and specific competencies were selected in conjunction with the federal contractors at each CBRO. Training was then designed to address those competencies and used training and technical assistance materials, aides, and guidelines developed by the Southwest ADA. Case-studies used in conjunction with lecture and discussion drew upon types of work, state, and disabilities present in the host-CBRO.

Training was offered to all 42 CBROs that were federal contractors in Region VI, which includes New Mexico, Louisiana, Texas, New Mexico, and Arizona. A total of five trainings were attempted in New Mexico where there was originally participation in the regional needs assessments. Four training events were completed with 61 employees of the two federal contractors in New Mexico. Training sessions ranged in time between three to four hours. All training sessions were conducted by Mr. Andy Winnegar on-site at the host-CBRO.

Training Evaluation Methodology

Originally, the training evaluation was to use an on-line instrument that linked to the previously designed ADA competency database and a pre-post testing design. However, there was a low level of interest in the on-line training on ADA compliance from respondents in New Mexico. It was felt that all trainees would not likely have easy access to an Internet-connected computer determined that a post-test only paper-and-pencil tool would be more likely to yield completed training evaluations. All efforts were made to reduce evaluation burden on trainees as part of the planning with CBROs on competencies to achieve, length of training, and delivery method.

Copies of the questionnaires are included in Appendix A.

The training evaluation questionnaire addressed the CBRO-identified training competencies for each customized program and was administered at the conclusion of the training session. Questionnaires were collected by the trainer from 60 of the 61 trainees and data were recorded on a spreadsheet for later analysis.

A minimal set of demographics were collected to describe the trainees and explore differences in impact of training on competency development: Length of employment at CBRO and on federal contracts, highest education completed, and work title.

Each trainee answered factual and true-false questions to estimate their level of knowledge and skill attained from the training. Ten competency questions were prepared for each training program by the trainer. Counts of correctly answered questions were data available for evaluating the training.
Quality of training was estimated using a 5-point Likert scale (Strongly agree, agree somewhat, neutral, disagree somewhat, strongly disagree) against three statements:

  • The presenter had adequate knowledge of training.
  • The content was organized to promote learning.
  • I would recommend this training to someone else.

The rate of trainee competence (out of 10 items) and perceived quality of training (out of 3 items) were the metrics for evaluating the effects of the ADA trainings on CBRO contractor staff and perceived quality of training. As there were neither pre-tests nor comparison groups, only the consistency of competence and training quality across the CBROs could be judged with these data.

Training Evaluation Results

Four customized on-site training programs (3-4 hours in length) were conducted at two CBROs in New Mexico. Both CBROs successfully completed their OFCCP compliance review not too long before the training. There were two four hour training sessions at each of the two CBRO.

A total of 60 questionnaires were returned by the 61 trainees with little missing data, most of which were among responses to demographic items. Descriptive analyses (percent’s, means, and standard deviations) were computed to describe the 60 trainees, their training outcomes (competence, rated quality).

Comparative analyses (ANOVAs, correlations, Chi-Squares) were conducted to determine whether assessed level of knowledge and skills at the end of training and rated quality of the training were consistent between the two CBROs and among combinations of trainees based upon their job roles, education, and employment longevity.

Appendix B provides results from the statistical analyses (available upon request).

Trainee Characteristics

Approximately 52 percent of trainees described their job role as in administrative (CEO, contract management) or human resources (HR, support, recruitment) and 40 percent were directly involved in delivering on federal contracts (training, supervision, placement). The highest educational level completed by trainees was split between high school graduate/GED (26.7%), some postsecondary education, including college classes, apprenticeships, certificates, 2-year degrees (31.7%), and those who completed a bachelors or masters degree (35%). On average, trainees had been working for the CBRO for 7.7 years and, of the 83 percent reporting they worked on federal contracts, their average tenure on federal contracts was 5.4 years.

Attained Competencies

The average number of competencies items answered correctly by trainees at the completion of training was 5.45, which translates to a 54.5 percent competency rate following 3-4 hours of training for each participant. A negative correlation was found between number of correct answers and length of employment at the CBRO (r=-.283, p=.034), suggesting that newer employees gained more from the training. The two CBROs differed in the average number of items answered correctly (F=4.674, p=.035), with CBRO1 (N=15, Mean=6.27) answering 1.09 more items correctly than CBRO2 (N=45, Mean=5.18). Given that items were not the same for both trainings; it is not clear whether this difference is due to item differences or to trainee response to training.

Quality of Training

Trainees reported generally high regard for the presenter's knowledge (85.2% strongly or agree), the organization and presentation of content (78.9%), and would recommend the training to others (75.9%). No significant relationships were found between quality ratings (rescaled to 5-points scale with 5=Strongly Agree, etc.) and employment tenure (with CBRO, on federal contracts). Average ratings did not differ, either, between CBROs, by job role, or by education.

Conclusions and Recommendations

Benefits of Training

  1. Given the generally high marks given to the quality of the training, the trainer and subject matter were appreciated by the trainees. However, the training yielded only a moderate level of competence among the trainees.
  2. A 54.5 percent competency rate following training customized to audience need does not speak well to whether 3-4 hours of customized training can develop competencies expected of federal contractors for ADA compliance. An average level of competence above 80% would be not appear to be too rigorous a standard for trainees from short-term training designed directly at specific competencies.
  3. Assuming that the competency items are representative of the ADA related knowledge and skills CBRO contractors need to meet compliance requirements, the results from this small sample of training raise questions about how knowledgeable are key staff engaged in administering or delivering on the federal contracts. Their outcome scores appear more like the average score expected from experienced contract workers (averaged 5.4 years) on a test prior to competency directed training.
  4. Given the 1 point (or 10%) slight difference between CBROs and that younger staff tended to score higher, the question of the causes for these variations should be considered: Is there resistance to training by older workers and/or was there low interest/participation in spending time in training when the organizations had been determined to meet ADA compliance requirements?
  5. Further, given that the differences in number of items correctly answered by the two CBRO staff may be either due to item differences or to trainee response, the above concerns with level and sufficiency of competency demonstrated by the trainees should be viewed for their probative value and not as an aspersion on the preparedness of the staff at either CBRO.
  6. The 54.5% competency level for staff at ADA compliant CBROs suggests two future steps in training evaluation: First, verify the correspondence between training objectives (which competencies are addressed) and competency items used to assess trainee knowledge and skills. Second, conduct an independent analysis of wrong answers (e.g., which alternatives chosen, reasons for giving an answer) to determine whether item content and structure are sound and representative of knowledge and skills expected for each ADA competency.

CBRO Interest in Training

  1. There is clearly limited interest in ADA compliance training in Region VI as shown by the following:
    1. First, thirty-seven CBROs with federal contracts were surveyed for needs in 2011-2012 and 15 useable surveys were returned from eight CBROs, for an effective response rate of 21.6 percent, and 60% of surveys came from three CBROs in New Mexico.
    2. Second, trainees in the 2012-2013 training came from two federal contractor CBROs that had served on the needs assessment project advisory committee and were the largest federal contractors in New Mexico.
  2. The extent to which trainees considered the training either redundant post-OFCCP review or not a good use of their work-time is unknown and may even be masked in the ratings of the quality of training.

Alternatives for Developing CBRO Competence

  1. The level of need for ADA compliance training in Region VI remains unknown. Need for training or technical assistance is not likely to be given thought to or considered until a CBRO federal contractor may not be in compliance or has failed their OFCCP review. Demand for training or technical assistance will likely occur when a CBRO has to raise overall competence to ensure compliance or when a direct request is made to the ADA center for training or technical assistance to ensure they pass the CBRO passes its next OFCCP compliance review. Therefore, a more profitable direction may be to create/redevelop technical assistance materials that address historic and recently added CBRO specific intents, expectations, and requirements.

References

Menz, F. E., and Winnegar, A. (2012) Training Need Report. Unpublished report prepared for the Southwest ADA Center at the Institute for Independent Living and Rehabilitation Research, University of Texas, Houston, TX.

Appendices

End Notes

1.  Title 41: Public Contracts and Property Management. PART 60-741—Affirmative Action and Nondiscrimination Obligations of Contractors and Subcontractors Regarding Individuals with Disabilities..