CONTENTS

Executive Summary
1.Introduction
2.Course Design

3.Evaluation

4.Review
5.Conclusions
6.References
7.Appendices

Case Studies

3 Evaluation Strategy

Planning the evaluation

Successful evaluation begins with a plan (Tessmer, 1993) and to do this it is essential to begin by considering the purpose of the evaluation. When this has been decided, it is easier to identify when to evaluate and who should be involved.

Purpose of the evaluation

Within the context of course development and course delivery, there are many reasons why evaluation is important. In designing, developing and delivering this staff development course, the following objectives were identified:

• to identify the training needs of the intended participants to ensure that the course content and delivery meets their needs;
• to identify aspects of the course that need modification - this will lead to a more efficient and effective course; and
• to measure the extent to which the aims and objectives have been met - this will help to establish the appropriateness and validity of the course.

Phases of the evaluation

Based on these objectives the following three evaluation phases were identified: pre-course evaluation, formative evaluation and summative evaluation.

Pre course evaluation conducted during the analysis of the training requirements. This comprised a training needs analysis questionnaire which helped establish the scope of the course, both in terms of the course itself and the way in which it will be developed and delivered.

Formative evaluation conducted during development and delivery of the course. Such information informs the revision and modification of the course.

Summative evaluation carried out on completion of the course. This type of evaluation is usually designed to inform a review of the course before it is used again.

Each phase has the same broad objective: to produce the most effective course, right from the beginning of the development process.

Pre Course Evaluation

A number of different issues were considered in the training needs analysis including:

the frequency and nature of C&IT use among the target group. This helped establish the level of experience with IT, video conferencing and other CMC technologies such as email and the web;

the teaching methods and media used - these questions ensured that the training would focus on what the participants currently do in their teaching and what they hoped to do when using video conferencing;

the participant's perceptions of video conferencing - this gave an indication of the interest and motivation regarding video conferencing; and

in the case of the technical staff, the nature of technical support provided to staff in their teaching.

This information was gathered using a questionnaire (see Appendix 1 for the Training Needs Analysis questionnaire and Appendix 2 for detailed results of the analysis).

Results and Discussion

Eighteen out of twenty participants responded to the questionnaire.

The first category of questions were designed to assess the level of IT experience. The results, in Table 3 show that the majority had used a computer often or very often. However, apart from two, who use the web often or very often, the majority occasionally or never used CMC technologies and none had experience in using video conferencing.

How often do you use:   never   occasionally/sometimes  often/very often 
computers                 0%               23%               78%
e-mail                   56%               17%               11%
web                      61%               23%                0
video conferencing       89%                0%                0

The reasons for using the various technologies varied as can be seen in Table 4. The majority however use the computer for administration with a smaller number, 53% using the computer to support their teaching.

Technology     Reason for Use	
Computer       Teaching - lecture preparation and presentation.      50%
               Admin - related to teaching &	
               General - memos, general correspondence, spreadsheets 94%
Email          Communication with colleagues or students             13%
Web            Resource accessing information, researching topics    13%

69% of the respondents indicated that they use computers for the preparation of lectures. The types of usage included preparation of acetates, handouts, lecture notes and MS PowerPoint slides.

The second part of the analysis looked specifically at the teaching methods and media currently used. The results show that staff use a range of methods (Table 5: Teaching Methods) and media (Table 6: Teaching Media) in their teaching.

Teaching method       currently use    would like to use
lectures                  72%               12%
tutorials/group 
discussions               72%               28%
practicals                56%               11%
case studies              33%               11%

Interestingly however is that most respondents would only like to lecture when using video conferencing to deliver their teaching and only a small number are interested in using tutorials, group discussions, practicals and case studies.

Furthermore, apart from computer presentation, which more participants would like to use with video conferencing, they plan to use the other media less.

Teaching media           currently use   would like to use with VC
blackboard/white board         72%                 22%
flipchart                      72%                 17%
OHP                            72%                 56%
slides                         33%                 22%
video                          72%                 28%
computer presentation           6%                 56%

The purpose of the latter part of the questionnaire was to assess the participants' perception of video conferencing. From this information, their level of interest and motivation could be gauged. The results showed that they already had an appreciation of the benefits of video conferencing, especially those related to cost effectiveness. Regarding what they perceived as being the difficulties with video conferencing, most were concerned with how it may restrict their teaching. (Please see Appendix 2 for more detailed responses.)

The results of the questionnaire were very useful in informing the design of the course and helping assess the level of C&IT experience. Although the participants occasionally or never use CMC technologies, they all had experience of using a computer and 61% indicated that they already used a computer to prepare their teaching. More particularly, it highlighted the methods and media that they currently use and would like to use in video conferencing. It is interesting to note, that although a significant number used practicals, case studies and tutorial or group discussions, the most common approach, which they wanted to use with video conferencing, was lecturing.

Formative Evaluation

A number of different types of evaluation were carried out during the development and delivery of the course, including:

• expert review of the course outline; and
• debriefing with all trainers following each session.

Expert review of the course outline

A range of both internal and external experts were consulted to provide comments and feedback on three main issues:

• the content:
• is it appropriate to the target audience (both support and teaching staff)?
• does it address the appropriate issues?
• is it free from any important omissions?
• the sequence:
• are the issues being addressed in the appropriate order?
• the teaching methods:
• are the teaching methods appropriate?

It is preferable to have more than one expert evaluating the content to ensure these aspects are given a balanced evaluation. A further advantage of using expert reviewers, is that not only can they identify problems but they can also offer advice on how to resolve them (Phillips, 1997). Table 7 shows the review panel that was used.

Manager of the Teaching Support Group, Computing Services

Advisor/trainer within QUB on use of technology in teaching and learning; user of video conferencing in teaching.

Lecturer in ICBL

User of video conferencing in teaching.

Staff development officer, QUB

Staff trainer, specialising in academic training, for example, teaching skills etc.

Head of Division in the School of Nursing and Midwifery

Business and Contracts Manager

The clients - representing the target group. Some experience in the use of technology in the School and the potential use of video conferencing.

TALiSMAN Co-ordinator

Co-ordinator for the staff development strand of the Use of MANs Initiative (UMI).

Staff Development Co-ordinator and a member of the Centre for Learning and Assessment, Robert Gordon University, Aberdeen

Involved in staff development. Design and administer courses on video conferencing for staff.

Debriefing with all trainers following each session

After each session, the trainers met to discuss and evaluate the session. Each session was discussed in terms of:

• any problem(s) identified;
• observed reactions of the participants;
• the appropriateness of the method used;
• scheduling of each topic - time taken; and
• general impressions regarding content and pacing.

This enabled any necessary changes to be made to the subsequent sessions and also to plan updates for future courses.

Summative Evaluation

Two types of evaluation were carried out:

• an evaluation of participant's reaction to the course; and
• an assessment of the learners' performance.

Participant's Reaction

This type of evaluation provides an analysis of the participant's attitudes towards the course. When the participants completed the course, they were asked to evaluate, via a questionnaire:

• the content of the course - did the instruction cover the right material; did they feel that they had learnt and understood the content;
• the training methods used on the course;
• the resources, that is, facilities and refreshments;
• the facilitators: their professionalism and presentations;
• beneficial and non beneficial aspects of the course - what had they found to be particularly clear, helpful interesting and what had they not; and
• any recommendations for change to the course.

Results and Discussion

In general, the results of the summative evaluation to the course were very positive. The following highlights some of the findings, but please see Appendix 4 for detailed results of the evaluation. The evaluation questionnaire used is in Appendix 3.

Regarding the content of the course, the participants agreed or strongly agreed with the comments relating to content of the course, see Table 8.

Statement                     Strongly Agree
The content was relevant to 
my training needs                  100%

The purpose and objectives 
of each session were clear          83%

I now have a better understanding
 of VC terminology                  83%

In evaluating the course, it was important to find out what the participants felt about the methods that were used throughout the four sessions. The results in Table 9 show that there was a favourable response to the materials. When asked about the balance of teaching methods, only two thirds felt that the balance was good (Table 9).

There was a good balance 
between types of teaching 
(presentation, video, group 
discussion, hands on)	           33%	          67%

The material was presented 
clearly and in a logical sequence  17%            83%

Appropriate use was made of visual 
aids, materials and handouts	   17%             83%

This was again reflected in the next section of the questionnaire as the participants indicated that the practice sessions and the last session (opportunity to present using video conferencing) were the most useful. In addition, to providing them with the opportunity to practice what they had learnt, they also felt that it gave them an indication of their own ability and confidence in using the facilities.

None identified any of the sessions as being the `least beneficial'. However, two of the participants felt that the first few sessions could have been used for practicing with the equipment. A suggestion was made to integrate some of these sessions to leave more practice time with equipment in later sessions.

Lastly, the respondents identified a need for further training in a) use of Microsoft PowerPoint and b) video conferencing - to try out more imaginative approaches to using the facilities.

Session 4 provided an opportunity to assess the performance of the participants in making a presentation using video conferencing. This did not address all the issues covered on the course but provided an opportunity to assess the participants:

• verbal presentation skills;
• appropriate use of non-verbal communication;
• inappropriate or distracting behaviour;
• planning and preparation of their teaching material; and
• skill in using the technology, for example switching between cameras or using the document camera.

The participants were asked to plan and prepare a five minute presentation based on material they currently use in their teaching and to include any media, objects or teaching aids that they would typically use. The aim of this exercise was to provide the participants with an opportunity to apply what they had learnt from the previous sessions and give them the opportunity to deliver a session in a supportive environment.

The participants delivered their presentation to a local and remote audience and trainers assessed their performance (both locally and remotely). An observer from the Enterprise Unit in QUB was also present to comment on the presentation and delivery of the teaching. Performance was assessed via a checklist (please see Appendix 5). At the end of the presentations, general feedback was provided to all the presenters and the experiences of the participants were discussed.

This exercise did not enable us to assume that a successful performance was due only to the previous training sessions but it did however provide an opportunity to:

• assess some aspects of the participants' performance thus ensuring that they could use the technology effectively;
• provide feedback to the participants on the preparation of their teaching material;
• comment on the presentation of the teaching; and
• identify any weaknesses or mistakes which would indicate a need to change or modify the course.

Note: at time of writing this report, the summative evaluation is not complete as only half of the participants have completed session 4.