|Main Report and Appendices available in Acrobat format|
Using Video Conferencing to Support Distance Learning
A Staff Development Course
3 Evaluation Strategy
Planning the evaluation
Successful evaluation begins with a plan (Tessmer, 1993) and to do this it is essential to begin by considering the purpose of the evaluation. When this has been decided, it is easier to identify when to evaluate and who should be involved.
Purpose of the evaluation
Within the context of course development and course delivery, there are many reasons why evaluation is important. In designing, developing and delivering this staff development course, the following objectives were identified:
Phases of the evaluation
Based on these objectives the following three evaluation phases were identified: pre-course evaluation, formative evaluation and summative evaluation.
Pre course evaluation conducted during the analysis of the training requirements. This comprised a training needs analysis questionnaire which helped establish the scope of the course, both in terms of the course itself and the way in which it will be developed and delivered.
Formative evaluation conducted during development and delivery of the course. Such information informs the revision and modification of the course.
Summative evaluation carried out on completion of the course. This type of evaluation is usually designed to inform a review of the course before it is used again.
Each phase has the same broad objective: to produce the most effective course, right from the beginning of the development process.
Pre Course Evaluation
A number of different issues were considered in the training needs analysis including:
the frequency and nature of C&IT use among the target group. This helped establish the level of experience with IT, video conferencing and other CMC technologies such as email and the web;
the teaching methods and media used - these questions ensured that the training would focus on what the participants currently do in their teaching and what they hoped to do when using video conferencing;
the participant's perceptions of video conferencing - this gave an indication of the interest and motivation regarding video conferencing; and
in the case of the technical staff, the nature of technical support provided to staff in their teaching.
This information was gathered using a questionnaire (see Appendix 1 for the Training Needs Analysis questionnaire and Appendix 2 for detailed results of the analysis).
Results and Discussion
Eighteen out of twenty participants responded to the questionnaire.
The first category of questions were designed to assess the level of IT experience. The results, in Table 3 show that the majority had used a computer often or very often. However, apart from two, who use the web often or very often, the majority occasionally or never used CMC technologies and none had experience in using video conferencing.
How often do you use: never occasionally/sometimes often/very often computers 0% 23% 78% e-mail 56% 17% 11% web 61% 23% 0 video conferencing 89% 0% 0Table 3 Level of C&IT Experience
The reasons for using the various technologies varied as can be seen in Table 4. The majority however use the computer for administration with a smaller number, 53% using the computer to support their teaching.
Technology Reason for Use Computer Teaching - lecture preparation and presentation. 50% Admin - related to teaching & General - memos, general correspondence, spreadsheets 94% Email Communication with colleagues or students 13% Web Resource accessing information, researching topics 13%Table 4 Reasons for using the technologies
69% of the respondents indicated that they use computers for the preparation of lectures. The types of usage included preparation of acetates, handouts, lecture notes and MS PowerPoint slides.
The second part of the analysis looked specifically at the teaching methods and media currently used. The results show that staff use a range of methods (Table 5: Teaching Methods) and media (Table 6: Teaching Media) in their teaching.
Teaching method currently use would like to use lectures 72% 12% tutorials/group discussions 72% 28% practicals 56% 11% case studies 33% 11%Table 5 Teaching Methods
Interestingly however is that most respondents would only like to lecture when using video conferencing to deliver their teaching and only a small number are interested in using tutorials, group discussions, practicals and case studies.
Furthermore, apart from computer presentation, which more participants would like to use with video conferencing, they plan to use the other media less.
Teaching media currently use would like to use with VC blackboard/white board 72% 22% flipchart 72% 17% OHP 72% 56% slides 33% 22% video 72% 28% computer presentation 6% 56%Table 6 Teaching Media
The purpose of the latter part of the questionnaire was to assess the participants' perception of video conferencing. From this information, their level of interest and motivation could be gauged. The results showed that they already had an appreciation of the benefits of video conferencing, especially those related to cost effectiveness. Regarding what they perceived as being the difficulties with video conferencing, most were concerned with how it may restrict their teaching. (Please see Appendix 2 for more detailed responses.)
The results of the questionnaire were very useful in informing the design of the course and helping assess the level of C&IT experience. Although the participants occasionally or never use CMC technologies, they all had experience of using a computer and 61% indicated that they already used a computer to prepare their teaching. More particularly, it highlighted the methods and media that they currently use and would like to use in video conferencing. It is interesting to note, that although a significant number used practicals, case studies and tutorial or group discussions, the most common approach, which they wanted to use with video conferencing, was lecturing.
A number of different types of evaluation were carried out during the development and delivery of the course, including:
Expert review of the course outline
A range of both internal and external experts were consulted to provide comments and feedback on three main issues:
It is preferable to have more than one expert evaluating the content to ensure these aspects are given a balanced evaluation. A further advantage of using expert reviewers, is that not only can they identify problems but they can also offer advice on how to resolve them (Phillips, 1997). Table 7 shows the review panel that was used.
Debriefing with all trainers following each session
After each session, the trainers met to discuss and evaluate the session. Each session was discussed in terms of:
Two types of evaluation were carried out:
This type of evaluation provides an analysis of the participant's attitudes towards the course. When the participants completed the course, they were asked to evaluate, via a questionnaire:
Results and Discussion
In general, the results of the summative evaluation to the course were very positive. The following highlights some of the findings, but please see Appendix 4 for detailed results of the evaluation. The evaluation questionnaire used is in Appendix 3.
Regarding the content of the course, the participants agreed or strongly agreed with the comments relating to content of the course, see Table 8.
Statement Strongly Agree The content was relevant to my training needs 100% The purpose and objectives of each session were clear 83% I now have a better understanding of VC terminology 83%Table 8 Assessment of the Content of the Course
In evaluating the course, it was important to find out what the participants felt about the methods that were used throughout the four sessions. The results in Table 9 show that there was a favourable response to the materials. When asked about the balance of teaching methods, only two thirds felt that the balance was good (Table 9).Methods Used Agree Strongly Agree
There was a good balance between types of teaching (presentation, video, group discussion, hands on) 33% 67% The material was presented clearly and in a logical sequence 17% 83% Appropriate use was made of visual aids, materials and handouts 17% 83%Table 9 Assessment of the Teaching Methods on the Course
This was again reflected in the next section of the questionnaire as the participants indicated that the practice sessions and the last session (opportunity to present using video conferencing) were the most useful. In addition, to providing them with the opportunity to practice what they had learnt, they also felt that it gave them an indication of their own ability and confidence in using the facilities.
None identified any of the sessions as being the `least beneficial'. However, two of the participants felt that the first few sessions could have been used for practicing with the equipment. A suggestion was made to integrate some of these sessions to leave more practice time with equipment in later sessions.
Lastly, the respondents identified a need for further training in a) use of Microsoft PowerPoint and b) video conferencing - to try out more imaginative approaches to using the facilities.An Assessment of the Learners' Performance
Session 4 provided an opportunity to assess the performance of the participants in making a presentation using video conferencing. This did not address all the issues covered on the course but provided an opportunity to assess the participants:
The participants were asked to plan and prepare a five minute presentation based on material they currently use in their teaching and to include any media, objects or teaching aids that they would typically use. The aim of this exercise was to provide the participants with an opportunity to apply what they had learnt from the previous sessions and give them the opportunity to deliver a session in a supportive environment.
The participants delivered their presentation to a local and remote audience and trainers assessed their performance (both locally and remotely). An observer from the Enterprise Unit in QUB was also present to comment on the presentation and delivery of the teaching. Performance was assessed via a checklist (please see Appendix 5). At the end of the presentations, general feedback was provided to all the presenters and the experiences of the participants were discussed.
This exercise did not enable us to assume that a successful performance was due only to the previous training sessions but it did however provide an opportunity to:
Note: at time of writing this report, the summative evaluation is not complete as only half of the participants have completed session 4.
Graphics Multimedia Virtual Environments Visualisation Contents