AGOCG logo
Graphics Multimedia VR Visualization Contents
Training Reports Workshops Briefings Index
This report is also available as an Acrobat file.
Back Next Contents

Chapter 4

Evaluation Tables for Videoconferencing Software

Introduction

In this section, we convey our impressions of various video conferencing products. Evaluation tables are provided so that readers can get a feel for how typical products compare. We have, on the whole, selected features which we thought were relevant to 'desktop' video conferencing, This section will be useful to anyone carrying out a future evaluation of video conferencing products.

The feature tables were classified using the following headers.


Table headers       Summary of features

General             What top level facilities are available, on which 
                    machines ....?

Conference Manager  Invitations, selection of tools, nicknames for participants
                    ...

Video               Adjust speed, cope with flash, zoom, snapshot ....

Audio               Adjust volume, duplex switch ....

Shared Whiteboard   Drawing tools, text, stamps ....

Shared Applications Take control, keep control ....

Address Book        Add cards, sort, navigate ....

In the tables, the first column contains a feature and the second column contains an indication to show how relevant we feel the feature is to video-conferencing in a helpdesk environment.

** means   Found to be very useful
*  means   Found to be desirable
?* means   We have not been able to try this but we 
           think it would be very useful
?  means   We have not been able to try this and we 
           cannot assess its importance to us.

The next 5 columns show how well the features are implemented in the different items of software. (SHOWME, IVS, LBL, INPERSON and PRO-SHARE). The grades assigned are

** means   well implemented

*  means   partially implemented

x  means   not implemented.

NA means   Not Applicable i.e. the product is not designed to support the feature.

We have checked each item from the manuals, from the help pages and from experimenting with the package but, of course, we cannot guarantee that a feature marked as not implemented is really missing!

General

General Table          Imp    SMe    IVS    LBL    INP    PRO

SUN SUNOS * * ** ** NA NA SUNOS Solaris ** ** ** ** NA NA SGI IRIX * NA ** ** ** NA DEC OSF ? NA ** ** NA NA DEC ULTRIX ? NA ** ** NA NA HP 9000/700 UIX ? NA ** * NA NA IBM RS6000 AIX ? NA X * NA NA IBM PC or compatible ** NA NA NA NA ** Technical Support ** ** * * ** ** Online Help ** ** * * ** ** Tutorial ? x x x x ** Documentation ** ** * * * ** Video boards supported ** 2 for Many Many SGI PC Sun Standards (video) ** CellB H261 CellB HDC Indeo Standards (audio) ** G.711 PCM LPC G.711 Indeo Conference Manager ** ** * ** ** * Video ** ** ** ** ** ** Audio ** ** ** ** ** ** Whiteboard ** ** ** ** ** * Shared Application ** ** x x * ** Address Book ** ** x x ** ** Multi-casting ? * ** ** ** x

Comments: LBL also uses PCM and GSM to transmit audio and supports variety of video compression algorithms

Conference Manager


Conference Manager           Imp    SMe    IVS    LBL    INP    PRO

Call using nicknames ** ** x x * * Call full name + address * ** ** ** ** x Call by selecting a group * ** x x ** x Call by selecting user icons ?* x x x ** x Call last number ?* x x x x ** Call using a public list ** ** x x ** ** Give a subject * x * ** ** x Call a 1-way conference * ** x x x * Join conference in progress ** * ** ** ** x Add selected tools to conf ** ** * ** * * Reject tools selectively * ** x x * x Leave tools selectively * ** x x x * Quit conference unilaterally * ** ** ** ** x Flashes at invitation ?* x x x ** x Audible rings at invitation ** ** x x ** ** Invitation panel ** ** ** ** x ** Change rings ?* * x x ** ** Hangup if no reply ** ** ** * ** ** Set/change timeout ? x * ** ** x Own status in conf ** ** ** ** * x Status of others ** ** ** ** * x

Video

Video Table                  Imp    SMe    IVS    LBL    INP    PRO

Adjust speed manually ** ** ** ** ** * Adjust speed automatically ** ** ** x ** ** Adjust picture resolution ** ** ** * * * Use grayscale ** ** ** ** x x Adjust number of colours ** ** x ** x x Adjust colour balance ? x x ** ** x Brightness control ** ** ** ** * ** Picture in picture * ** x x x x Use static image instead * * x x ** x Hide one's own image ** ** ** ** ** ** Hide image of others * ** ** ** * ** Zoom image sizes ** ** ** ** ** ** Allow one-way video only * ** x x x x Videotape * ** ** ** ** x Switch between video ports * ** x ** ** x Mirror image * ** x x ** ** Snapshot of oneself/others * ** x x * ** Send snapshot to others * ** x x * * Save/print snapshot * ** x x * ** Actual frame rate info ** ** x x ** x Bandwidth information ** ** ** ** ** x Status area for participants ** ** ** ** x x

Audio

Audio Table                  Imp    SMe    IVS    LBL    INP    PRO

Adjust outgoing volume ** ** ** ** ** x Automatic volume control ** ** ** ** x x Adjust incoming volume ** ** ** ** ** ** Audio test facilities ? x x ** x ** Suppress background noise ?* x ** ** ** x Choose audio encoding ?* x ** ** ** x Playout delays ? x x ** x x Headphones ** ** ** ** ** ** Speakers ** ** ** ** ** ** Full/half duplex switch * ** * ** x x Mute oneself ** ** ** ** * * Mute others ** * ** ** ** ** One way audio * ** x ** x x Side conversations ?* x x ** x x Record voice ? ** ** x x * Status message (comms) ** ** ** ** x x Status area for particpants ** ** ** ** x x Play audio CD/tape ? ** ** x ** * Audio acquisition ? x x ** x x

Comments : Audio acquisition is to indicate what takes control of the audio hardware when there is a competition for the audio facilities

Whiteboard

Whiteboard Table            Imp    SMe    IVS    LBL    INP    PRO

One way * ** NA x x x Status area for participants** ** NA ** ** * Import images * ** NA x ** ** Import PostScript file ?* x NA ** x x Save window contents * ** NA x ** ** Print window contents * ** NA ** ** ** Clear window ** ** NA * ** ** Layers/Pages * * NA * ** * Clear one layer/page * ** NA * * x Sort layers/pages ?* x NA x x ** Move items to another page * ** NA x * * Snap window/screen/region * ** NA x ** * Erase drawing item ** ** NA ** ** ** Undo last operation ? x NA * x ** Move objects ? x NA ** ** * Select group of objects ? x NA x ** * Cutting/Pasting ? x NA * ** * Distinctive cursors ** ** NA x ** x Ask who drew this object ? x NA * ** x Ability to point * * NA x * ** Make private notes ? x NA ** x ** Freehand/line ** ** NA ** * * Ellipse/Rectangle/Circle ** ** NA ** * ** Line thickness * ** NA ** ** ** Line colour * ** NA ** ** ** Arrow/arrow colour ? x NA ** ** x Filled shpaes * ** NA x x ** Patterns * ** NA x x ** Text ** ** NA ** ** ** Text size/colour * ** NA * ** ** Text fonts ? x NA ** ** ** Edit text ? x NA x ** ** Import text ? x NA ** ** ** Highlighter colour * ** NA x x ** Variety of stamps ** ** ** x x x

Comments : The makers of wb (part of LBL) claim that distinctive cursors do not work well, according to some human factors research. A deliberate decision was made to omit them.

Shared Application

Shared App Table            Imp    SMe    IVS    LBL    INP    PRO

Status area for participants** ** NA NA NA * Legend ** ** NA NA NA * Take control ** ** NA NA NA * Keep control ** ** NA NA NA * Cursor colours * ** NA NA NA x Hide other colours * ** NA NA NA x Select application from list** ** NA NA NA ** Exchange files ?* * NA NA ** ** View-only files ? * NA NA x ** Public edit of files ** ** NA NA x **

Address Book

Address Book Table          Imp    SMe    IVS    LBL    INP    PRO
Create address books        **     **     NA     NA     **     **
Open old book               **     **     NA     NA     **     **
Import address book         *      **     NA     NA     **     **
Create a group of addresses *      **     NA     NA     **     x
Save address book           **     **     NA     NA     **     **
Print address book          *      **     NA     NA     x      **
Add a card                  **     **     NA     NA     **     **
Copy a card                 *      **     NA     NA     *      *
Delete a card               **     **     NA     NA     **     **
Sort by firstname           *      **     NA     NA     x      **
Sort by lastname            **     **     NA     NA     x      **
Define structure of cards   *      **     NA     NA     x      **
Add fields to card          *      **     NA     NA     x      **
Navigate cards              *      **     NA     NA     **     **

Conclusions

This section is to give a summary of the evaluation tables and will help to give an indication of how many features each package supports.

We start by assigning marks and simply finding the totals for all but one of the previous tables (the General table is not included)

Features Table      Maximum    SMe    IVS    LBL    INP    PRO

Conference Manager 42 30 17 19 29 16 Video 44 41 22 25 28 19 Audio 38 25 27 34 17 13 Whiteboard 72 46 NA 35 47 50 Shared Application 20 18 NA NA NA 12 Address book 28 28 NA NA 17
We then construct another table where the features are weighted according to how important we feel they are in supporting desktop video conferencing. The marks assigned are again but this time have weighting factors: This is the resultant table
Weighted Table       Maximum   SMe   IVS   LBL   INP   PRO

Conference Manager 68 48 28 30 47 30 Video 66 63 40 42 42 29 Audio 62 42 49 58 32 22 Whiteboard 92 62 NA 48 59 62 Shared Application 34 31 NA NA NA 22 Address Book 40 40 NA NA 27 35

These figures should clearly be used with caution. There are various reasons for recommending that the figures are treated as guidelines only. One is that neither of these final tables (with the emphasis on the number of features supported) has any indication about the actual quality of the video or audio received by a conference participant. However, it is reasonable to assume that an increase in the number of performance options, colour controls, or features to ascertain the performance for both video and audio, leads to more flexibility in obtaining higher quality video and audio. So the figures should not be dismissed out of hand!

One could conclude from the tables that ShowMe is the best package but it has to be pointed out that our experiences of desktop videoconferencing were mostly in using ShowMe. This meant that the choice of features for the tables was made with ShowMe in mind.

If the two tables are examined side by side, we can deduce that in video and audio, both IVS and LBL have a high proportion of 'weighted' features. So although they may not have as many features, they do score well on the important ones.

Therefore, the public domain products, since they are free, must be taken seriously. IVS is a good audio/video tool. It has the advantage over LBL that it supports the H.261 standard. Its main disadvantages, relative to proprietary products, are that it is comparatively poorly documented, it has a relatively technical user interface (compared, for example, with ShowMe) and is not integrated with the whiteboard tool, wb. The LBL set of tools, nv, vat, wb and sd, are better integrated but otherwise suffer from the same disadvantages as IVS.

We feel that the public domain products are more suited to IT specialists at present and would not be suitable for a general service environment. Also, we are not aware of any public domain offerings that support shared applications.

In a more extensive evaluation than the one presented here, one would expect to see a classification of features according to their importance to various types of videoconferencing. Also, there would be performance and quality tests, particularly for the video and audio components.


Back Next Contents

Graphics     Multimedia      Virtual Environments      Visualisation      Contents