Project Selection Process S220

S220 - Confer with Reference Users

DEFINITION

Various contacts with existing users of the vendors’ systems aimed at obtaining independent opinions regarding their proposed solutions.

SUMMARY

Contacts with existing users of a vendor’s system provide independent views of the vendor and of the vendor’s proposed solution.  This can be a valuable component of the selection process.  The views collected will be used to corroborate or challenge the claims of the vendors.  The opinions expressed will be of great value during the final evaluation of the competing options.

The forms of contact typically include:

• telephone conversations,
• attendance at user groups etc,
• visits to the users for meetings, discussions and demonstrations of the proposed system.

For each contact, whether formal or informal, there should be:

• an understanding of the objectives of the contact,
• a prepared approach for how this should be achieved, and
• some form of notes to show the outcome.

PATH PLANNING GUIDANCE

This process is optional.  It is normal practice where appropriate users exist and are contactable.

DEPENDENCIES

This process normally runs during the evaluation of tenders received in response to the Invitation to Tender (ITT).

Prerequisites (Finish-Start):

• Receipt of tenders (S190)

Dependent procedures (Finish-Finish):

• Agree final marks and issues (S230)

RECEIVABLES

• vendors’ proposals
• selection issues and zero scores etc

DELIVERABLES

• Site Visit Agenda
• Site Visit Questionnaire
• Updated Issues lists / logs
• Visit reports
• Thank you letters

TOOLS

• Examples: Site Visit Agenda
• Examples: Site Visit Questions
• Examples: Selection Issues List

DETAILED DESCRIPTION OF TASKS

Objectives and General Approach

Contacts with existing users of the vendor’s system are intended to improve the quality of the evaluation by gaining independent views of:

• the functionality, reliability and usability of the software,
• corroboration of specific queries or issues concerning the vendor’s proposal,
• the reliability of the vendor organisation and their suitability as a potential “business partner”,
• the real-life experience of implementing the software in terms of effort, timescales, problems, support from the vendor etc.

Existing users are often very keen to co-operate.  They will normally be proud of their achievements and be convinced that their decision was right.  They will often be keen to encourage more users of the same systems since the size of the user base endorses their personal decisions, leads to greater investment in the products and adds to the strength of the vendor upon which they rely.  The reference users will, therefore, tend to present a good picture of their systems and their organisation, but may be defensive if any suggestion is made concerning defects, problems or inefficiencies.

The selection team members must, therefore, be proactive but tactful in their approach.  They should try to set the agenda and steer the discussions to meet the team’s objectives.  They should prepare in detail the specific points they wish to have answered and take steps to see that the points are covered.  They should make sure that their findings are documented and reported back to other members of the team where appropriate.

Remember, however, that the team are guests and that the reference users owe no direct allegiance either to the project team or to the vendor so their participation will be entirely voluntary.  They should always be treated with respect and thanked for their assistance.  No attempt should be made to force the reference users to do things or say things against their will.

Forms of contact

Contacts with reference users should efficiently provide exchanges of information, ideas and views.  They should take the most effective format to meet the objective of that particular required contact.

Form of contact	Comments
Telephone calls	These are ideal for collecting summary level views from a large number of user organisations.  The team should try to make a wide range of contacts in this manner to make sure that they have a complete view of the opinions of existing users.  It is important that a file note is written immediately to record any significant points.
Conference calls / teleconferencing / video conferencing	Similar to telephone calls (see above) but allows a number of people to participate in the discussion.
Site visits / meetings / demonstrations	This is usually an important factor in the evaluation process.  Direct contacts with the reference users allow the free exchange of a large amount of information and opinions.  They also allow the evaluation team to get a feel for the real-life success (or otherwise) of the system in the context of a working operation.  It is important that the meetings address the selection team’s needs for information.  The team should seek to set the agenda and have prepared questions and topics that they feel should be explored based on the issues identified during the evaluation.  Make sure, however, that the hosts are expecting this and have advance notice otherwise they might feel their hospitality is being abused.
Building relationships	Other users of a packaged system can be of great help.  Their genuine advice during the evaluation processes can be a valuable contribution.  Their help may also be of benefit during the subsequent development and live running of a system.  For example, they may often lend independent practical advice, contribute their own add-on customisations free of charge, or work together to pressurise the vendor.  It is, therefore, of value to continue to develop those relationships after the initial fact finding contacts.
User groups	Many packages have independent user groups.  (Some have non-independent user groups.)  User groups are a good way of locating a large number of user organisations and experienced designers, implementors and users of the systems.  At user group meetings there will usually be a series of informative sessions provided free of charge.  Very often there will be key talks given by experts.  Altogether these meetings and the user groups themselves can provide valuable sources of support and information.
Written communications	During the evaluation process, written communications may be appropriate for: administrative matters, eg confirmation of date for meetings etc questionnaire approaches to fact finding.

Choosing the reference users

Where telephone calls are to be used, it is possible to approach a wide range of users. Where visits are to be made it may be impractical to make more than one visit per vendor.  In either case, it is important to seek appropriate contacts which are most likely to give a relevant complete view of the vendor’s proposals.

Some considerations might include:

• Reference users should normally be using the same versions of the same applications on the same equipment and operating systems, although with specialist or new applications this might not be fully achievable.
• Reference users should ideally be easy to contact and discuss matters with.  Where visits are to be made, they should be conveniently located.  It is usually best if they speak the same language and are subject to the same legal and cultural environment.
• Reference calls should be made to relevant key users, department managers and the MIS manager of the reference company.  Requests should be made for the names of other system users that may lead to someone who is less satisfied with the system.
• Good reference sites will be in a similar line of business so that their way of using the system is entirely relevant.  They should where possible have similar significant business issues to address, for example, group structures and relationships, multi-national issues,  multi-location issues.
• Good reference sites will be of a similar size and be handling similar volumes of data.
• Depending on culture and the industry concerned, it may not be appropriate to seek reference visits to rival organisations.  In practice, however, rivals are often happy to be consulted and will usually give honest opinions.  Make sure no confidential information is disclosed to them concerning the client organisation.
• It should be understood that vendors are likely to propose sites of only their best customers.  It is often useful to obtain a complete client list from the vendor (or from an independent source such as a user group or directory) and select suitable reference sites without the advice of the vendor.
• Sometimes vendors feel it is their right and duty to select the reference user and to act as guide and chaperone throughout the visit.  It is usually best to avoid this (politely) and to make all the arrangements through direct contacts once the contact details have been established.
• Sometimes user organisations find themselves inundated with requests.  This can often happen with the first live users of a new technology or system.  In such cases there may be long waiting lists or the organisation may refuse to accept any further visits.  It may be possible to restrict the reference call to a telephone call.
• When there is no direct comparison available, it may still be worth taking up “persuasive” references, where the views of the users are indirectly relevant.  For example, users of old versions of the software may still hold valid opinions about the reliability of the vendor and the quality of the vendor’s other services.
• There will always be cases where the proposed solution is so new that there is no reference site available.  These situations should clearly be judged on their merits.  Sometimes the newest solution is the best solution, but it is often the highest risk option.  The lack of a reference site should not be considered as a disqualification for the proposal.

Meetings, demonstrations and presentations

Reference site visits should be based upon the issues and queries generated during the evaluation process.  Generally they should be called after the system demonstrations so there is a basic understanding of the system’s operations and the issues that have been identified.

Site visits usually consist of observing the proposed system in an actual operating environment at one of the vendor’s customers’ place of business.  The purpose of these visits is to validate the vendor’s claims of customer support, determine the difficulty in implementing and operating the system, and evaluate the systems performance and ease of use in a real life situation.  Much can also be indirectly learned about the vendor, operating techniques, and in-house expertise needed.  Site visits can be especially useful in cases where new technologies or methods are being applied.

From the perspective of the organisation being visited, site visits are disruptive to normal business operations and provided only as a courtesy to other potential customers.  Visits should be scheduled well in advance and the agenda of discussion issues and attendees confirmed with all participating parties.  After the visit, a letter of appreciation should be sent to the participants from the host companies.

The selection team should normally suggest an agenda for the meetings based on the specific issues relating to the relevant vendor’s proposal.  An example can be found in Examples: Site Visit Agenda.  Try to give reasonable advance notice of the team’s requests.

In addition to the agenda, it may be useful to prepare a detailed questionnaire.  This is of particular value where not all the selection team will be attending and where the attendees are not fully conversant with all the issues. In addition to asking factual questions, it is always worth seeking opinions and guidance, for example “if you were to do this again, would you do it the same way?”

Where a detailed questionnaire is not prepared and completed, the attendees should prepare a visit report noting the significant findings and their general feelings about the vendor and the proposed solution based on the visit.

In addition to the functional and support issues, it may be useful to explore the education and training needs.  Establish how usable the system is, where the major needs for training and education were found, where problems were encountered, what training programmes and materials were developed.  (It may be worthwhile asking if you might be allowed to borrow or copy any non-copyright materials they have produced, both for the evaluation process and possibly for actual use during the Delivery Segment.)

The results from these sessions should be circulated throughout the team and the issues list should be updated to reflect any new information or changes.  The results will be of particular importance during the final assessment of the competing solutions.

Some considerations concerning the logistics of the sessions might include:

• The reference user organisation is unlikely to welcome a major invasion - discuss with them what number of people and times would be convenient.  Make sure, however, that they understand the range of people and information that you are interested in.
• It may be possible to condense the visits by having some general sessions applicable to the full team, and then some specialist sessions held in parallel, where team members talk to the relevant personnel at the reference site individually .
• In a small team it may be practical for all members to attend all the meetings.  More often, it will be necessary to delegate.
• Try to cover the range of skills required for the session, eg both functional and technical specialists may be required, and it may be necessary to cover several separate functional areas.  The best results are often achieved where members of the host organisation are interviewed by their direct equivalents in the client organisation.
• The project’s sponsor or key decision makers may wish to attend, but this may not always be appropriate.  If senior staff who are not part of the selection team do attend it is important that there is at least one regular team member to guide them and report back,.  They should be briefed and debriefed accordingly.  The use of questionnaires can be of particular value in such cases.
• Do not let the vendor control the agenda or interfere with the gathering of independent views.  Often the mere presence of the vendor’s representative can block free communication.
• So far as is possible, similar investigations and contacts should be made for each vendor’s proposals to provide a balanced view.