Project Selection Process S230

S230 - Agree Final Marks and Issues

DEFINITION

All information, analysis and opinions collected during the evaluation processes are reviewed and an opinion is agreed concerning the relative merits of the proposals and the preferred solution.

SUMMARY

The project team, the project sponsor and other key decision makers meet to consider the evidence collected throughout the evaluation.  This may include:

• “Zero Scores List” showing “business critical” requirements that have not been met,
• outstanding selection issues,
• comparative data from the proposals, in particular the comparative costs and benefits,
• the scored Requirements Matrices
• the results from the vendors’ demonstrations and other contacts
• the results from consulting reference users of the vendor's’ software,
• results from the Package Fit tests (if appropriate).

The detailed findings may be revised if appropriate (although this should not be treated as a complete reworking of the analysis).  The selection team will reach a consensus on the overall issues and findings.  They will agree the key comparisons with the project sponsor and key decision makers to establish which options are preferred, not preferred and rejected.

PATH PLANNING GUIDANCE

This process is normal practice.

DEPENDENCIES

This process normally runs during the evaluation of tenders received in response to the Invitation to Tender (ITT).

Prerequisites (Finish-Finish):

• Mark responses(S190)
• Identify Issues (S200)
• Confer with Vendors (S210)
• Consult with Reference Users (S220)
• Package Fit Tests (if appropriate) (S180)

Dependent procedures (Finish-Finish):

• Benchmark (if required) (S240)
• Prepare Selection Report (S250)
• Negotiate terms with preferred vendor(s) (S300)

RECEIVABLES

• vendors’ proposals
• comparative data based on vendor's’ proposals, including costs and benefits
• Zero Scores List
• selection issues list / log
• reports / questionnaires from vendor demonstrations
• reports / questionnaires from user reference site visits
• report from Package Fit tests (if any)

DELIVERABLES

• finalised comparative information
• finalised issues list
• provisional agreement on issues and preferred solution

TOOLS

• Examples: Zero Scores List
• Examples: Selection Issues List
• Examples: System Demonstration Questions
• Examples: Site Visit Questions,
• Examples: scoring spreadsheets,
• Examples: Zero Scores List,
• Examples: Selection Issues List,
• Examples: Application Comparison Worksheet,
• Examples: Hardware/Software Comparison Worksheet,
• Examples: Vendor Comparison Worksheet,
• Examples: Cost Comparison Worksheet,
• Examples: System Quality/Vendor Quality (SQ/VQ) Charts,
• Examples: Cost Summary Chart.

DETAILED DESCRIPTION OF TASKS

Collate and review evaluation materials

The project team should gather together all materials collected during the evaluation.  These may include:

• Original proposals and further correspondence from the vendors,
• System Demonstration Questionnaires or visit reports,
• Site Visit Questionnaires or visit reports,
• scoring spreadsheets,
• Zero Scores List
• Selection Issues List
• Package Fit Test reports,
• Application Comparison Worksheet,
• Hardware/Software Comparison Worksheet,
• Vendor Comparison Worksheet,
• Cost Comparison Worksheet,
• System Quality Chart,
• Vendor Quality Chart,
• System Quality/Vendor Quality Chart,
• Cost Summary Chart.

Key findings may be reviewed.  During this review it should be checked that the findings have been fair, concise, comprehensive and complete.  It is not normal practice to recalculate the weights used or scores awarded except where it is clear that they misrepresent the actual position.

It is normally appropriate to prepare finalised cost/benefit projections for the competing solutions based on all the knowledge now available (see below).  These will be used to balance the financial case against the perceived merits of the competing proposals.

The significant findings should be highlighted for consideration by the project sponsor and key decision makers.  This may conveniently be done in the form of presentation materials as prepared in draft form for the final Selection Report (see Process S250).

Modifications and customisations

Attention should be paid to any part of a solution requiring programmed customisation or modifications to the package.  System enhancements by custom programming may be necessary within a given solution or may seem desirable to meet a shortfall in the proposed functionality.  Custom programming issues may not be defined in detail, team members should refer to a more specific methodology for such work.  In particular, appropriate methods should be used to estimate the workload and timescales involved.

Careful consideration is required as modifications to the package may render it unsupported by the vendor, and could become worthless when a new version of the basic package is released.  Customisation is best performed by building external functionality attached to the package using vendor-defined and supported hooks, rather than by amending the basic source code.  Always discuss and agree the position with the vendor before relying on an approach which includes customisation of the package.  If system enhancements are identified, alternative development approaches include:

• Vendor – Contracting with the vendor to integrate the enhancements into the base system.  This may have significant appeal if the functions will be “vendor supported” in all future releases of the software and it frees in-house staff to focus on implementation and maintenance tasks.  In many cases, the vendor will not allow any other body to amend the source code (or even to see the source code) of the package.
• Contractor/Consultant – Contracting with a Consultant or an outside contractor to develop the enhancements.  Vendors usually will not support system changes made outside their organisation, unless the enhancements will provide significant advantages to other customers.  In such cases, vendors often insist that they are either contracted to perform or manage the design phase.
• In-house – Developing the system enhancements using internal resources.  If the client organisation’s internal MIS department has sufficient resources and skill, there can be some potential cost savings.  However, undertaking significant system enhancements would tie up resources that may be better used for other MIS-related responsibilities.

Evaluating hardware options

A detailed analysis of the proposed hardware for each alternative generally requires specialised expertise.  If the hardware is critical to the selection, then a specialist should assist with the evaluation.  At a minimum, it is recommended that the hardware analysis should cover the following issues:

• Compatibility with existing systems
• Sufficiency to support expected load
• Expandability and upgrade path
• MIS support needed.

The financial case

The cost evaluation should include all one-time costs (including implementation support), and on-going maintenance costs.  An expected “system life” of five to seven years is typically used.  Because of changes identified during the evaluation, vendors are frequently required to recompute cost estimates before the cost evaluation can be completed.  To establish a common baseline, calculate the present value of the expenditure stream less the expected savings from scrapping and not supporting the existing system.

Comparing only the maintenance costs to continue operation of the existing systems versus the purchase costs to implement the new technology strategy assesses only the “capital outlay” and may not provide an accurate assessment of the alternatives.  Adding the operating costs to the direct costs often reveals that while the new system may be more expensive to purchase, it saves significant user and operating costs.  Operating costs can be quantified by adding in the estimated time for users to perform the related manual functions and the MIS time to operate and maintain the system factored by a reasonable rate per hour representing cost of service.

Consultant personnel should avoid making claims as to any efficiencies, reductions or savings as a result of implementing a particular solution, although they may assist in the preparation of a Cost/Benefit Analysis, and the client will need to be sure that the preferred specific solution will support the Cost/Benefit Analysis.  Consultant personnel should also avoid making any claims as to the performance of any specific solution or to its freedom from system bugs or programming errors.

How is the decision made?

There is no mathematical formula or algorithm which can generate a decision based on the information gathered.  The team must present the key differentiating issues to the project sponsor and key decision makers, and discuss the relative merits of the different proposals.

The information will help support the process.  In particular it will be used to highlight the key differences between the solutions and the key issues which affect the decision:

• Discuss the “business critical” requirements which cannot be met based upon the Zero Score List.
• Discuss the weighted scoring of the proposals paying particular attention to parts of the comparison where there is a significant difference in the scores.
• Discuss the issues identified and their implications
• Discuss the views expressed by reference users concerning the quality of the vendors and of the vendors’ proposals
• Discuss the findings of the team resulting from the vendors’ demonstrations and other vendor contacts.
• Discuss the comparative costs and benefits of the solutions
• Discuss the overall benefit to the organisation of proceeding with each option.

The facts must be balanced to identify the best solution, other acceptable solutions and unacceptable solutions.  Note the importance of this classification:

• The preferred solution is the one that the team believes is best suited to meet the organisation’s needs.  It may be assumed that this solution will be followed and following actions can be based on that assumption.
• Other acceptable solutions are identified to provide alternative approaches in case a problem is identified with the preferred solution.  It is also useful to continue to negotiate with all the acceptable vendors as frequently the vendors will improve their financial offers in the face of the competition.
• Unacceptable solutions should be identified where there is no point pursuing a particular solution any further as it would not be acceptable to the client organisation.  Letters should be sent to confirm to the relevant vendors that their proposals have been rejected.

The choice of a preferred solution is rarely clear cut.  In most cases there will be competing merits in the competing solutions.

It is often found that the best solution is not the cheapest and the cheapest compliant solution is not the best.  The cost benefit analysis may cast some light on this but, in practice, it is not normally possible to match the perceived greater value of a given solution to a calculated improved financial benefit.

The final balance of these issues can only be judged definitively by the project sponsor and key decision makers within the client organisation.  Where assumptions are made or where particular conclusions have been formed to support the preference, these should be noted.  The fact that the ultimate decision rests with the project sponsor and key decision makers within the organisation should be made clear to all participants and should be clearly stated in any resulting documentation.

The considered comparison and preferences will be documented formally in the Selection Report - see Process S250.