Project Selection Process S610

S610 - Prepare, issue and agree Selection Validation Report

DEFINITION

The findings from the selection confirmation process are discussed, agreed and published in the Selection Validation Report.

SUMMARY

Discuss and agree the findings with the project sponsor and any other key decision makers within the client organisation.  Prepare and agree the Selection Validation Report containing the approach, key requirements, issues, options and recommendations.  If the chosen solution appears unable to meet the client's knockout mandatory requirements, implementation should not commence and there should be a review of the project.

PATH PLANNING GUIDANCE

This process is normal practice in “Selection - Confirm”.

DEPENDENCIES

Prerequisites (Finish-Finish):

• Check business critical requirements against package (S600)

Dependent procedures (Finish-Finish):

• Optionally - Prepare contracts  (S310)
• Optionally - Prepare Implementation Strategy (S710)
• Optionally - Feasibility Prototype (S400)

Dependent procedures (Finish-Start):

• Delivery Segment processes

RECEIVABLES

• Knockout mandatory and business critical requirements
• Evaluation of package’s capabilities against requirements (see Process S600)

DELIVERABLES

• Selection Validation Report

TOOLS

• Skeleton Deliverables: Selection Validation Report

DETAILED DESCRIPTION OF TASKS

Fact finding and evaluation

The knockout mandatory and business critical requirements will have been identified (see Processes S040 and S070).  In Process S600, the package’s capabilities were investigated and analysed to establish how well they could meet the client organisation’s key requirements.

Reviewing the findings

The project team should now consider these findings and present them to the project’s sponsor and any other key decision makers within the client organisation.  This may be done in the form of a meeting or workshop depending on the numbers involved.

The facts and findings should be presented paying particular attention to:

• what were the key requirements - ie knockout mandatory requirements and business critical requirements,
• in which areas (if any) the package failed to meet these requirements fully,
• results from any other investigative processes if conducted, eg benchmarking or package fit study,
• the potential impact of any shortfalls identified,
• options to circumvent the problem, eg
• changes to procedures or business processes,
• custom add-on developments,
• vendor modifications to the package.

To facilitate this process, background information and broad estimates concerning the options should be prepared in advance.

The issues and options should be debated with the representatives from the client organisation.  The resulting recommendations should reflect their wishes and decisions.

Recommendations may include the need to perform more detailed evaluations prior to accepting the pre-selected solution.  For example, it may be appropriate to set up a feasibility prototype before commencing the Delivery Segment work (see Process S400).

Any effect on the proposed Path Plan or approach to the Delivery Segment should be considered and the implications reviewed.  Changes may be made to the planned approach as appropriate.

It is normally appropriate to prepare an Implementation Strategy before proceeding with the Delivery Segment - see Process S710.  If the project is a major undertaking, it might, alternatively, be appropriate to develop a full conceptual design and implementation strategy in the form of a Delivery Approach Definition - see Process S700.

Selection Validation Report

The results from the fact finding, analysis, review and recommendation work will normally be reported in the Selection Validation Report (although note that the client organisation might be willing to accept the results from the workshop without the production of a formal report).

This report might typically contain:

• Introduction
• Background
• Scope, objectives and terms of reference of the project
• Approach to the Selection Confirmation
• Key requirements
• Knockout Mandatory Requirements
• Business Critical Requirements
• Evaluation
• Shortfalls against requirements
• Benchmark results (if appropriate)
• Other issues identified
• Options
• Recommendations

If the requirements identified are extensive, it may be appropriate to summarise them in the report and to attach the full matrix as an appendix to the report.

The report should be formally agreed and signed off by the project sponsor and any other relevant key decision makers within the client organisation.

Project Selection Process S600

S600 - Check Business Critical Requirements against Package

DEFINITION

The capabilities of the pre-selected package are evaluated against the requirements identified as knockout mandatory or business critical.

SUMMARY

Desk check the “knockout” mandatory and business critical requirements against the specified technical solution.  This may be done by a combination of:

• putting specific questions to the package’s vendor,
• evaluating the package’s documentation,
• using a demonstration version of the system,
• using a system that is already installed,
• consulting other internal users, for example, existing users within related companies,
• consulting external existing users and visiting reference sites,
• referring to specialist consultants within Consultant Company,
• consulting and attending user groups.  

If appropriate, the full requirements matrix may be issued to the vendor to request a formal response from the vendor.

PATH PLANNING GUIDANCE

This process is normal practice in “Selection - Confirm”.

DEPENDENCIES

Prerequisites (Finish-Finish):

• Format Requirements Matrix (S040)
• Prioritise Requirements Matrix by criticality (S070)

Dependent procedures (Finish-Finish):

• Prepare, issue and agree Selection Validation Report (S610)

RECEIVABLES

• Knockout mandatory and business critical requirements
• Any information available concerning the capabilities of the pre-selected solution

DELIVERABLES

• Findings reported in Selection Validation Report - see Process S610.

TOOLS

• Library of example requirements matrices

DETAILED DESCRIPTION OF TASKS

The need to evaluate business critical requirements

Although a technical solution has been pre-selected, it may be wise to check that the solution will, in fact, meet the essential requirements of the client organisation.  It is also useful to identify and evaluate the impact of any areas where the pre-selected solution would fail to meet fully the organisation’s business critical needs.

The requirements will normally have been noted and listed in Process S040, and prioritised according to criticality in Process S070.  The prioritisation will have identified which requirements are:

• Knockout Mandatory - such requirements absolutely must be met by the solution or it would be completely incapable of meeting the client organisation’s needs,
• Business Critical - these requirements are critical to the effective operation of the business and any failure to meet them would cause severe difficulties which must, therefore, be considered in depth and addressed before deciding whether the solution could be accepted.

These key requirements should be compared with the capabilities of the pre-selected package to ensure that the technical solution will be capable of meeting the client organisation’s fundamental needs.  This will normally be noted and reviewed by recording the results against the prioritised requirements in the Requirements Matrix.

Where there is a shortfall in the capability of the package, this should be noted and evaluated.  Consideration should be given to any alternative way of meeting the needs, for example, by introducing additional software or changing manual processes.  Customising the basic package is not normally a cost-effective solution given the likely difficulty and cost of supporting and upgrading the system, but it should be evaluated on its merits where appropriate.  The impact of the shortfall plus the estimated costs and benefits of any options should be reviewed.  A course of action should then be discussed and agreed with the client organisation.

The findings from this evaluation will be reported in the Selection Validation Report - see Process S610.

Fact finding techniques

It is important to get sufficiently detailed information upon which to base the evaluation.  The appropriate techniques will depend upon the particular circumstances of the client organisation and the reason why the pre-selection was made.

If the pre-selection has been made because of a decision taken by a parent organisation or related company, it may be possible to use the information upon which the decision was made, for example, the existing responses to an Invitation to Tender, vendor’s proposals and  contracts.  It would also be possible to discuss the merits of the solution with the relevant personnel in the parent or related organisation.

Where the system has already been supplied to the client organisation or a related organisation, it should be possible to refer to the documentation supplied and to investigate the actual experiences of those people who implemented it.  It might also be appropriate to prototype some of the needs on that system should there be any areas of doubt.

Specialist consultants will normally be available within Consultant who have experience of the packages concerned.  In some cases Consultant may be able to prototype the client organisation’s needs on a Consultant “showcase” system.  

Other users of the system may be consulted, for example, by visiting reference sites and user groups.  This technique alone is unlikely to give a definitive answer to the issues, but does give valuable corroboration to the information and views gathered from other sources.

The package vendor is often the best source of information concerning the package’s capabilities.  However, there are some particular considerations:

• the vendor may not be interested in any substantial amount of work if the decision will not result in an additional sale of the system (eg where there is a group licence or where an existing centralised system will be used),
• as with any selection, the vendor may distort the information in their favour - accordingly, the project team should always take steps to corroborate the information and investigate further any aspects of which they are unsure.

Information may be sought from the vendor in the same way as in a normal selection exercise, for example:

• by using a formal request for information listing the critical requirements,
• by issuing a full invitation to tender including the detailed requirements matrices
• by organising demonstrations or question and answer sessions,
• by asking for documentation
• by asking for a demonstration copy of the system or a Package Fit Study (see Process S180)

Further guidance on these techniques may be found in the relevant Selection Segment processes.

It may also be appropriate to perform a formal benchmark exercise (see Process S240), or to set up a Feasibility Prototype (see Process S400) before a final decision is taken.

Project Selection Process S560

S560 - Select Preferred Vendor

DEFINITION

All information, analysis and opinions collected during the evaluation processes are reviewed and an opinion is agreed concerning the relative merits of the proposals and the preferred solution.

SUMMARY

The project team, the project sponsor and other key decision makers meet to consider the evidence collected throughout the evaluation.  This may include:

• outstanding issues from the selection processes,
• comparative data from the proposals, in particular the comparative costs and benefits,
• the results from the vendors’ demonstrations and other contacts.

The detailed findings may be revised if appropriate (although this should not be treated as a complete reworking of the analysis).  The selection team will reach a consensus on the overall issues and findings.  They will agree the key comparisons with the project sponsor and key decision makers to establish which options are preferred, not preferred and rejected.

PATH PLANNING GUIDANCE

This process is normal practice in Requirements / Selection Fast Track.

DEPENDENCIES

Prerequisites (Finish-Start):

• evaluate responses (S530),
• confer with vendors (S540).

Dependent procedures (Finish-Finish):

• prepare Selection Report (S250),
• negotiate terms with preferred vendor(s) (S300).

RECEIVABLES

• vendors’ proposals,
• comparative data based on vendor's proposals, including costs and benefits,
• selection issues,
• reports / questionnaires / work books from vendor demonstrations.

DELIVERABLES

• finalised comparative information,
• provisional agreement on issues and preferred solution.

TOOLS

• Examples: Selection Issues List,
• Examples: System Demonstration Questions,
• Examples: Site Visit Questions,
• Examples: scoring spreadsheets,
• Examples: Application Comparison Worksheet,
• Examples: Hardware/Software Comparison Worksheet,
• Examples: Vendor Comparison Worksheet,
• Examples: Cost Comparison Worksheet,
• Examples: System Quality/Vendor Quality (SQ/VQ) Charts,
• Examples: Cost Summary Chart.

DETAILED DESCRIPTION OF TASKS

Collate and review evaluation materials

The project team should gather together all materials collected during the evaluation. These may include:

• Original proposals and further correspondence from the vendors,
• System Demonstration Questionnaires or visit reports,
• List of selection issues,
• Application Comparison Worksheet,
• Hardware/Software Comparison Worksheet.

Evaluating Hardware Options

A detailed analysis of the proposed hardware for each alternative generally requires specialised expertise.  If the hardware is critical to the selection, then a specialist should assist with the evaluation.  At a minimum, it is recommended that the hardware analysis should cover the following issues:

• compatibility with existing systems,
• sufficiency to support expected load,
• expandability and upgrade path,
• MIS support needed.

The Financial Case

The cost evaluation should include all one-time costs (including implementation support), and on-going maintenance costs.  An expected “system life” of five to seven years is typically used.  Because of changes identified during the evaluation, vendors are frequently required to recompute cost estimates before the cost evaluation can be completed.  To establish a common baseline, calculate the present value of the expenditure stream less the expected savings from scrapping and not supporting the existing system.

Comparing only the maintenance costs to continue operation of the existing systems versus the purchase costs to implement the new technology strategy assesses only the “capital outlay” and may not provide an accurate assessment of the alternatives.  Adding the operating costs to the direct costs often reveals that while the new system may be more expensive to purchase, it saves significant user and operating costs.  Operating costs can be quantified by adding in the estimated time for users to perform the related manual functions and the MIS time to operate and maintain the system factored by a reasonable rate per hour representing cost of service.

Consultant personnel should avoid making claims as to any efficiencies, reductions or savings as a result of implementing a particular solution, although Consultant will assist in the preparation of a Cost/Benefit Analysis, and the client will need to be sure that the preferred specific solution will support the Cost/Benefit Analysis.  Consultant personnel should also avoid making any claims as to the performance of any specific solution or to its freedom from system bugs or programming errors.

How is the Decision Made?

There is no mathematical formula or algorithm which can generate a decision based on the information gathered.  The team must present the key differentiating issues to the project sponsor and key decision makers, and discuss the relative merits of the different proposals.

The information will help support the process. In particular it will be used to highlight the key differences between the solutions and the key issues which affect the decision:

• discuss any key requirements which cannot be met,
• discuss the issues identified and their implications,
• discuss the findings of the team resulting from the vendors’ demonstrations and other vendor contacts,
• discuss the comparative costs and benefits of the solutions,
• discuss the overall benefit to the organisation of proceeding with each option.

The facts must be balanced to identify the best solution, other acceptable solutions and unacceptable solutions.  Note the importance of this classification:

• The preferred solution is the one that the team believes is best suited to meet the organisation’s needs.  It may be assumed that this solution will be followed and following actions can be based on that assumption.
• Other acceptable solutions are identified to provide alternative approaches in case a problem is identified with the preferred solution.  It is also useful to continue to negotiate with all the acceptable vendors as frequently the vendors will improve their financial offers in the face of the competition.
• Unacceptable solutions should be identified where there is no point pursuing a particular solution any further as it would not be acceptable to the client organisation.  Letters should be sent to confirm to the relevant vendors that their proposals have been rejected.

The choice of a preferred solution is rarely clear cut.  In most cases there will be competing merits in the competing solutions.

It is often found that the best solution is not the cheapest and the cheapest compliant solution is not the best.  The cost benefit analysis may cast some light on this but, in practice, it is not normally possible to match the perceived greater value of a given solution to a calculated improved financial benefit.

The final balance of these issues can only be judged definitively by the project sponsor and key decision makers with the client organisation.  Where assumptions are made or where particular conclusions have been formed to support the preference, these should be noted.  The fact that the ultimate decision rests with the project sponsor and key decision makers within the organisation should be made clear to all participants and should be clearly stated in any resulting documentation.

The considered comparison and preferences will be documented formally in the Selection Report - see Process S250.