The City of Pittsburgh goes to the cloud: a case study of cloud solution strategic selection and deployment

Abstract

Faced with diminished resources and a need to provide enhanced employee email services, the City of Pittsburgh’s Chief Information Officer (CIO) had to navigate a stubborn bureaucracy to achieve his objectives. Using an IT-business alignment strategic approach to decide on a suitable IT solution, the City was able to select and deploy a cloud-based system in an objective and uncontested way. The City employed an Analytic Hierarchy Process prioritization methodology, which used various criteria obtained from the stakeholders to develop a decision-making framework to select the most suitable cloud solution. The case allows a review of best practices in the cloud solution selection process, using theoretical lenses from the fields of IT alignment, decision-making, and stakeholder management. This case study is particularly useful for learning how strategic decisions to adopt IT are made in public organizations.

Appendices

Appendix A

The AHP decision-making methodology

The approach proposed for cloud vendor selection is called AHP and was developed by Saaty (1982). This method has been widely used for selection and prioritization of alternatives and is particularly suitable for group decision-making (Vaidya and Kumar, 2006). An AHP group-oriented decision support software package, Decision Lens (2008), allowed the cloud committee to quickly develop a decision model for cloud technology evaluation, based on the issued RFP. One main advantage of the AHP methodology is its ability to measure both tangible and intangible criteria and for this reason it has been widely used in multi-criteria selection analysis (Vaidya and Kumar, 2006). The process may be described as involving the following steps: (1) model the decision as a hierarchy of goal, criteria/sub-criteria, and alternatives; (2) prioritize evaluation criteria/sub-criteria; (3) evaluate alternatives with respect to each criterion/sub-criterion; (4) synthesize the model to obtain the best alternative, and; (5) perform a sensitivity analysis. Next, a brief description of these steps will be given. The reader is referred to Saaty (1982) for a more detailed discussion of the methodology. Also, the use of Decision Lens software hides the calculations discussed here.

Developing the hierarchical decision model

The first step in the AHP approach consists of modeling the decision as a hierarchy. The hierarchy starts at the top with a goal (first level) and is followed by decision criteria in the subsequent lower level. Under each criterion, a third level of sub-criteria can be added. The bottom and last level of the hierarchy is constituted by the alternatives to be evaluated. Figure 7 shows the decision model developed for the cloud vendor selection.

Prioritizing the criteria and sub-criteria

In the second step, prioritizing or weighting the criteria, all the criteria are compared pairwise (e.g. what is more important: ‘technical requirements’ or ‘financial considerations’?) with respect to the goal at hand (i.e. selecting the most suitable cloud vendor). This is important because rarely the different criteria/sub-criteria will have the same priority for the decision-making. The intensity of these judgments is represented by a value in Saaty’s (1982) numerical scale, which ranges from 1 – equally important to 9 – extremely important in a criteria comparison matrix. Next, the weights of the criteria are derived by calculating the eigenvalues of the comparison matrix. This process can be done in two ways. In one approximate method, the matrix of judgments can be normalized and the average of each row will provide the criterion weight or priority. However, the accuracy of the results depends on certain characteristic of the matrix of judgments called inconsistency, be it 0 or very low (provided as an inconsistency index by the software). In another method, more accurate and used by software packages, the matrix of judgments is raised to powers consecutively until a limit, where all the columns are equal, is reached. Any column values (called the eigenvalues of the matrix) constitutes the criterion weights or priorities. The results are robust, provided that the inconsistency index is less than 0.1 (as reported by the software package). Figure 8 shows the criteria/sub-criteria with their respective weights or priorities.

Evaluating alternatives and synthesis

The third step consists of evaluating the alternatives (in our case, the different cloud vendors) with respect to each criterion/sub-criterion. In this step, the decision-maker has two possible lines of action. Similar to the previous step, one can compare the alternatives in a pairwise fashion with respect to each of the criterion/sub-criterion, build an alternative comparison matrix with respect to each criterion/sub-criterion, and obtain the final priorities for each of the alternatives by calculating the matrix eigenvalues. This last process of calculating the final priorities constitutes the fourth step in the AHP approach and is called synthesis of the model. This approach, however, is not possible when the number of alternatives is large, or if the possibility of adding or deleting alternatives exists. This was precisely the situation for the cloud vendor evaluation. For this reason, a different method, called the ratings approach, was used. Using this method, a rating scale consisting of the anchors excellent, good, fair, and poor was developed and each cloud vendor was rated with respect to each of the selection criterion using this scale. The cloud vendors were then prioritized according to their score in the rating process.

Sensitivity analysis

The final step, sensitivity analysis, consists of varying the weights of each of the criteria to determine how the preferences might change if the importance of selected criteria were to change. Doing this provides considerable reassurance that the outcome is stable to variations in the judgments (Saaty, 1982). The cloud vendor evaluation committee conducted the sensitivity analysis at a later stage once other stakeholders had a chance to analyze the model and discuss the results. The AHP method is particularly easy to use when it is implemented with a decision support software such as Decision Lens™ (2008); which was selected for this project due to its strength in group decision-making support as it allowed the group to quickly and visually understand the results and sensitivity analyses.

Appendix B

Table B1

Table B1 City cloud-computing selection committee members

Appendix C

AHP sample questionnaire for item 3.1, sub-item 3.1.1.

3.1. Email

3.1.1. Email City of Pittsburgh’s RFP requirements

illustration

Based on the above requirements and the vendor proposal to address them (shown on the next page), indicate to what extent you agree or disagree with the statement below:

illustration

3.1.2. Email Vendor (VENDOR 4) Proposed Solution

Google’s Government Organization branded Google Apps for Government (Google Apps for short) provides an email solution that is comprehensive, functional, and scalable. It provides the following non-optional requirements:

• a) Google has a comprehensive email environment that provides functionality, including but not limited to send, receive, format, and attachment;

• b) Google Apps is generally accessed from any widely available Web browser (Chrome Browser, Firefox, Safari, Internet Explorer, etc). Google support integration to major email clients such as Microsoft Outlook;

• c) Google integrates with mobile devices such as Blackberry, Win mobile, Android, iPhone, and iPad. Technologies such as POP, IMAP, Active Sync, BES, and Web-based emails are supported;

• d) Google Apps does not require Microsoft Exchange, or any local storage solution to operate. Using our Application Program Interfaces (APIs), Google has the ability to integrate with on-premise local storage solutions should the City require this functionality for other solutions;

• e) Google Apps has the ability to create user-defined email groups or personal folders based on search criteria;

• f) Google Apps has its own integrated archiving and discovery solution branded Google Message Discovery;

• g) Google Apps has the ability to define roles for email handling;

• h) Google Apps has the ability to add both personal signatures and notes;

• i) Google Message Discovery allows for Retention Policies to be implemented and distributed in a number of different ways;

• j) Google Apps allows for the Government Organizations to retain email. Individual Users may retain 25 gigabytes of mail. Google Message Discovery also allows Users to set their retention policy up to 25 gigabytes. Archive Administrators for the total email environment have no limit to the amount of email they would like to store based on size or time frame;

• k) Google Apps has the ability to copy, move, save, and store information to desktop or local storage. Google provides support for POP, IMAP, Google apps for Microsoft Outlook etc. These will allow users to have local copies of their email if permitted by the administrator;

• l) Google Apps has the ability to print stored information to a City facility;

• m) Google Apps support scan or fax from multifunction devices to email if the multifunction devices can connect via Pop or IMAP;

• n) natively, Google supports task creation for individual users which they control. Also there are many partner solutions that provide the ability to create, delegate, share, and collaborate on tasks. Google apps marketplace is located at google.com/enterprise/marketplace;

• o) Google Apps provides the ability to use email system remotely;

• p) Google Apps has the ability to delegate email functionality to another staff member (i.e., proxy assignments, mail folders, etc);

• q) yes, Google Apps supports these concepts. Using collaboration, users access control could be controlled for documents, spreadsheets, etc. Additionally, notifications could be created for users who are subscribed to sites where if something is added or changed they would be notified; and

• r) integrated Google Message Discovery provides ability to recall and/or retrieve within City email system and/or current/future City archiving system.

Appendix D

Model element definitions in the cloud SaaS technology AHP decision-making hierarchy

These definitions refer to the elements of the model in Figure 7.

I. Goal. The goal is to decide on a fully hosted Solution to meet the City of Pittsburgh’s SaaS Email, Data Recovery, and Data Collaboration RFP. The Solution will transition the City’s current on-premise email system to a secure, hosted email system for its approximate 3000 users. The City’s goals are to reduce expenditures and enhance productivity by ensuring that the Solution meets the following criteria.

II. Assumptions. The following are the assumptions of the City for review of the Vendors and their corresponding proposed solutions:

• 1 All Vendors and proposed solutions fulfill all the City’s requirements, including those related to disaster recovery.

• 2 It is assumed that the City has provided all the Non-Optional and Additional Technical Requirements necessary for the vendors to provide a Solution to work with City resources.

• 3 All decisions made are based on the Vendor proposal and clarifications made in writing.

• 4 Each proposed solution includes approximately 3000 City users and the corresponding number of mailboxes.

• 5 The City of Pittsburgh must address the different training requirements based on each employee group, and the Vendor is required to provide a customized training strategy for each employee group within the Solution. The employee groups are defined as:

Knowledge Worker: a City employee whose job requires 50%–100%computer use;

Occasional Worker: a City employee whose job requires 20%–50%computer use;

Light Worker: a City employee whose job requires 0%–20% computer use.

The information provided by each Vendor is true and trustworthy. Whenever there were questions, vendors have been contacted and required to reply in writing.

III. Criteria. The key criteria below will be used to decide which Vendor/Solution to choose from Vendor proposals and written documentation. All these sub-criterion will be used in the assessment of each Vendor qualifications.

1. Vendor Qualifications. This criterion refers to the vendor’s ability to implement the proposed cloud-based email solution. Four important sub-criteria to consider are: relevant experience, quality of references, vendor’s ownership minority participation and vendor/subcontractor partners’ location.

1.1. Relevant Experience. The first sub-criterion is the relevant experience of the vendor in reference to the City’s requirements. Relevant experience is comprised of:

1.1.1. Number of years. The vendor has been working with and implementing the proposed (or as similar as possible) solutions.

illustration

1.1.2. Number of mailboxes. The vendor is currently managing. Obviously, the more relevant experience a vendor has, the better qualified it will be to implement the proposed solution successfully.

illustration

1.2. Quality of References. The second sub-criterion is the quality of the references provided by the Vendor. It is expected that each of these references has worked with the vendor in the proposed (or as similar as possible) solution. The better the opinion and the more similar the solution, the better qualified the vendor is expected to be.

illustration

1.3. MBE/WBE Participation. The third sub-criterion will be MBE/WBE participation by the Vendor. The Idea of equal opportunity required for in the City of Pittsburgh Code is met by making a good faith effort in accordance with the criteria established by the City for the participation by Minority and Women Business Enterprises (MBE/WBE). The awarded bidder shall produce evidence that a good faith effort was made to include the participation of Minority and Women Business Enterprises. The City’s current goals are 18% MBE participation and 7% WBE participation. All things equal, MBE/WBE participation is preferred for the bid winner. This sub-criterion is compromised of two categories:

1.3.1. MBE Participation, that is the participation of MBE participation measured as:

illustration

1.3.2. WBE Participation, that is the participation of WBE participation measured as:

illustration

1.4. Location of the Vendor and partners/subcontractors. The fourth criterion is location of the Vendor and partners/subcontractors. This element will acknowledge the location of the Vendor and Partner(s)/Subcontractor(s) in proximity to the City of Pittsburgh, Pennsylvania. Vendors and Partner(s)/Subcontractor(s) located in the proximity of the City of Pittsburgh have an advantage to respond more quickly. Also, and more fundamentally, the City of Pittsburgh has an obvious interest in further the development of local enterprises. This criteria is comprised of the following:

1.4.1. Vendor Location. Vendors are those firms that have assumed the legal responsibility to participate in the RFP, that is, those signing the offer.

illustration

1.4.2. Partner/Subcontractor Location. Partners and Subcontractors are those firms co-participating in the implementation of the proposed solution.

illustration

2. End-User Usability/Transparency. The ability of the Vendor to provide an easy-to-use solution (transparent) for the city employees will be assessed under this criterion. This means taking into consideration that the users are already trained and familiar with the City’s current system. The most transparent (or easiest to use) solution would be one that would require either no or minimum retraining of the City employees.

3. Technical Requirements. This criterion refers to the degree of fulfillment of the technical specifications requested by the City of Pittsburgh in the RFP. Although all proposed solutions have been reviewed to fulfill the minimum (called non-optional) technical requirements, they will vary in the degree of suitability and technical excellence. Also, some solutions may offer additional technical functionalities that, although not essential, were requested as a convenient bonus. All requirements will be measured in terms to what extent the Vendor has met the City’s requirements. This criterion constitutes the core of the RFP. In any IT selection, the primary concern is that the new system will technically do what is expected to do. Two groups of sub-criteria were proposed: non-optional requirements, that is, those functionalities that all proposed email services should fulfill, and additional requirements, that is, convenient functions that, although not required, facilitate cloud-based email service deployment and use. Within each group, a detailed list of technical characteristics was included as follows:

4. Non-Optional Requirements: Refers to the degree of excellence in the fulfillment of the non-optional technical requirements. There are eight (8) subcategories to be evaluated:

4.1. Email: Solution must include basic email capabilities and functionalities.

4.2. Contact Management: Contact management functionality, current contact information transfer and management, and applications for desktops and mobile device.

4.3. Calendar: Basic calendar functionality and management.

4.4. E-Discovery: Ability to search, retrieve, store, identify, and report information.

4.5. Archive and Backup: Ability to store and retrieve live email data for a minimum of 180 days (90 days for user, and 90 days for system administrator), adhere to City policy or legal requirements, retrieve and/or E-Discovery.

4.6. Collaboration: Ability to share, store, and access files throughout the solution while providing a process flow feature for the users.

4.7. Solution Administration: Ability to meet all needs of the system administrator including, but not limited to, full access and control of administrative console, management of domain names and aliases, synchronization of email identities with internal directory, control of industry standard devices, control over email storage limits perused based on maximum storage limits, integration with internal applications, migrate Historical or user Archives from current proprietary format, and administrator distribution throughout departments.

4.8. Disaster Recovery: Solution’s ability to restore email and data within 1 h of interruption, written detailed business continuity plan, annual testing of DR plan, defined identification of roles and responsibilities for both city and vendor, and qualifications for initiating and ceasing ‘Disaster’ condition.

5. Additional Requirements. This refers to the degree of excellence in the fulfillment of the additional technical requirements.

5.14. Project Plan: A constructed and organized plan including implementation method, installation plan, maintenance and support features, and necessary resources (City and Vendor) consumed specific policies, specific plans, specific procedures, specific techniques, and general management of project plan.

5.15. Milestones: A schedule of tasks and completions that will be maintained throughout the process.

5.16. Documentation: Proper documentation has been provided by the vendor for the Solution’s components used by the user and administrator.

5.17. License: All license agreement terms, security policies, and terms of service have been submitted by the vendor.

5.18. Maintenance: All compatible software, required hardware, installation services, all applicable warranties, all maintenance, and solution updates have been provided by the Vendor. The Vendor has agreed to implement and test the entire system, and any defects or problems found by will be fixed by the vendor.

5.19. Support: The Vendor has provided technical expertise, staff, and effective procedures mandated for systematic errors. Support for the solution is the Vendor’s responsibility including but not limited to: monitoring of the solution, immediate report of any malfunction to the City, term of service for 1 year, restore all City data in the event of a system malfunction, failure, or compromise.

5.20. Training: Training options specific to the City’s three different user groups.

5.21. Ownership (Data): The Vendor either declares that it is the sole owner of the solution, or has provided proper documentation and authorization from the solution’s owner. The City remains the sole owner of all City data in the Solution.

5.22. Security: The Vendor has provided specific details, policies, procedures, compliances, regulations, and other resources related to the security plans. The City’s data will be segregated from other customer’s data, accessed only by authorized personnel, and remain in the continental United States.

5.23. Materials and Equipment: The vendor has provided all items that are required within the City’s internal network to support the solution, Service Level Agreements (SLAs) for escalation of issues/product improvement, and minimum workstation requirements for the solution.

5.24. SLAs/Sustainability: The vendor has provided details of how the SLA and sustainability required by the City has been meant in the Solution.

5.25. Reports: Various samples of reports of the solution have been provided by the vendor.

6. Financial. When thinking about financial considerations we tend to think about cloud service pricing (4.5); however, this is only one of the elements to be considered in financial evaluation. For example, the form of payment and how different this was from other vendors (distinguishing compensation – 4.2) was considered to be important, as well as the scope and specificity of services (4.1 and 4.3, respectively) provided by the vendor (which would allow savings by not having to contract them out). Finally, how stable the vendor would commit the pricing to be (4.4. Ongoing preservation) was also deemed important as part of the financial sub-criteria. The financial criterion is important because it refers to the amount of money needed to cover payment of the Solution and its associated costs for using it. The lower the costs of a solution, the more attractive the solution is.

7. Opportunities. Allows the Solution to address the Optional Requirements and provide innovative elements that would benefit the City in the long run. Consistent with the desire of remain innovative, it was expected that the new system would open opportunities for innovation in the long term (e.g. deployment of office productivity tools, video conferencing, virtual drives, unified communication, instant messaging).

Therefore, all things equal, the solution that would offer more potential for these opportunities would lead the vendor choice. In other words, the more the solution offers possibilities to enhance functionalities/services rather than proposing a static dead-end solution, the better. The Vendor is given the opportunity to show the Solution’s unique growth factors while meeting all necessary requirements within this criterion.

IV. Alternatives

Alternatives are constituted by the different cloud solution vendors that turned bids in response to the City of Pittsburgh’s RFP. To protect the identity of the vendors, they will be referred only by a number ranging from ‘Vendor 1’ to ‘Vendor 7,’ corresponding to the seven proposals submitted to the City of Pittsburgh for consideration. These alternatives are under consideration based on their official proposed solutions.