AI for Solution Design
Before we explain how AI could be leveraged to dramatically transform the Solutioning phase, it is important to outline the role of this phase across the Service Delivery lifecycle as well as the major personas involved.
Before we explain how AI could be leveraged to dramatically transform the “Solutioning” phase, it is important to outline the role of this phase across the Service Delivery lifecycle as well as the major personas involved.
The Technical Solution Manager (“Adam”) is responsible for mapping client requirements and baselines (volumes) into a complete solution and ensuring the right price point is met
The domain Subject Matter Expert (“Jeremy”) is in charge of defining architectural patterns, blueprints, and best practices for his own specific domain (e.g. Storage, Network, Change Mgmt., …). Jeremy also acts as primary “information source” for Adam during solution definition.
The Client provides requirements (generally in textual documents)
The process is typically initiated by the client issuing a “Request for Proposal” (RFP) accompanied by a set of documents which contain the client’s point of view concerning the IT services to be provided as well as the detailed requirements (functional and nonfunctional, i.e. constraints) that must characterize the solution itself. The information contained in these documents is largely unstructured and quite often needs a lot of clarifications in a quite long convergence process.
In a subsequent step, Adam, the Technical Solution Manager searches for relevant information on the provider’s service delivery capabilities to fulfill the client’s requirements. This step relies a lot on tacit knowledge that is scattered, neither codified not mapped. In most cases, success depends on the skill of the individual practitioner and/or on her ability to connect with the relevant players and SMEs. Moreover, the solution design suffers from a lack of data-driven evidence and references to similar solutions that are already in the Operate phase of the life cycle.
In addition to solution creation depending on the individual’s knowledge and network, tool support is fragmented and disconnected. As a result, process execution is very waterfall and sequential. Issues or changes requested by clients in at one step generally require a “back to square one”. This greatly limits true innovation in the arrangement of the solution (tradeoff assessments, what-if analysis, optimization) and severely inhibits the degree of collaboration or even co-creation with the client himself.
Speed and business agility: The entire flow is basically now transformed into a set of micro-feedback loops that build one on top of the other along a very agile construct. This ultimately leads to shorter solutioning times and a far better way to adjust to sudden and unexpected business conditions (eg. new requirements, security threats, etc.).
Continuous collaboration across the major parties and stakeholders: Every persona (e.g. the solutioner, the architect, the client, …) has the right context to collaborate and contribute; iteration and innovation are fostered and supported by the Cognitive Solutioning Knowledge base.
Continuous learning: The system is constantly learning; for example, the knowledge acquired in previous engagements makes the system perform better in following iterations. In this way, the huge set of data available in IBM, for example architectural blueprints, solution building blocks, data coming from the universe of client implementations etc. could be fully harvested for new solutions.
Social curation: We assert that it is critical to truly leverage the technical community as “social curator” of the entire Cognitive Solutioning Knowledge base to be able to capitalize on the important expertise and ‘real life’ experience that every practitioner is gathering every single day and that would be otherwise lost or relegated to scattered and anecdotal conversations.
Client co-creation and intimacy: We have observed that the benefits coming from the Cognitive Solutioning approach go well beyond merely the Solutioning phase and are completely changing the game in terms of client relationship as well. This allows a true co-creation with the client and a deeper level of interaction and understanding of the client landscape.
In the following we describe in detail the technical challenges and their AI-based solution in the Cognitive Solution Designer Tool.
Extraction and Topical Classification of Requirement Statements from Client Documents
As a first step, in the Cognitive Solution Designer tool, the service types and attributes requested by the client need to be identified in and extracted from the text documents as well as topically classified. For this, client documents of different file types are loaded into the system and transformed into a structure-preserving sentence-level representation for extraction, i.e. for each sentence its association with a subsection, section or list is preserved. Subsequently, functional requirements (“which services?”), nonfunctional requirements (“under which constraint?”) and baselines (“volumes for the service”) are extracted and topically categorized.
The accuracy of the proposed solution critically depends on the initial requirements extraction step and topical classification step: a mediocre level of accuracy not only negatively affects the downstream solutioning process but it also diminishes the confidence of the practitioner.
Example sentences from RFP documents
The client looks to the service provider to perform all remote operations services, such as common software image development, anti-virus, patch management and other associated services.
Provide requirements for high-level dashboard performance monitoring, including real-time and historical analysis
Establish system monitoring thresholds (e.g., utilization, capacity) to track data center system components (servers, storage, databases, etc.) performance in accordance with service levels
The client expects the supplier to provide a realistic approach to meeting the timeline, and will be evaluated on their ability to execute that approach.
Collaborative computing services are the activities associated with supporting collaborative tools (e.g., MS exchange/outlook).
All RFP responses must be provided no later than the date and time specified in Table x.
Topical classification of requirement statements. In order to identify matching service capabilities, the extracted requirements statements need to be topically classified. Several aspects make this classification problem especially challenging, yet it is expected that this challenge is present in many requirement-to-delivery capability matching scenarios. First of all, to match the appropriate capability to the requirement we need fine-grained classes. In particular, we have about 20 classes at the first level, e.g. “Enterprise Security”, and about 300 classes across the first and second categorization levels, e.g. “Enterprise Security – Identity and Access”. Consequently, large numbers of training data are needed for any supervised classification approach. However, unlike in other domains, like movie sentiment classification or labeling of pictures, such training data did not exist prior to this effort. Training data needs to be created for the purpose of the project and involves a massive labeling effort. Third, classification of a sentence or a functional requirement needs to be done in the context of a paragraph, subsection, or list, as the information contained in the statement itself sometimes proves to be insufficient.
Identification and extraction of sentences containing requirements
As mentioned in the challenge section, identification of requirements merely based on the choice of independent verbs is insufficient. However, verbs do contain a strong differentiation between requests and definitions and descriptions. Consequently, we follow the identification approach described in : identification of sentences with (1) verbs indicative of a request for service and (2) learned linguistic features for refinement. A third step to the approach in  was added comprising a final filtering using binary classifier to reduce false positive identification.
Step (1) is solely based on a positively-labeled set of sentences, i.e. sentences containing requirements statements. From these data sets, we determine the independent verbs and store them as <responsibility verb|true>. If a verb encloses other verbs, it is stored as <responsibility verb|false> unless there exists at least one instance of the verb as independent in which case it is stored as <responsibility verb|true>. This algorithm creates an inclusive set of responsibility verbs, i.e. a sentence containing a responsibility verb is a necessary but not sufficient condition for it being identified as a requirement sentence.
In the second step, the linguistic features indicative of requirements are learned. The training set contains all labeled sentences with responsibility verbs. Feature candidates consist of ordered sequences of POS tags, semantic role features of the verb of interest , and language token types. It is immediately clear, that a feature space created in this manner is very large and sparse. Thus we have chosen a Winnow classifier which is theoretically guaranteed to quickly converge to a correct hypothesis in a setting with many irrelevant features, no label noise, and a linear separation of the classes . It belongs to the family of online learning algorithms thus also enabling immediate updates from feedback.
In the third and final step, we prune false positives from the requirement candidate set. The majority of such false positive identifications are requirement statements which do not pertain to a requested technical service but to the way the service provider needs to submit the offer, e.g. the last statement in Table 1 They use a similar set of independent verbs and have a similar structure and will thus be identified as requirements in steps 1 and 2. Yet they are irrelevant for selecting the correct technical capability for the solution. We have found that training a final binary classifier e.g. gradient boosting machine on bag of word representation of the section title is most efficient and effective to distinguish technical from process requirements.
Precision and Recall for Requirement Extraction
Topical classification of requirement statements:
To bootstrap the classification, together with senior SMEs, we developed a lexicon with single keywords and multi-word key phrases for all classes in the 2-level class hierarchy. In the initial version of the tool, sentences were classified using this lexicon and by applying a rich set of rules to evaluate position-based occurrence of the lexicon key phrases with the learned weights for these rules.
To continuously improve the accuracy, we enable the user to add/remove/change tool-identified requirements and their classifications and thus provide feedback on the tool-generated classification. These manual corrections serve the immediate purpose of correcting the identified requirements for the RFP documents in process. The corrected and validated set will subsequently be used to select the matching service capabilities. At the same time, the user feedback generates new training samples to build a training data set sufficient for supervised classifiers and improve the accuracy.
To boost the user experienced accuracy and focus the user feedback in an active-learning manner, similar to , we leave sentences unclassified where classification confidence is low.
To switch from a manually-created and maintained lexicon we evaluated a number of supervised machine learning methods and found convolutional neural networks (CNNs) to perform better than additive tree models (ATM) training using a bag of words (BOW) representation in the domain embedding space even with smaller training data sets.
As discussed, in some cases, the information contained in the sentence itself is insufficient to yield an accurate enough classification. If the confidence is below a threshold, the classification of surrounding sentences belonging to the same subsection as well as that of the subsection title is taken into account. This approach leverages the structure-preserving parsing of incoming documents. All parameters of this approach are optimized during the model selection phase.
For the requirements extraction and classification, we strongly focused on creating a viable first version and learning on the job. To this end, the first deployed instances of the tool could be partly based on applying rules for extraction and classification. Yet, even in the very first version, the user would be guided to perform selected corrections with the dual purpose of creating a validated set of requirements for the downstream solution creation and the creation of a larger training set. As soon as the training set becomes large enough to yield sufficient performance with supervised methods, these could be put into the field and trained directly from subsequent user feedback for continuous improvement. In our view, the pragmatic combination of rule/regular expression based methods and best in class neural network algorithms allows to overcome the initial lack of training data and excessive cost of dedicated label creation.
Matching Client Requirements to Service Capabilities
To achieve a deeper comparison between client requests and service capabilities bundled in Offerings, we further distinguish between two types of requirements: (1) functional requirements, i.e. “what” types of services are requested, and (2) non-functional requirements, i.e. “how” these services should be delivered. Nonfunctional requirements include constraints like regulations and delivery countries and design points like service level agreements on availability or resolution times for issues. While the textual and conceptual representation of the functional requirements is varied and complex, nonfunctional requirements can be described through fewer concepts and thus lend themselves to a structured representation.
In the following we first describe the methods used for the extraction of nonfunctional requirements into structured representations for a deeper comparison described above. Subsequently, we discuss solutions to the open questions around deeper comparisons for functional requirements.
The first step relies largely on the word forms and work lemmata as well as the POS tag and its word vector, while word forms, POS tag, and dependency between words are most important to identify the other attributes of an SLA.
For the assessment of functional requirements, it is rather infeasible to parse them into an ontology to assess whether the service bundles have the right capabilities. Conceptually, text entailment methods applied to requirement statements from RFPs and capabilities from the service Statements of Work are most promising. However, these are based on training deep neural networks and thus require large amounts of training data that is expensive to create from scratch. We consequently opted for an intermittent step of equipping the user to assess the fit for purpose and while doing so creating training data for future text entailment models. In particular, we plan to determine the most similar capability statement for each extracted functional requirement for visual assessment and user feedback.
Unlike in the case of Software Requirement parsing, requirement statements in the IT Services RFPs – and in other complex service relations e.g. construction – can be very diverse and the underlying concepts very complex. We have thus opted to not attempt a full parsing of both functional and non-functional requirements after extraction and topical classification. Rather we have limited the adoption of SRL for the constraints and design points contained in the non-functional requirements and plan to leverage text entailment models to assess whether capability statements entail, i.e. are compatible with service requests.
Social Curation and Continuous Learning
The actual business value provided by the tool is of course very dependent on the ability of the Cognitive Solution designer tool to effectively capture the requirements and be able to deal with the various nuances that are present in every client driven documentation and that may vary depending on the specific service area or even on the client geography.
During the early test phases, it became immediately clear that the standard “una tantum” training approach was largely insufficient and we had to augment it with a “continuous learning” mechanism. We needed to make the “learning process” dynamic and continuous so the Cognitive Solution Designer tool can continuously improve and better adapt itself to changing business conditions. In other words, we had to find a way to elevate the “training phase” of the Cognitive Solution designer tool: from being a “one off” phase performed in the lab typically at the beginning of the project to become an integral part of the e2e DevOps process for the tool under the “continuous improvement” cycle.
Services organizations, like IBM Global Services, are large and have many solution architects, so it is in general very difficult to unify the entire knowledge and expertise of such a technical community. Therefore, the second challenge resides in the ability for Cognitive Solution Designer to have the technical community as primary actor behind the “continuous learning and improvement” cycle – not just for extracting and topically classifying clients’ requirements but also to select the best matching offerings and services. In this way, the Cognitive Solution Designer becomes not only a tool to dramatically streamline the solution lifecycle but also a powerful catalyst to federate and organize in a cohesive and machine consumable knowledge base the entire set of service expertise that would have been otherwise scattered and confined just to people heads.
The great value of the “Social Curation” approach mentioned in the previous challenge may be severely undermined if we do not also include a strong governance system that “accepts” user feedback which will ultimately lead to a better accuracy level for the Cognitive Solution Designer tool and automatically reject annotations that would produce a “quality drift”.
We address the challenges outlined above with three solution concepts: (1) capture user feedback pervasively, (2) design the appropriate incorporation of the user feedback, (3) design methodologies to prevent quality drift through either poor or malicious training feedback.
Capture feedback pervasively
User feedback is essential to create an ever improving and evolving system. Yet, explicit feedback from users might be hard to motivate if it involves a large effort from the user. Thus, we designed and redesigned our user feedback mechanism aiming at a perfect balance of minimalism and usefulness.
The continuous feedback collection described above allows for the continuous improvement/continuous learning in all machine learning and knowledge driven decision support models in the tool. However, the update strategy and learning algorithms depend on the model specifies. For the requirement extraction where we use an online-learning scheme, updates to the machine learning algorithm will be performed incrementally and near-real time. For requirements classification where we use a deep-learning based approach, updates are performed regularly by retraining the network. Going forward, to increase the computational efficiency, we are looking into applying incremental learning approaches that have been recently developed for deep neural networks . Another aspect of continuous learning based on user feedback is the prevention of drift or degradation of the training corpus. Here we are applying and extending existing approaches to clean ground-truth data set [5, Agarwal 2007] to reduce the labeling noise and keep it low in the continuous learning on the job deployment.
The ability to capture “implicit” user feedback (typically monitoring user undo or retries) in order to further tune the underlying engine is today becoming more and more common in AI systems (typically chatbots).
With the “social curation” approach we have implemented for the Cognitive Solution designer tool we have moved one step further: we solicit and incentivize the technical community to contribute and augment the overall knowledge and company expertise in solution design in an open and highly collaborative fashion. This has not only produced better business outcomes, but has also become a motivating factor for the solution architect workforce who find personal reward in contributing to the success of the overall organization.
Coverage of a quite extensive and broad set of scenarios, personas and “interaction styles”. In fact, as emerged from the various Design Thinking sessions we had at the beginning of the project, the set of user scenarios that CogSol must realize covers a very diverse set of use cases (requirement mapping, solution assembly, data curation, knowledge management, etc.) involving a set of personas with very different goals and professional backgrounds (solution architects, domain subject matter experts, deal manager, etc.)
Main realization of the “outer loop” for the entire Platform. CogSol has been designed not only as a key application that would streamline the entire solutioning phase but also the core component of the “outer Loop” for the entire Platform that would provide “knowledge driven service improvement” capabilities to all the rest of consumable services hosted on the Platform.
Data driven approach. As described above the very large majority of use cases in CogSol rely on data manipulation and transformation (for example ingestion of RFPs / RFS into the system or annotation of data corpus). CogSol must provide a very efficient way to handle these use cases both from a “quality of service” standpoint (e.g. performance, latency, availability, etc.) as well as from a “data semantic” perspective (eg. domain entity recognition, data abstraction, data composition in higher representations, etc.)
Knowledge management foundation for the Platform. CogSol needs to provide the basic primitives in terms of knowledge management for the rest of the Services Platform in terms of knowledge representation, knowledge manipulation and knowledge federation.
Upfront definition of major scenario flows (with links to use cases and key personas) with articulation of the most relevant “interface” points and mapping with key high level architectural components.
As an example the figure below represents the high level view for the key components and services concerning the RFP phase.
- (b)Adoption of an MVC (Model View Controller pattern) approach based on micro-services.
Definition of groups of related micro-services (especially the ones with similar responsibilities and working on adjacent data), together in higher level «services». Assignment of clear responsibilities and APIs to «services»
Leveraging the Kubernetes framework
Shielding the details of data-model and data structures through a data service that provides forward compatible REST APIs and data helpers.
Implementation of a flexible Data service that is a modular component that can be split into different data services that can be used by other components to provide data-persistence and retrieval function in a standard way
This approach has also enabled lot of reuse of existing components through simple refactoring
The following picture expands the RFP use cases shown above and presents the linkage with “micro-services groups” and data layer.
- (c)DevOps chain. We have defined an e2e DevOps chain to provide a continuous improvement process at two levels:
Delivery process: Continuous-integration build process, automatically triggered each time a developer pushes new code into a GitHub repository and then managed through the various DevOps chain stages till promotion into production; we are mainly leveraging UrbanCode as well as the core of the DevOps toolchain available in the Platform to manage this delivery process
Continuous knowledge refinement: CogSol is not only a typical aaS application that needs to live on the Platform and leverage the e2e DevOps continuous delivery process just described. CogSol also enables somewhat “continuous delivery and refinement of knowledge “powered by our practitioner community thru the annotation and “social curation” methods described in this paper. This is the reason why we have augmented our DevOps chain to include all the steps and mechanisms needed to keep our CogSol knowledge base always fresh and up to date
In the “Cognitive Solution Designer” tool, we have developed and applied AI technologies to fundamentally transform the IT Delivery solutioning process to enable speed and agility in understanding the clients’ requirements, co-creation of solutions with the client and continuous learning from the practitioners using the system. In order to create such a system, we solved and continue to solve NLP and machine learning challenges from identifying client requirements in large text documents to deep matching of such requirements with service capabilities to establishing efficient and robust continuous learning approaches.
We’d like to thank the whole “Cognitive Solution Designer” research and development team and all visionary supporters of this project.
- 1.Nezhad HRM, et al. eAssistant: cognitive assistance for identification and auto-triage of actionable conversations. WWW2017.Google Scholar
- 3.Nigam K, Hurst M (2004) Towards a robust metric of opinion. In Proceedings of the AAAI Spring Symposium on Exploring Attitude and Affect in Text: Theories and Applications.Google Scholar
- 5.Roth M, Klein E ( 2015) Parsing software requirements with an ontology-based semantic role labeler. Proceedings of the 1st Workshop on Language and OntologiesGoogle Scholar
- 6.Käding C et al (2016) Fine-tuning deep neural networks in continuous learning scenarios, Asian Conference on Computer Vision. Springer, ChamGoogle Scholar
- 7.Sumeet A, et al. (2007) How much noise is too much: A study in automatic text classification. Data Mining, 2007. ICDM 2007. Seventh IEEE International Conference on. IEEE.Google Scholar