1 Introduction

There have been global trends that more people will live in vulnerable areas (e.g. floodplain), which potentially lead to more development and urbanization in high hazard regions (Cutter et al., 2008). Urban flooding caused by an increasing number of hurricanes or storms is a significant source of property loss, social disruption and inequality. Communities can largely reduce their hazard vulnerabilities and increase their social resilience through design and planning, which could put cities on a trajectory for long-term stability. However, the silos within the design and planning communities and the gap between research and practice have made it difficult to achieve the goal for a flood resilient environment. Therefore, an AI (Artificial Intelligence)-driven framework is needed to facilitate the multi-scale flood resilience design and planning. This platform, with the active engagement of local residents, experts, policy makers, and practitioners, will break the aforementioned silos and close the knowledge gaps, which ultimately increases public awareness, improves collaboration effectiveness, and achieves the best design and planning outcomes. We define resilience—adapting from previous ecology and hazard research and using a built environment design lens—as a system’s capability to experience disturbance and changes, and retain its function and structure for dealing with future hazard events (Meerow et al., 2016; Walker & Salt, 2006; Yu et al., 2020). Resilience has become an important foundation for understanding and improving built environment in an increasingly uncertain era (Vale et al., 2005).

According to Watson and Adams (2011), design for flood resilience is a multidisciplinary and multi-scale effort that involves architects, landscape architects, urban designers, water resources engineers, and others. As built environment practices require higher standards which are multifunctional, experiential and sustainable, interdisciplinary collaboration is essential. Without a seamless collaboration, the outcome often presents compromised performances and lacks full considerations (Hjort et al., 2018). More expertise in the planning and design process could help avoid costly mistakes (The Power of Collaboration: LAND, 2017). However, due to project scopes and limited budget, those disciplines often work in silos, and may not have opportunities to obtain sufficient inputs from others. Projects may fall short in different aspects when expertise is lacking. Designers and planners often face tremendous challenges and fail to make the right decision for an overwhelmingly complex project. Hence, a highly intelligent, evidence-based, data-driven system with a customizable evaluation rubric may help stakeholders shorten the lengthy evaluation/analysis process and gain the insights of possible solutions (Lähde & Di Marino, 2019). It could also promote mutual learning and communication for local communities who often lack the awareness of the flood risk (Chen et al., 2016). This is especially crucial after a devastating disaster, when the communities desire rapid recovery and a fast decision-making process is needed.

2 Challenges

Geodesign originated from overlaying a series of map layers of for a comprehensive understanding of the study site for collaborative decision making (Goodchild, 2010). Geodesign is an interactive and sequential process in the built and natural environments, requiring successive critical thinking and synthesizing across different scales (urban, neighborhood, building) and disciplines. Many practitioners advocate for multidisciplinary collaborations among three key built environment professionals (e.g., architect, landscape architect, and urban planner). Resilience largely depends on the ability of a community to deal with and adapt to unanticipated changes (Walker & Salt, 2006). Those changes will direct and influence the approaches taken by planners and designers (e.g., changes in the intensity of hazard events, effects of climate change on coastal cities, dynamics of urban forests and parklands) (Novotny et al., 2010). The landscape architecture and architectural design fields take the leadership role in building capacity on ecosystem services at multiple scales and in multiple contexts in communities (Novotny et al., 2010; Vale et al., 2005). Such changes will also be affected by the work of planners, including demographic and cultural shifts due to land use changes and development. These may also affect social resilience. Hence, a scientific framework of Geodesgin to involve stakeholders and professions in collaboratively decision making is essential in achieving a sustainable solution for spatial challenges. However, in the current research and practice, there are four major bottlenecks in planning and design disciplines targeting flood resilience:

  • Challenge1. Fragmented Research: Resilience is often studied in silos within different disciplines. However, community resilience requires an across-scale holistic investigation.

  • Challenge2. Knowledge Divides: Scholars with different specialties may not share the same language and/or criteria for approaching community resilience. However, design and planning deliverables need to be communicable and understandable by all practitioners involved.

  • Challenge3. Decision-Making Barriers: It is challenging to make appropriate decisions considering the overwhelmingly available information and complex contextual factors. It lacks a platform to filter redundant information and establish a customizable evaluation system to facilitate decision-making.

  • Challenge4. Community Engagement Barriers: The current design/planning process is mainly an expert-driven closed system. Public participation is limited due to the lack of awareness of resilience. However, knowledge co-production between local residents and experts is essential to make the most consensus-based design decision.

3 A framework

To overcome the above-mentioned challenges, a platform is needed to examine physical and spatial representations of urban scenes with various interactions between socio-economic, technological, and environmental factors. This platform is inspired by the Resilient Urban Form framework developed by Sharifi and Yamagata (2018). The urban planning (macro-level) elements include 100-year flood plains, zoning, and vulnerability assessment. The landscape architecture (meso-level) elements are concerned with the neighborhood conditions such as green infrastructure elements, vegetation, sewage infrastructure, and soft/hard surfaces. The architecture (micro-level) elements cover building-related information such as foundation, structure, and material. By incorporating these three levels, this platform can promote systematic thinking and acknowledge the overlaps and cross-scale relationships of flood resilience issues, by specifying the tasks to address the challenges raised above (Fig. 1):

  • [Task-1] (For Challenge 1 and 2) Designing and implementing a large-scale Knowledge Graph (KG) for flood resilience research. A knowledge graph is a multi-relational graph composed of entities (nodes) and relations (different types of edges). Each edge is represented as a triple of the form (head entity, relation, tail entity), also called a fact, indicating that two entities are connected by a specific relation, e.g., (on top of, close to, has). A node can have a number of attributes, such as “a house” is a node and “cape cod style” is an attribute. Our designed KG can link information across architecture, landscape architecture, and urban planning, which can generate theoretical and practical convergence across different scales. We adopt and augment computer vision, natural language processing, and geographic information system to assist and automate the creation of knowledge graph for flood resilience research with the input from domain scientists and users.

  • [Task-2] (For Challenge 1 and 2) Designing and implementing an Information Hub for collaborative and collective knowledge sharing. Our Information Hub is an interactive web system that is loaded with millions of images relevant to flooding. It serves as a central place where users can interactively browse evidence-based design options across scales, revolutionizing the resilience practices that are often isolated in different disciplinary silos. The users can share their own images, evaluations, design standards, comments on others’ opinions/standards/designs. The Hub will have the full-stack model website functionalities, such as search, recommendation, customization, and notification.

  • [Task-3] (For Challenge 3) Designing an AI-driven System to assist and facilitate the decision making process. Our system will be built on top of the graphs designed in the Task-1, which can help researchers, practitioners, and policy makers make full use of the newest advancements in AI in their decision-making process. Our system will enable users to search information based on design criteria or knowledge concepts.

  • [Task-4] (For Challenge 4) Promoting Community Engagement and Platform Evaluation. The AI-driven open platform will allow stakeholders to investigate and communicate the design criteria. The researchers can work with the community and local residents to gather data, and with practitioners to analyze and validate the data.

Fig. 1
figure 1

The Overall Platform Architecture

Full size image

3.1 Architecture for resilience

There are many sets of codes related to building and construction, such as the International Building Code (IBC), International Residential Code (IRC), International Existing Building Code (IEBC), and International Mechanical Code (IMC). Each codebook has very specific provisions on flooding design. IBC, for instance, has provisions on structure requirements, interior and exterior materials, flood loads, foundation systems, etc., many of which require years of professional training to be able to comprehend. FEMA (2018) compiled all the flood resistant provisions from the 2018 editions of various codes, which is a 53-page document and may be even difficult for an experienced practitioner to digest alone. Moreover, the American Institute of Architects (AIA) and other organizations have developed guidance and principles for resilience design. Yet, there is no such a platform that integrates these evaluation criteria.

In the architectural design process, there is a very important phase named pre-design/programming, when architects or programmers work with clients and users to identify the project goals, facts (e.g., site and climate conditions), needs (e.g., space requirements), concepts, and finally design problems (Pena & Parshall, 2012). There is also a growing interest in participatory design, an approach that fully engages stakeholders into the design process as a means of better understanding and meeting their needs (Bannon & Ehn, 2013). Most of those stakeholders are laypersons who may not understand design and planning languages nor be able to read design drawings such as plan and section. Thus, during programming or the participatory design meetings, architects often use various visual simulation tools such as image, video, animation, and/or virtual reality. These are effective means to convey ideas, initiate discussions and gather inputs. However, it is costly and time consuming for collecting and producing visual materials. Moreover, those materials, once created, are not flexible to modify and therefore, may not be useful to facilitate further discussion within a meeting. An open platform is needed to help the public and policy makers instantly visualize the impacts of a design approach and involve in efficient discussion, without being interrupted or intimidated by technical languages or jargons needed explanation.

3.2 Landscape architecture for resilience

The role of green infrastructure as a build environment solution for resiliency is highly recognized. There is a trend of transforming ‘grey infrastructure’ to ‘green infrastructure’ (GI). The Grey infrastructure represents human-engineered and centralized water management approaches such as pipes, pumps, ditches, detention ponds and drainage, and sewer systems. The efficacy of these systems is critical and requires expensive maintenance and operation as many governments are struggling to keep up with the challenges of operating effective grey infrastructure to defend flood hazards. EPA (Environmental Protection Agency) promotes the practices of GI and describes it as “natural resource management interventions that use vegetation, soils, and natural processes to manage water and create healthier environments”. The new law “Water Infrastructure Flexibility Act” was passed by the Senate in early 2019, which requires EPA to promote the option of GI and allows communities to use natural processes to infiltrate or reuse stormwater runoff beneficially on-site where it is generated. This will help increase the likelihood of putting green infrastructure principles into practice throughout the country.

However, GI faces the aforementioned bottlenecks that significantly hindered its development: (a) There is significant uncertainty around how best to plan, design, implement and maintain GI (Baptiste et al., 2015). Technological barriers or lack of tools, deficiency of performance data, insufficient knowledge, and experiences in the practice world are primary sources of difficulties, according to resilience managers. Landscape architects, who are responsible for the construction of many public spaces/lands, didn’t receive systematic education about GI (Kiers et al., 2020). Even though landscape architects are big advocates for GI, the lack of collaborations among landscape architects, architects, urban planners, and civil engineers, often hindered the implementation of GI. There is also a lack of standardized design process for site-specific situations. Instead of focusing on design standards that are a one-size-fits-all approach, standardized design processes offer design thinking models that are case specific, which is more effective and beneficial (Zuniga-Teran et al., 2020). Without a standardized design process, a resilience manager or staff turnover creates problems through the loss of past experiences. (b) GI projects usually take place where communities already have grey infrastructure in place and want to shift to GI. This leaves poor neighborhoods behind on the timeline. The current public engagement approach for planning and design may be difficult to consider for low-income residents since they don’t have leisure time and lack information to participate (McBride et al., 2006). (c) The public’s willingness to pay for GI is crucial because GI works better when it is widely implemented. However, people usually expect the government to deal with stormwater issues rather than paid by themselves (Parr et al., 2016). The government only has limited funding sources that could implement GI on a large scale. When comparing ‘gray infrastructure’, oftentimes the tipping point to favor Green Infrastructure is its ‘co-benefits’ such as beauty, health, and social cohesion through new green/public spaces. Without a comprehensive approach to analyze and communicate its benefits, community leaders often struggled to make determining decisions on GI investments. An open platform is needed to lower design development costs and mitigate the problem of lacking GI knowledge, especially for underprivileged communities.

3.3 Urban planning for resilience

An effective risk management strategy is to encourage less development in vulnerable areas. Compared to structural strategies, non-structural measures, such as land use policy strategies, are highly recommended because they can direct community development away from flood-prone areas. In addition, non-structural measures can directly influence the amount, type, location, cost, and quality of development and redevelopment. Brody et al. (2011) found that the most effective strategies consisted of setbacks, zoning, construction and building codes, and integrating flood policies into local land use plans. Zoning is an effective strategy in reducing community flood vulnerability mainly because zoning is the key to controlling new development in the United States. Zoning can also prevent certain structures from being located in hazard areas as a result of its ability to determine the building type of the new development. Moreover, zoning is crucial in preserving natural/sensitive areas that mitigate hazards, such as wetlands, natural habitats, and dunes. Zoning is the state-authorized assignment of functions for designated areas. The zoning ordinance is supposed to follow the direction of the comprehensive plan. Zoning is a tool of comprehensive planning and it is the most popular land-use control action for local government. Bengston et al. (2004) stated that open spaces had been deemed as a land use strategy for multiple uses, such as wetlands, public parks and recreation centers, buffers between conflicting land use zones, natural corridors, wildlife habitats, and water storage areas. Policy makers began to view open space protection as a means to mitigate flood since the 1990s (Randolph, 2004). Protecting open space has become a feasible non-structural solution to mitigate floods at the local level after that. In particular, protecting floodplains can stabilize natural storage capacity and reduce flood damages (Opperman et al., 2009).

Flood risk communication is a vital element of risk management. It typically includes two main phases: identifying specific locations of high flood risk, and informing those who are at risk when the event is likely to happen (Rollason et al., 2018). Both are fundamental for helping neighborhoods at risk to prepare for and anticipate flood events, which fosters a more resilient community. Thus, communicating flood risk before and during flood events is crucial to mitigate flood impacts. Stakeholders (e.g., residents, policy-makers, designers, planners) are critical in adaptation efforts and building more flood-resilient communities, playing an important role in the implementation of effective flood protection measures in urban areas. Stakeholder capacity allows a community to learn from the last flood event and adapt to the next event. In this way, the community will become more resilient. Brody et al. (2011) stated that organizational capacity also directly influences property damage from floods at the local level. Grube and Storr (2014) examine self-government capacity at the community level in places affected by Hurricane Katrina and its effectiveness on disaster recovery. Communities with high social capital and networks are more resilient to the natural disasters. Social capital is the key element that makes a difference in the speed and capacity that the community recovers from natural disasters. Greater economic support, more assistance from the government, and lower damage are the key aspects that facilitated more immediate recovery from the disaster (Aldrich, 2012). Grube and Storr (2014) pointed out that the social capacity of a community lies in four aspects: coordination between organizations, the ability of members to bridge the social capital, histories and perspectives sharing, and social network stability. Aldrich (2012) also identifies that the most effective and least expensive way to build up social capital within a community is to build up the bonds between vulnerable populations facing disasters, which is a crucial aspect in reducing social vulnerability as well. It is highly recommended that the local government set out strategies to develop organizations to build up capacity for the communities facing floods.

It is imperative for the local governments to inform and educate residents about flood exposure through an open platform. The more the public is aware of their exposure to flooding, the better the mitigation process will be. People will be more prepared if they know how dangerous it is to live in or near the floodplain. Additionally, because more informed residents are likely to have greater participation, this may also be a way to build stakeholder capacity with respect to flooding hazards.

4 Platform design and development

In this section, we will introduce the key components of our platform in detail: (1) Knowledge Graph for linking information/knowledge across scales; (2) Information Hub for crowd-based knowledge sharing and fusion; and (3) AI-driven Systems for assisting and facilitating the flood resilience research.

4.1 Knowledge graph

To alleviate the gaps among researchers, practitioners, professionals fighting against devastating flooding, as shown in Fig. 1, we aim to build a large knowledge graph (Wang, Mao, et al., 2017) to link and reason the physical and spatial representations of urban scenes with various interactions between socio-economic, technological, and environmental factors. A knowledge graph is a multi-relational graph composed of entities (nodes) and relations (different types of edges). Each edge is represented as a triple of the form (head entity, relation, tail entity), also called a fact, indicating that two entities are connected by a specific relation (e.g., on top of, close to, has). A knowledge graph acquires and integrates information into an ontology and applies a reasoner to derive new knowledge. An ontology is a template defining and representing entities, ideas, and events, with all their interdependent properties and relations, according to a system of categories (i.e., flood resilience in this project). To create such a knowledge graph, there are many technical challenges that can be addressed through AI innovations in computer vision, natural language processing, and geographic information system (Amiruzzaman et al., 2021; Ye et al., 2021). Massive resilience relevant information can reside in different forms of data types, such as documents and images. The typical documents can include, but not limit to, flood plain info, vulnerability scores, dwelling information, socioeconomic indicators, and local demographics. The image data can include, but not limit to, earth observations and street views. The key technical question is “How do we extract and organize the massive information from these two types of data?” The following activities will be conducted to answer this question:

  • First. Ontology Building. We will design an ontology using domain knowledge that links the concepts, properties, and relations in the textual and imaginal information. We have developed a preliminary flood resilience ontology that has over 170 categories of entities with properties and relations. Figure 2 shows a part of categories we have developed. Currently, we are designing our ontology using the free and open source, and collaborative platform Protégé, developed by Stanford, (Protégé, 2020), supporting effective and collaborative ontology editing and design using well-defined web standards. We plan to keep using this tool for further development with a larger network of communities and collaborators. To build the graph, we need to extract the information from documents and images for the ontology.

  • Second. Scene Graphs from Images. We will first describe our plan of extracting and organizing information from images. Images under the context of flood resilience research can contain massive rich information of design elements and criteria, such as buildings, green infrastructure, and neighboring landscape. To better understand the scenes in images for flood resilience research, it often requires the recognition of multiple objects in a scene, together with their spatial and functional relations. The set of objects and relations is often represented as a graph, connecting objects (nodes) with their relations (edges) and is known as a scene graph (Fig. 3). Figure 3 shows that an example of a scene graph that shows the relations (e.g., in front of) among objects (e.g., car_01 and house_01) with attributes (e.g., 2-storey).

Fig. 2
figure 2

The Partial Ontology for Flooding

Full size image
Fig. 3
figure 3

An example of Scene Graph of a Household Building

Full size image

We will adopt the scene graph generation research where the objective is to build a visually-grounded scene graph of a given image. The task of recognizing objects and the relationships has been investigated by numerous studies in the artificial intelligence and computer vision research fields. This includes the detection of human-object interactions (Chao et al., 2018; Gkioxari et al., 2018), the localization of proposals from natural language expressions (Hu et al., 2017), or the more general tasks of visual relationship detection (Li, Ouyang, Wang, & Tang, 2017; Plummer et al., 2017; Zhang et al., 2017) and scene graph generation (Li, Ouyang, Wang, & Tang, 2017; Li, Ouyang, Zhou, et al., 2017; Newell & Deng, 2017; Tang, Niu, et al., 2020; Tang, Xu, et al., 2020; Xu et al., 2017; Zellers et al., 2018). Among them, scene graph generation research has recently drawn much attention. A variety of solid techniques have been proposed, such as fixing structure of the graph, refining node and edge labels using iterative message passing (Xu et al., 2017); using associative embedding to simultaneously identify nodes and edges of a graph (Newell & Deng, 2017); extending message passing to identify regions for captioning and solve tasks jointly (Li, Ouyang, Zhou, et al., 2017).

  • Third. Building Graphs from Documents. We will describe our plan to extract and organize information from documents. Once the ontology is collaboratively defined with a larger network of communities, we can adopt the existing knowledge graph building techniques in natural language processing research to extract concepts, properties, and relations from resilience design documents such as building information, communities’ information, and flooding zones setting (Zhang et al., 2020).

  • Lastly. We will merge different graphs from images and documents to a large knowledge graph based on the ontology we develop in Step 1. Our plan for building the knowledge graph will only serve as the first step to achieve the ultimate research convergence. However, this first step is critical and bears significant importance to flood resilience research. It can serve as the knowledge and conceptual foundation for overcoming the above-mentioned four challenges in the current research and practice.

4.2 Information hub for resilience research

We plan to build a website that serves as an Information Hub for different communities in the flood resilience research. The Info-Hub website is designed to serve as a central place where users can communicate, share, and apply the design and planning knowledge for flooding. The website will have the following key functionalities:

4.2.1 Browsing and commenting

The hub will be equipped with millions of images and documents for flood resilience designs. These resources will be organized by different criteria, such as locations, time, topics (e.g., designs), users (who uploaded), types of resources. Users can use the above search criteria to browse the resources and add comments. For example, a comment can describe the design elements in an image of a neighborhood or a dwelling household. For a user, all of the browsing and commenting history can be recorded by our website. We will also provide the deletion function to a user who does not want the above info to be recorded.

4.2.2 Graph-based browsing

Based on the Knowledge Graph, users will be able to browse different concepts and their properties and relations relevant to flood resilience designs, and most importantly, registered users can contribute to the knowledge graph by making comments/direct edits (with permissions). Collectively, we can push the research frontiers.

4.2.3 Basic searching

The Info-Hub will provide a Google-like search engine to enable users to search all available sharable artifacts from all users on our website. Given a query to the search engine, this web system will return a list of relevant artifacts that are shown in categories. We will use an open-source search engine, Solr (Solr, 2020), to develop our system. Apache Solr is built on top of Lucene that provides all of Lucene’s search capabilities through HTTP requests. Solr has been recognized as a mature product with a broad user community.

4.2.4 Publishing and notification

Our team and our partners will constantly publish resources on this Info-Hub. For example, one of our partner companies can publish their design images to the Hub, on which other users will be able to make comments. We will also provide the subscription function to users to keep them engaged, from which they can receive notifications relevant to their interests. The users will be able to unsubscribe at any time.

4.2.5 User management

Users can register for our website. The registered users will maintain their history on the website for better self-references and use the website as a self-archive for future research. In the meantime, other users can see the profile of a registered user and follow that user’s work.

4.2.6 Personalization and recommendation

We will provide three types of simple recommendation to a user: First, Individual-level: The five most recent activities done by the user; Second, Group-level. We will group users by roles (researchers, students, local residents, companies, and agencies), disciplines and research interests. We will recommend three lists of the top-five most similar resources to those on which the user is working: one list from users having the same role; one from users having the same discipline; and one from users having the same interests. Third, Website-level. We show a list of resources (such as images, documents, comments, and design criteria) that are shared or re-shared the most by all users. We will use TensorRec (2020), an open source recommendation algorithm and framework in Python.

4.3 AI-driven systems for advanced sharing, searching, and recommendation

In this section, we will introduce our plan of building three systems with our preliminary work.

4.3.1 Annotation and knowledge sharing system

The goal of this system is to provide a place where researchers can share and mingle their flood resilience research. In addition, a large dataset with the multi-scale flood resilience designs/principles can be created as a benchmark for the whole research community.

Preliminary work of annotation systems

We have built a preliminary working prototype in Fig. 4 for planners and designers to collaboratively annotate images with the description of flood resilience design elements of a building. There are three basic strategies for flooding design, as defined by Proverbs and Lamond (2017): (1) Avoidance: selecting an appropriate site or elevating building foundation/main living area/entry to avoid flooding, (2) Water Exclusion: using barriers, water gates or other resistant technologies to stop water from entering the building, and (3) Water Entry: recognizing the fact that water will enter the building but using appropriate strategies and materials to reduce possible damages and impacts, and facilitating fast aftermath recovery. Building features related to these three strategies (e.g., elevated building foundation, water barrier, and interior water-resistant materials) can be captured by photos and street-view Images. Specifically, we have collected millions of street view images; a typical image can contain important street objects, such as front lawn, trees, and household buildings. Our annotation prototype allows users to write descriptions of resilience elements/designs of a given household building image. Every user is able to explicitly review others’ descriptions and leave comments. In this way, users working at different scales of resilience designs can have a common place to share their views. Our prototype also allows users to upload their own images with descriptions.

Fig. 4
figure 4

The User Interface of Tagme.us

Full size image

The above preliminary system and dataset have helped us to collect initial thoughts from researchers from different disciplines and develop our initial ontology. They have laid out a very solid foundation for our future work. However, they are limited in the following ways: (1) We need to expand the dataset for more knowledge to build a larger knowledge graph; (2) They are not designed for large collaborative work involving our partners; (3) They are not designed to generate the data to train our advanced AI solutions. We plan to significantly update our preliminary system in the following ways: We plan to develop a collaborative image annotation system that allows flood resilience researchers working on different levels of resilience designs/principles to add detailed textual descriptions of any resilience elements in an image of a household building. The system is designed to show an image of a household building and other satellite images of its surroundings, also societal and economic information (e.g., previous flooding descriptions and even images of flooding) of its surrounding areas. The researchers will be able to consider all sources of information with their built and linked Knowledge Graph and give annotations and share their knowledge of the flood resilience design/principles about the place. Our system is equipped with millions of images collected from the open web. The users can annotate a given image in two ways: (1) circle the areas relevant to the flooding designs with descriptions; (2) directly write textual descriptions with clear descriptions of the objects in an image. To have better human-interface interactions, we will use the state-of-the-art user interface and visualization techniques to build an interactive and user-friendly web system on top of our preliminary labeling system. The data quality control has been well developed in the crowdsourcing contributions literature, and we will adopt such protocol (Daniel et al., 2018).

4.3.2 Search system

In the information hub, we will provide functionality for searching every resource. However, the design is more visual. Thus, we will develop an advanced image-based searching system to assist researchers/users to search relevant and desired images using flood resilience designs/principles from millions of images and massive societal and geographic information. This system will enable researchers/users to obtain inspiration from massive knowledge embedded in the images and textual information. This search system will also be very useful for design/architecture education, as students can easily find relevant knowledge with visual representations for the given design ideas/principles.

Preliminary work of image retrieval system based on design elements

Using our initial dataset, we have built a preliminary and simple image retrieval system using an open source tool (Babenko et al., 2014) that takes an image as input to search similar images with labeled design elements. In Fig. 5, the image on the left side is the query image as input to our system, while the four images are retrieved from our dataset. As seen in Fig. 5, all images have the elevated stairs to the front entrance. The elevated foundation is an important element of resilient buildings. The red boxes are manually labeled after we retrieved the images, not generated from the system, which reveals several key limitations: (1) The retrieved images are not explained with design elements, which requires human labeling afterward; (2) The current system only retrieve several design elements in images; (3) The current system can only take images as input, but no natural language supported.

Fig. 5
figure 5

Retrieved Results using Our Preliminary Search System

Full size image

We will develop an image retrieval system that takes natural languages describing flood resilience design principles as a query and retrieve relevant images based on the design principles. A natural language query can describe specific flood resilience principles (e.g., different types of elevated steps of the front door/entrance) or even very general query (e.g., Can you show me resilient houses?). Our system will employ the most state-of-the-art natural language processing and image scene graph generation techniques, e.g., VilBERT (Lu et al., 2019) and Large-Scale Visual Relationship Understanding (LSVRU) (Zhang et al., 2019), to analyze the input and retrieve the relevant images having distinct types of elevated steps of a house. Recently developed using the BERT (Devlin et al., 2018), the VilBERT is trained on pairs of textual descriptions with an image to obtain a model that can translate a natural language query into a set of relevant images. In addition, the system can simply take an image as a query and retrieve the relevant images with the same or similar flood resilience principles. We will build a new novel feature extraction approach using in LSVRU to extract the flood resilience elements and scene graphs from images and use the similarity of elements/graphs to retrieve similar images. As a complementary component of the above two ways of searching, we will also collect the societal and geographic information of a given location.

4.3.3 Prediction and assistance system (PAS)

To better help the users annotate an image or search images, they may need some assistance to complete a query or a textual description, leading to a high productivity.

Preliminary work of resilience classification

Using the initial labeled dataset, we adopt a state-of-the-art deep learning based image classification, Residual Attention Network (Wang, Jiang, et al., 2017), to classify a given image of a household with street scenes to be resilient or non-resilient or unknown. The Residual Attention Network is a convolutional neural network using attention mechanism, which can incorporate with state-of-art feed forward network architecture in an end-to-end training fashion. Through our initial empirical evaluation, our classification system can achieve a very high accuracy of about 85%. Our initial classification system has been integrated with the above-designed annotation system to give users a better experience. We use the generated resilience classification results to assist and facilitate the increase of the annotation productivities. However, the preliminary work is limited in the key ways: (1) It is built for a small set of design elements; (2) It is a classification system, not a recommendation system.

With the accumulation of information, we will build a deep-learning-based annotation assistance system that facilitates the annotation of images and the creation of a meaningful query to the search system. We will design our system to cope with the following scenarios: (1) Given an image to annotate, before the users give a description, our PAS automatically recommends users a list of phrases learned from existing annotations to work on. If the users/researchers start typing into the description, our PAS can automatically recommend users a list of phrases based on what the users have typed-in and also the learning from the existing annotations. (2) Before the users/researchers start to type-in a query, our PAS recommends users with queries using user profiles (if exist) and contexts (e.g., frequent queries/ similar geo-locations, previous query from the same user). If the users start to input the query, our PAS can automatically expand the unfinished query or even a fully typed-in query from user’s perspective. The query expansion/refinement can help users to achieve their goals. We will use the most state-of-the-art ranking and expansion algorithms, e.g., Padaki et al. (2020), to develop our PAS.

5 Summary

This platform will enable users to interactively browse evidence-based design options across scales, revolutionizing the resilience practices that are often isolated in different disciplinary silos (Wyatt, 2004). This platform will tackle research barriers by promoting triple convergence research, including: (1) Theoretical and practical convergence across architecture, landscape architecture, and urban planning for flood resilience; (2) Methodological convergence across natural language processing, computer vision, and geographic information system; and (3) Disciplinary convergence across design, social science, spatial science, and computer science. Such convergence enables us to capture more complexity and facilitate systematic thinking across encountered plans.

Resilience planning and design involve a great amount of expertise and experiences. However, an individual is limited in his/her own understanding and knowledge of the specific resilience issue he/she is facing. Therefore, designers always spend lots of time looking for references and case studies during their creative and intellectual exploration process. The popularity of Pinterests is a great example here. Our tool could be a destination platform for people who are working on resilience mitigation or recovery projects. We could tailor our tools and test them for our target users, such as architecture, landscape architecture, urban planning, and real estate professionals. Moreover, our query tools were structured with domain knowledge in multiple disciplines, users could query their interests and we could provide relevant information of perspectives in multiple disciplines to encourage multi-disciplinary collaboration. This is a natural way of multi-discipline convergence and education, because the best way to learn this is through live examples.

Automatically generated supporting design options and materials could be a significant advancement for the design, construction and planning industry. By leveraging AI and computer vision accessible to practitioners and the general public, this platform can help retrieve information associated with urban resilience for users. The benefits here are similar with platforms such as Simpleanalytics.com and Strava.com that are the nature of large-scale data and data visualizations. Because we build datasets and conduct analysis covering large areas, we will be able to reach a large number of people. There will be potential to build network effects to connect people who concern urban resilience. The AI-driven open platform will allow stakeholders to investigate and communicate the design criteria.

Our platform will allow stakeholders to participate in evaluating and building their own resilience standards. We combine local knowledge with expert evaluation to achieve a sustainable future. Through revealing fundamental design principles with implications for actions, our research can also improve the nation’s flood resilience, in support of science-based measures for accessible, affordable, and universal design interventions. More comprehensive knowledge of how cross-scale resilience design criteria influence adaptive behavior can inform more effective approaches as planning practitioners reevaluate their community network of plans (Berke et al., 2019; Yu et al., 2020).