Keywords

1 Introduction

‘Can machines think?’ asked Alan Turing, a British polymath who explored the mathematical possibility of artificial intelligence (AI). In 1950, Turing discussed how to build intelligent machines and test their intelligence (Turing A., 1950). But what is AI? John McCarthy, a prominent computer and cognitive scientist, coined the term in 1956 when he held the first academic conference on the subject. Although many definitions span different disciplines, from philosophical to very applied, AI can be defined as the computer science field that attempts to develop machines (i.e. algorithms and robots) with human-level intelligence, hence creating machines that can ‘think’.

Two of the most well-known subfields of AI are machine learning (ML) and computer vision (CV). Both subfields enable algorithms to automatically analyse, learn from, and/or derive meaningful information from patterns in data and take actions or make recommendations based on that information. The field of ML includes deep learning, where a large amount of data is used to teach a neural network, consisting of multiple layers of increasing complexity and abstraction, how to perform a task. It also includes reinforcement learning, a type of ML that learns by doing to identify the best solutions based on rewards and penalties. Figure 4.1 illustrates a diagram of AI and subfields discussed in this chapter.

Fig. 4.1
A Venn diagram of Artificial intelligence and subfields. It is labeled Artificial Intelligence, Machine Learning, Reinforcement Learning, Deep Learning, and Computer Vision.

Artificial intelligence and subfields. Machine learning (ML), computer vision (CV), deep learning (DL), and reinforcement learning (RL) are all subdomains of artificial intelligence that can be employed together for solving perceptual tasks. The decision to combine them depends on the nature of the task

Full size image

The field of CV focuses on digital images, 3D point clouds, and other visual inputs. Although CV can exist without ML to solve tasks related to photogrammetry (such as measuring distances, area, or volume), when combined with ML, it aims to develop algorithms with human-like visual perception and reasoning. In 1966, the Summer Vision Project at the Massachusetts Institute of Technology attempted to reach human-like visual perception (Papert 1966). However, the task proved to be harder than expected. Decades later, we are still far away from reaching this goal even though substantial progress has been achieved with the curation of large datasets (e.g. Deng et al. 2009), the development of more powerful computation, and the advancements in deep learning algorithms that began around 2010 (e.g. Krizhevsky et al. 2017). Regardless, CV and ML research have matured significantly and entered the daily lives of consumers (e.g. fingerprint identification, face recognition, virtual reality games, and more), as well as many industries (e.g. retail, agriculture, manufacturing, medicine, and more). In this chapter, we examine the impact, potential, and present limitations of CV and ML on a circular built environment.

Throughout the chapter, we refer to using CV algorithms for recognising information on visual data. (Hereafter and for simplification, algorithms that address visual perception will be considered as belonging to the CV field, even if they contain an ML component.) But how do these algorithms work? Commonly they are supervised learning techniques, i.e. during the training process, they are given paired data points of inputs (raw data) and correct outputs (the answer that needs to be predicted, e.g. the type of an object or a material) to learn the task at hand. Given a random unseen input, the goal is for the algorithm to predict the output as accurately as possible. The most commonly used algorithms for the above tasks are convolutional neural networks, which is a type of deep learning network that can operate on image data. This means that datasets with numerous and diverse (paired) data points need to be curated that allow the algorithms to gain generalised (i.e. broadly applicable) understandings of varying real-world scenarios. However, this curation is a difficult process, and, depending on the task, one must carefully design the collection and annotation process that creates the paired data points.

We specifically discuss, among other topics, recognising defects or materials and their attributes in the context of the construction industry. This corresponds to the following CV tasks: image classification, object detection, and/or semantic segmentation. Image classification is the task of predicting a single object label for an entire image in which the object of interest is highly represented in it. Object detection is the task of finding object instances in an image and localising them on it in the form of bounding boxes (2D boxes that tightly include all pixels of each identified object instance). Semantic segmentation is similar to the previous task, but instead of localising an object with a bounding box, an object label is given to each pixel that corresponds to it. It should be noted that, in all cases, an object can also be a material or any other type of semantic information. In their basic formulation, all three tasks follow the close-world assumption. This means that the algorithm has knowledge of a fixed set of object labels during training, and any object outside of that set is considered unknown or irrelevant. Hence, the algorithm is only able to recognise objects that it has been explicitly trained on and cannot recognise new or unexpected ones.

2 Computer Vision and Machine Learning in the Built Environment

Present use of CV and ML at the different life cycle stages of the buildings (whether operation, construction, or design) shows the potential that such algorithms can have in the domain of architecture, engineering, construction, and operations (AECO) towards multiple building lifetimes (Hong et al. 2020).

The operation stage is one of the first life cycle stages for which CV and ML algorithms were developed. In this life cycle stage, algorithms contribute to high-performance building operation. The term ‘high-performance building’ means a building that integrates and optimises all major high-performance building attributes, including energy efficiency, durability, life cycle performance, and occupant productivity (Act, E. P., 2005). The way such algorithms contribute is by measuring, modelling, and predicting building performance for existing buildings based on metrics such as energy consumption and loss (Macher et al. 2020; Barahona et al. 2021; Mirzabeigi et al. 2022; Rakha et al. 2022; Mayer et al. 2023; Motayyeb et al. 2023) and occupant trajectory and density (Sonta and Jain 2019; Tien et al. 2020; Sonta et al. 2021) to develop better strategies and processes for building operation. Temperature, stress, and ventilation levels, among other indicators, are similarly employed to achieve better user comfort (Acquaah et al. 2021; Ashrafi et al. 2022).

Digital twins of buildings are being developed to provide more insights into and control over building operation processes. (A digital twin is defined as the geometric, semantic, operational, and organisational twin of a real-world building in the digital world. See more in Chap. 3 by Gordon et al.) They connect metrics, such as the aforementioned, with spatial and building system information for more accurate modelling and prediction. Presently, a digital twin is a manually updated building model that reflects the as-is state of the building, and in most cases, it is an adapted, as-designed building information model (BIM) of the building. Research has made steps to automate this process with the use of CV and ML for both new and existing constructions (Bassier and Vergauwen 2020; Chen et al. 2022; Ma et al. 2022; Pan et al. 2022; Kim et al. 2023); however, as mentioned in Chap. 3, the capability to fully replicate a building automatically has not been entirely reached.

During the construction life cycle stage, CV and ML are used to monitor and quantify in detail construction progress (Kim 2020). This level of monitoring is not achievable through traditional practices which lack automation. These technologies can improve construction scheduling by connecting, optimising, and spatially distributing sequences of tasks given prior knowledge (Fischer et al. 2018; Amer and Golparvar-Fard 2021; Awada et al. 2021; Fitzsimmons et al. 2022). They can, for example, automatically count material loaded in trucks to verify delivery to the construction site without the use of a human (Yella and Dougherty 2013; Sun et al. 2021; Li and Chen 2022; Li and Zheng 2023), or machinery (e.g. dump trucks and excavators) can be autonomously navigated and operated to relieve humans of dangerous tasks and to provide labour despite an ageing workforce or pandemic situation (Guo et al. 2020; Ali 2021; Eraliev et al. 2022). They also help ensure the safety of workers by predicting hazardous settings when given visual data with respect to their surroundings and activities (Liu et al. 2021; Pham et al. 2021; Tang and Golparvar-Fard 2021; Koc et al. 2022; Alkaissy et al. 2023; Xu et al. 2023). Here again, the most advanced solutions are still in the research stage, with many industry players relying largely on manual and time-consuming work and with a few start-ups leading new efforts towards automation or integrating AI into construction work. Developing robust, safe, and generalisable solutions that automate entire processes remains a challenge. Several technical questions are unsolved, and additional challenges include the need for hardware located in strategic positions to track on-site activities of humans and machines; problems with illumination conditions, clutter, and occlusions that prevent from using CV on site; the absence of clear regulations and guidelines with respect to capturing and monitoring human activity on sites; and the resistance of construction workers and contractors to introducing these technologies on site (especially given the absence of regulations and guidelines).

The design stage is one of the most recent stages to benefit from CV and ML technology. Although expert-based AI algorithms have been developed since many years for the design stage (Maher and Fenves 1985; Fischer 1993), data-driven methods have mainly started to appear since the development of generative adversarial networks (GANs). GANs are a type of deep neural network that allow the generation of new data points by synthesising learned attributes and patterns given an underlying dataset. These networks are mainly used to generate 2D facades or small-scale simplified floor plans (Chaillou 2020; Chang et al. 2021; Nauata et al. 2021; Bisht et al. 2022; Sun et al. 2022; Yin et al. 2022; Zhao et al. 2023). These approaches demonstrate promising results, but there are certain shortcomings. GANs are well known for being hard to train and for suffering from mode collapse where, no matter the input, the algorithm will continually produce the same exact result.

The use of CV and ML goes beyond the spatial scale of the building: to the smaller scale of materials to the larger scales of neighbourhoods and cities. In addition to counting materials, CV algorithms are employed to recognise the type and spatial extent of material in images (Bell et al. 2015; Upchurch and Niu 2022). Most applications recognise the material of objects in daily indoor environments with the goal of enabling robotic agents to perform tasks. For example, the knowledge of the material on a traversable surface can inform the agent of how much friction can be expected (Suryamurthy et al. 2019; Hosseinpoor et al. 2021; Guan et al. 2022). Also on the material scale, CV algorithms have been used to understand the condition of materials, to detect defects (e.g. cracks on concrete), and to monitor the effect of time on them. This is very commonly applied in infrastructural settings, e.g. in tunnels, roads, and bridges (Dung et al. 2019; Fan et al. 2019; König et al. 2019; Xu et al. 2019; Li et al. 2020; Liu et al. 2020; Lei et al. 2021; Yu et al. 2021). In the scale of neighbourhoods and cities, CV is used to create 3D maps with the use of drones and satellite images, respectively, as well as to identify the land use type and material of a patch of land (e.g. vegetation, house, road) (Albert et al. 2017; Diakogiannis et al. 2020; Rousset et al. 2021) and to detect changes that take place over time (Shi et al. 2021; Zheng et al. 2021). Reinforcement learning is often used in the scale of cities to define new or renewed models for different types of transportation (e.g. private vehicles, buses, trams, trains, etc.) (Haydari and Yılmaz 2020).

Although the aforementioned applications progress the state of practice in several tasks that relate to creating and maintaining the built environment, they do not create a cohesive ensemble. In almost all cases, applications of CV and ML are viewed in isolation, without considering potential information exchange or the creation of standards for the collection, sharing, and processing of this information with different stakeholders. The AECO domain typically approaches projects in this way – in a siloed and non-centralised manner where knowledge learned from one project is limited to the people involved and not in processes and data.

3 Computer Vision and Machine Learning for a Circular Economy

CV and ML methods can be used to foster a transition towards a circular economy (CE). We will be approaching this from three levels: micro, meso, and macro (Ghisellini et al. 2016). The micro level focuses on a particular company, consumer, or product; the meso level on eco-industrial networks or supply chains; and the macro level on regions, cities, and municipalities. We will also approach this from four main resource loop strategies: narrowing, slowing, closing, and regenerating the loop (Bocken et al. 2016; Çetin et al. 2021).

3.1 Narrowing the Loop

Researchers have been investigating how to narrow the loop by minimising resources via improving efficiency (Bocken et al. 2016; Çetin et al. 2021) on several aspects of building performance during the design process. This commonly involves developing techniques that generate designs by jointly optimising several variables, such as those related to structural integrity (Kraus et al. 2022; Málaga-Chuquitaype 2022), fabrication (Ramsgaard Thomsen et al. 2020), waste (Akanbi et al. 2020), energy (Płoszaj-Mazurek et al. 2020; Di Natale et al. 2022; Tien et al. 2022), wind (Kareem 2020; Wang and Wu 2020; Li and Zheng 2023; Li and Yi 2023; Zuo et al. 2023), and more. The goal is to generate a design that serves exactly what is needed by the project definition. This line of work integrates domain expertise and proven mathematical or physics theorems to guide the design process. Such prior knowledge is essential for ML execution; although data-driven methods have the advantage of learning from real-world data distributions (i.e. learning from how the real-world looks and behaves), if left unconstrained, there is a high probability that they will make associations from data that do not lead to plausible solutions. Thus, approaches have employed physics-based reasoning, reinforcement learning, and other domain knowledge and constraints to steer the learning process.

Other researchers are exploring new methods of design and construction by using waste materials from the demolition of structures with the help of material-informed reinforcement learning (Huang 2021). However, the process includes time-consuming steps, such as a full high-resolution scan of each material fragment in isolation. In the future, easier pipelines should be explored that do not require high-end and expensive capturing systems in lab settings. Instead, pipelines and systems should be highly automated, operate in different conditions, and employ commodity devices. Fragkia and Foged (2020) proposed a new integrated circular design workflow that employs ML to predict fabrication files of material components for robotic production that follow specific material performance and design requirements. Such an approach optimises material distribution and promotes material economy. Akanbi et al. (2020) developed a deep learning method to predict the demolition waste of a building given basic building features, such as gross floor area, volume, number of floors, building architype, and usage. The authors demonstrate the use of the method to predict waste from four different design alternatives of a building. The above can be categorised as being at the micro level.

At the macro level, Cha et al. (2023) developed an ML algorithm to predict the demolition waste generation rate for cities in South Korea under redevelopment. They gathered data from hundreds of buildings before and after demolition. For the prediction, they employ as input similar information as Akanbi et al. (2020), on location, structure, usage, wall type, roof type, gross floor area, and number of floors. Such an algorithm can provide valuable information on the generated waste in case of demolition (or deconstruction), hence enabling to design regions that produce less waste and as such more sustainable.

3.2 Slowing the Loop

Achieving longevity of the built environment requires adopting appropriate measures for preventative maintenance (Çetin et al. 2021). In turn, a well-maintained product – be that an entire building, an element within, or a material – results in higher reusability potential in the future. Of particular interest for preventative maintenance is the understanding of a product’s condition, early detection of any defects, monitoring of their progression to ensure appropriate intervention, and predicting remaining useful life (Khuc and Catbas 2018; Srikanth and Arockiasamy 2020; Dong and Catbas 2021; Li et al. 2021; Berghout and Benbouzid 2022). Researchers employ CV algorithms to inspect infrastructure projects whose failure mode can have catastrophic consequences, such as bridges (Galdelli et al. 2022), tunnels (Tichý et al. 2021), and dams (Khaloo et al. 2018; Feng et al. 2020). A common pipeline consists of the following steps: (1) capturing video sequences and point cloud data with scanners or cameras, (2) processing this data to identify and analyse defects and localise the defects on the 3D model, and (3) comparing the data with the previous state of any defects. Not all steps are fully automated, and user participation is required to complete them. In the case of ML, critical data, such as year of construction, traffic load, and temperature range, are used to predict the remaining life of a product. To achieve this, historical data from inspections of similar products are extracted from – public – databases.

Although the ultimate goal of many such pipelines would be to reach full automation, it is unrealistic to expect that we can fully achieve this goal while guaranteeing robust algorithmic performance, especially in high-stakes settings (e.g. dam collapse). The inclusion of a user can bridge the gap between automation and accuracy requirements (Muin and Mosalam 2021). However, user actions should be integrated into helping the algorithm increase learning capacity over time with human-in-the-loop techniques (e.g. reinforcement learning-based). Such techniques involve human and machine collaboration, where input from the (human) user serves as a learning signal for the machine to correct or further expand previously learned states.

One might ask: why do we need both image and point cloud data? To answer this, we need to make the distinction between images and point clouds: images capture high-resolution information related to colour, edges, and texture, whereas point clouds capture geometry, shape, and general 3D composition. Hence, detecting defects that relate to cracks, discolorations, texture changes, etc., requires images, since point cloud data is not able to contain this visual information. On the other hand, detecting general structural failures, such as bending, dislocation, etc., requires geometric cues, which are contained in point clouds and are not easy to extract from images. With photogrammetry, one can create point clouds with a sequence of images (video), but the final point density is lower and geometric error is higher, which makes this technique less appropriate for this type of analysis. (For more details on photogrammetry, see Chap. 3 on scanning technologies by Gordon et al.)

3.3 Closing the Loop

With respect to closing the loop, and essentially reusing or recycling materials (Bocken et al. 2016; Çetin et al. 2021), CV can play an instrumental role in managing materials and waste by identifying useful recovery pathways at various scales during a project’s life cycle. On the micro level of recycling or reusing materials, CV algorithms can be used to reduce labour costs by automating the classification of recyclable materials (Mao et al. 2021; Zhang et al. 2021) at the end of a building’s life in sorting centres. The classification of scrap metals and estimation of their masses (Díaz-Romero et al. 2022) can also be carried out to obtain a better understanding of the physical properties of the objects being sorted. Multilabel waste detection models are additionally being explored to identify glass, fabric, metal, plastic, and paper waste (Zhang et al. 2022), as well as earth-based components such as concrete, clinker, and bricks (Kuritcyn et al. 2015). For effective construction and demolition waste management, inspectors at disposal facilities must determine the amount of different waste components loaded in incoming trucks. The large quantities of waste and their mixed nature make it difficult to access. Chen et al. (2022) developed an algorithm to automatically quantify specific materials from a single image obtained from monocular cameras installed on site, while Díaz-Romero et al. (2021) built a real-time deep learning system to separate cast and wrought-aluminium scrap on conveyor belts.

To evaluate the reusability of materials at the meso scale, tools are being investigated that employ ML on data acquired from networks of organisations and companies. At this level, efforts are made to optimise resource use in businesses and supply chains. This is achieved by designing products and processes that are modular and adaptable for a closed-loop system. An example is the predictive model in Rakhshanbabanari (2021) that was built to evaluate using ML the reusability of load-bearing building elements given data from a network of stakeholders. The results of this study indicate that the most important economic factor is cash flow implications associated with the need to purchase reused elements early in a project. This highlights the need for supply chain innovation to enable the timely sharing of resources and collaboration in reuse projects, which can in turn lead to significant resource and cost savings.

In recent years, the use of CV to recognise construction materials (such as concrete, brick, metal, wood, and stone) on existing buildings is being increasingly explored. This is of particular significance to creating a database of building materials in use on a global scale – information that is lacking for the present building stock. Material identification from imagery has been previously studied in constrained settings, where the imagery contains close-up pictures of the material (Dimitrov and Golparvar-Fard 2014), because curating datasets with this type of information in the context of a scene, a building, or a neighbourhood is difficult. As a result, CV algorithms trained on such data fail to recognise materials under real-world conditions and scenarios.

To overcome this limitation, Sun and Gu (2022) created a dataset of construction material found on building facades. The images contain the full building structure (facades), but only the prominent building material label is given (i.e. the building material which occupies the most pixels). This simplification can confuse the algorithm when processing data from regions in the image that correspond to other materials. Raghu et al. (2022) generated an image dataset of materials where street-view imagery of urban settings is annotated with material labels (for more details, see Sect. 4.4). In this case, since a building’s facade is only partially captured within one image, the strategy of the one prominent label provides a more accurate description of the image content. This is due to the nature of the data; there is usually not enough front-facing line of sight from the bottom to the top of a building in normal city streets.

Most existing CV algorithms developed for the circular analysis of material stocks focus on identifying the material in a certain image but still lack finer granularity with respect to the location, geometry, and boundaries of the materials both in the images and 3D space (semantic segmentation of or object detection in images and/or 3D data) (Dimitrov and Golparvar-Fard 2014; DeGol et al. 2016; Raghu et al. 2022). This is highly relevant information to create in-use material stocks since it can define quantities and usability. Such information can be further used for design decisions on reuse in new constructions and would allow the development of ML-based approaches that can assist or automatically identify these decisions.

There is vast literature in CV on semantic segmentation and object detection algorithms that address the problem of what objects exist in the data on a finer granularity. Although the underlying algorithms would be the same when addressing the problem of what materials exist in the data, the lack of comprehensive datasets that are tailored for CE tasks does not allow the algorithms’ easy use. An example is the OpenSurfaces dataset (Bell et al. 2013), commonly used in CV for material segmentation. It contains images from occupied real-world indoor environments (i.e. they contain furniture and other objects) along with pixel-wise material annotations for most contained objects. However, these annotations do not include building and structural elements that are important in a circularity setting, and as such they cannot be leveraged in a CE setting.

3.4 Regenerating the Loop

Regeneration is about leaving the environment in a better state than before (Bocken et al. 2016; Çetin et al. 2021). Zhang et al. (2022) developed a CV algorithm that operates on the macro level and detects construction and demolition waste in remotely sensed images in China. Specifically, the algorithm localises abandoned soil and other waste piled in open environments and targets a more efficient and prompt waste management. Konietzko et al. (2020) introduced the ‘regenerate’ strategy to place the focus on minimising the use of toxic substances and renewable materials. Studies with respect to such regeneration strategies on the macro level include calculating the maximum volume of long-term storage depots to optimise the flow of mineral resources (Globa et al. 2021) and predicting the presence of hazardous materials in urban building stocks (Wu et al. 2021, 2022). Such inventories can reduce the risk of unexpected costs and delays during the demolition process but, most importantly, enable the well-being of users and their surroundings. The removal of hazardous materials and their replacement with natural, non-harming, potentially reused ones enable a safer and regenerated environment for future generations. Çetin et al. (2022) report an AI-based system operating on the micro and meso levels with the capability to detect toxic or hazardous contents on building facades. However, it is important to carefully perform the disposal of hazardous materials; otherwise, the regeneration loop cannot be achieved. As discussed in Chap. 15, algorithms can help AECO stakeholders make informed decisions for regenerative design and architecture.

A summary of the above is offered in Table 4.1. Specifically, it contains examples of CV and ML use per resource loop strategy explored in this section.

Table 4.1 Examples of prominent tasks that CV and ML methods focus on with respect to CE. They are categorised per their contribution to each main resource loop strategy
Full size table

4 An Example of Using Computer Vision for Mining Materials in Urban Settings

In this section, we describe a research effort of using CV for a CE and particularly of creating a material inventory of urban environments using street-view imagery towards urban mining. Urban mining is the process of recovering and reusing a city’s materials when buildings, infrastructure, or products become obsolete. For more information on building demolition valuation and decision-making, we point the reader to Chap. 11 on digital deconstruction by van den Berg. This concept treats the entire city as a storage depot of prospective materials that can be recovered. Information on which materials are available and where they are located in the city is key to forecasting their end-of-life destinations. In this context, the discourse on buildings as ‘material banks’ is emerging, where high-value materials are retained for future use instead of discarding them as waste (BAMB 2022). To achieve a CE, it is necessary to adopt long-term and innovative thinking, considering that up to 25% of material in a traditional residential structure can be easily reused and up to 70% of material can be recycled (Bohne and Wærner 2014).

The construction sector still faces significant bottlenecks in scaling up urban mining. Given the lacking as-is documentation of the existing building stock, efficient reverse-engineering tools are required to appropriately quantify waste in the built environment. The increase of open access repositories documenting the world around us contains untapped potential. Using a street-view image API (i.e. application programming interface), ocular observations were conducted to analyse building-specific characteristics that can enable reuse using CV by Raghu et al. (2022). Images of buildings marked for demolition were retrieved, following which material and component detection algorithms were implemented to observe the various facade materials and window counts in two European cities (Barcelona and Zurich). The algorithm was tested on these two cities to understand the replicability of the model despite variation in architectural building styles.

Such forecasting techniques help create a material inventory that designers can check to verify the forthcoming availability of components and assess their suitability for reuse in new projects. In contrast to other data gathering techniques that are carried out at the building end of life, which require complex and sophisticated software (Uotila et al. 2021), the methodology developed in this project can help in expeditiously procuring information about reusable components earlier on in the project life cycle. Figure 4.2 illustrates the potential use of CV, specifically semantic segmentation of materials and components on building facades. This allows for automated recognition and information retrieval of availability of materials for reuse prior to building demolition. The planning of new construction projects with a circular agenda requires such information about reusable materials, along certainty about their quality and quantity, that is provided to designers or architects well in advance.

Fig. 4.2
An illustration of a project derived using C V. It marks residential building types with location and materials such as 12 windows, 2 doors, and 34 facade panels.

Visualisation of database with insights on materials and components in supply project (or donor project) derived using CV that can be used for new construction projects (image from Raghu, D., adapted from De Wolf et al., 2023)

Full size image

5 Business Models for Computer Vision and Machine Learning in a Circular Built Environment

Services related to CV and ML in CE are the identification and characterisation of building elements; their materials, types, quantities, and conditions; ways to repair them; patterns for reuse; and more. These can be tailored to the interiors and exteriors of buildings, and one can consider whether these can be provided as separate services. An example is that of Spotr (Spotr.ai 2022), a real estate inspection company based in the Netherlands, which has built a tool to conduct digital building surveys, thus enabling stakeholders to make well-founded decisions on property maintenance. As such, Spotr supports slowing the loop by targeting early identification of damages, hence extending the life of products through timely repairs. This information is identified on images collected from various sources, including human-captured images, drones, and satellite imagery. Using CV, defects are assigned to an exact location which helps easily plan follow-up actions. Spotr then enables large-scale analyses by projecting the results on a map to gain insights into specific areas. Primary information about the buildings is brought together, including images, information on their constituent elements, and present condition.

As more and more services are being developed, other aspects need consideration. While CV and ML are being marketed as a universal solution for many problems faced by the industry, the reality is that CV and ML readiness and adoption in organisations is slow and uncertain. Stakeholders that can benefit financially from these technologies in the frame of CE include real estate agents, asset managers, building owners, and construction companies. Although large organisations tend to prefer in-house solutions so that they may maintain a high degree of integration with and customisation to their existing processes, several considerations should be taken into account before deciding on an in-house or external solution.

First, these technologies require specialised personnel that will be able to develop the algorithms and systems from both research and an engineering perspective. Second, there is a requirement for certain infrastructure, such as computation and annotation services (when lacking access to curated datasets) or patent licencing (for CV and ML methods, systems, and hardware for data collection) if building an internal database and more. Third, apart from the inherent cost, it is important to make the following distinction with respect to in-house and outside solutions: the former provides more control, but the latter allows users access to better-performing algorithms. Dedicated providers of such services have access to more and diverse data that, in turn, can create better generalisable and more robust algorithms. Last, one should also consider the benefits and downfalls of performing data collection in addition to data processing: this allows companies to keep the data and build databases and datasets to train better algorithms, expand services, and sell or share the data. However, there is an associated cost with hardware and personnel for data collection, especially considering how fast hardware evolves and the bottleneck it can create due to the maximum amount of devices one can acquire with respect to cost and return on investment.

6 Discussion

Throughout the chapter, we highlighted limitations related to AI approaches and technologies. CV and ML can be considered the eyes and brain of machines, respectively, and can play a fundamental role in mapping and understanding what we have at hand now for reuse, maintenance, or demolition, as well as in identifying ways to design, fabricate, and construct for a CE. In our opinion, there are three fundamental barriers to creating such methods, systems, and tools:

Breaking Down Silos

The AECO industry is known for operating in a very isolated manner both in terms of projects and involved stakeholders. However, here we would like to point out shortcomings regarding data and processes. CV and ML technology, especially in a holistic approach such as CE, cannot be developed in isolation without a comprehensive understanding of how to connect the dots between different sub-problems. Efforts that take place in isolation result in duplicating data, processing, and information. Essentially, we should consider how we can apply circularity not only in buildings but also in the technology we are creating (i.e. recycle data, share data across processes, and more).

In addition, a non-targeted and convulsive way of trying existing AI methods – initially developed for other domains – for circularity will not help solve these problems. Practitioners and researchers would benefit from going beyond trying to see what existing AI approaches can do for circularity and instead first consider what should be achieved for the transition to a CE followed by how AI can help towards this goal. Only then should they consider what existing AI methods can solve now and what is still needed to be developed.

Certainly, using algorithms to better understand the micro, meso, and macro levels of the world around us is still an open question in the field of CV and ML, one that will take time to answer. However, advances in CE should not take place after technological developments in the field of AI but instead should proceed hand in hand, given the importance of this domain in sustainability and future generations.

Regulations and Standards

What happens when an algorithm provides a wrong prediction? What safety measures can we establish – both in terms of technology and organisation – to minimise or address such cases? What are the evaluation procedures, metrics, and thresholds that we will use to assess CV and ML algorithms for CE and that go beyond technical accuracy? How can we ensure privacy and security of data storage, processing, and sharing to allow operation on different scales and achieve CE in the built environment? What information should be in global public databases and what in private ones? What should be funded and how should it be regulated by the government and what is in the hands of private companies? How do we control and ensure the privacy of information in both public and private settings? These are important questions to answer going forward since they can drastically alter open access and trust of information on a problem that concerns the entirety of humanity.

Ethical Considerations

Using algorithms to create and sustain the built environment can offer benefits as long as we have ethical, legal, and environmental processes in place to keep the use of such technology in check. CV and ML are not only here to automate what humans can already do; they can enable us to explore aspects of the data that humans cannot associate or imagine ourselves. In certain cases, these technologies will help achieve superhuman understanding of the world around us. However, it is not only in the hands of CV and ML scientists to achieve this. Expertise from researchers that address different facets of realising CE needs to come together and collaborate with the CV and ML scientists.

What are fundamental steps going forward? In our opinion, it is imperative to formalise the minimum and maximum viable information to achieve a circular built environment. With this information defined, decisions can be made by CE and AI experts on which technologies can play a role and what developments are required to create truly robust systems that operate in real-world settings.

7 Key Takeaways

  • Machine learning (ML) and computer vision (CV) can play a pivotal role in circularity, by providing stakeholders with actionable information on the state of the building stock.

  • A variety of CV and ML methods can be helpful in each of the four strategies of narrowing, slowing, closing, and regenerating the loop.

  • Described methods focus on understanding the present state of the building stock. The next step is to develop methods that go beyond this and can propose best actions to take, even going as far as implementing actions directly.

  • Challenges in creating global and robust circular CV and ML circular systems for the built environment require multidisciplinary collaboration between experts in artificial intelligence, circular economy, and the architecture, engineering, construction, and operations (AECO) industry.