Keywords

See (Fig. 1).

Fig. 1.
figure 1

“Quantum Clouds” by the author, through collaging several NST along with post processing inspired by Archigram.Footnote

Cook [2].

See Fig. 5 for more detail about the NST process for this image

Full size image

1 Introduction

This work outlines an approach to using neural style transfer (NST) in a novel way as a part of the early architectural design process. We seek to use the tool in a way unintended by its creators, for results different from many examples or use cases.Footnote 2, Footnote 3, Footnote 4 Additionally we seek to composite the results of multiple NST iterations to create a composition that is more directed by the author (Figs. 3 and 5). The hypothesis of utilizing this procedure is that the results become hybrid between the tool and the designer. A feedback loop: both providing input and responding to the other.

The work presented herein attempts to blur the lines between architectural design, storytelling, digital media, computer science and other disciplines. The objective of the effort is to inspire humanity toward a brighter future through imagining and instantiating possible futures. Divergent from many other documented use cases within the design process, this use case of NST focuses on a more intuitive, iterative, feedback loop between designer and the computational process (Fig. 3). We attempt to achieve a guided vision, possibly beyond the potential conceptions of the designer in unaided circumstances to conceive of the outcomes, or potential design schemes. Built on several speculative architectural workshops, art exhibitions, and design studios at Tongji DigitalFUTURES, FutureLab Shanghai, Longxi Art Museum, and Wenzhou-Kean University, respectively, this paper seeks ontological uncertainty between the artist and a cognitive prosthetic using the machine learning technologies of a convolutional neural network (CNN), and more specifically NST in the early design phase for novel design solutions.

The collection of NST compositions represented in this article are primarily possible futures. One might ask about the merit of compositions that do not seem to be feasible or viable as constructed architecture. Yet these blue-sky explorations can exist only digitally, or possibly become a part of humanity’s-built reality at some point in the future. Regardless of their final instantiation, by very conception, these possibilities become exactly that, a possibility, otherwise impossible. This paper will first outline our process. Afterward consider some of the lineage from which this work springs and the potential power of the speculative world building process.Footnote 5 Next we will consider the low hanging fruit of instantiating these places at least virtually in the metaverse and/or through VR, animation to solidify their somewhat timid present reality (Fig. 2).

Fig. 2.
figure 2

This diagram outlines the artistic process, of “Plant First City” by Shannon Xiaotong Shi 施晓彤. A straightforward subversion of the NST tool by combining two “non-style” images to conceive of and create a novel urban environment

Full size image

2 Novel Methods Toward Future Possibilities

This paper reflects on a collection of works by the author and collaborators that seek to subvert the intended use case of NST. The NST process is built on VGG16 and subsequently VGG19 conceived by Oxford's Visual Geometry Group (VGG, 19 denoting the number of layers in the network).Footnote 6, Footnote 7 The technology is designed to apply the style from, for example, a famous painting such as “Starry Night” by Van Gough and apply it to a base image such as a photo taken with your phone.

Our unconventional approach to the process involves intuition on the part of the designer as a part of the style transfer and compositing process. We outline a use case for NST in the early design phase of an architectural project where the designer goes back and forth between the NST process and other inputs toward optimized results which meets the subjective, intuitive exploratory criteria of the designer. Our process encourages mis-use of the NST process by subverting the labeling mechanisms inherent to the NST process; encouraging a style image to be used as a base image and what may be considered a base image as a style, in other cases we forgo the style and instead opt to only feed the system base images; starving the system for a style. We encourage the compositing of multiple more traditional NST images into a single composition. After several NST exercises, the design can then respond to the collected results through collaging to create outcomes that otherwise would prove impractical (Fig. 2).

Several NST use cases within the framework have resulted in the output weighing significantly on final outcomes of the architectural design project. The aesthetic of an NST often approaches something akin to computational impressionism.Footnote 8 This style inherently lends itself to the potential for an emotional response from a viewer. Paired with unexpected, uncanny, or emotionally vivid elements the compositions often approach the sublime.Footnote 9 These results can then be taken by the designer as a prompt, for further speculation and development based on a set of constraints. Our process maintains that NST output commonly provides a small peek into a potential world. It is up to the designer to take the prompt further and develop what they see in the image. After the designers develop their interpretations and speculations as a more flushed out image; often literally expanding the frame and providing elements of scale and context, a more traditional modeling process can follow, with the outcomes moving from not possible to conceived and on to a navigable experience, and possibly built. Although, there are several developing technologies with increasing capabilities able to translate single images into 3D digital models. Exploring these tools will be at the heart of future work in this research (Figs. 3 and 4).

Fig. 3.
figure 3

Comparing a typical NST process (top) to the hybrid approach outlined in this paper (bottom)

Full size image
Fig. 4.
figure 4

Student processes example from image selection through to virtual environment

Full size image

Three of the architectural design studios that have used this process have ended with results which have been directly inspired by the early visioning produced with the designer and the NST process. In the spring of 2022, the author ran an architectural design studio with co-instructor David Vardy, where NST was a core aspect to the early design phase. Later in the semester the results were curated in the “Computational Impressionism 无极之地” exhibition at the Longxi Art Museum in Zeya, Zhejiang Province (Fig. 5).

Fig. 5.
figure 5

On the left is a photograph of the style transfer work in the Computational Impressionism 无极之地 exhibition at Longxi Art Museum. On the right is the collection of process work used to derive the “Quantum Clouds” composition

Full size image

The students began with site research. Collecting material samples, natural artifacts, photographs, etc. They also began processing the programmatic requirements, and in turn begin thinking about precedent projects. All these inputs inform the subject matter input into the NST process. Yet with the distinct interest in subverting the style transfer in favor of a mixing of ideas, the outcomes demand new subjective interpretations of the original object, not allowing the observer (including designers) to be passive about their viewing experience of the NST outcomes. The outcomes of this superficial process generally lack scale and context. The designer is called to engage with the art. The original subject is often no longer readily recognizable. Instead, the designer must layer their own psychological imprint on the piece to interpret it. It is the opportunity of the designer to define both. As mentioned earlier this happens through an expanding of the frame to define elements such as horizon, orientation, scale, and provide some reference indicators. The results of all these efforts are both inspiring and important in the development of hybrid intelligence within creative disciplines.

3 Imagination: Pioneering Effort for Humanity

Much of what drives design are performative aspects of the outcomes, yet if we are only driven by the utilitarian requirements, we will never allow ourselves the opportunity to imagine alternatives. By pushing beyond what is conceivable we enable new possibilities for the future.Footnote 10 This can be referred to as the development of possible futures. It is a significant part of guiding the future. Hybrid intelligent processes such as style transfer can play a key role in this and other problems humanity faces.

This research project builds on the heritage of the digital studios pioneered by Bernard Tschumi, and before him the work of Syd Mead (1933–2019), Lebbeus Woods (1940–2012), Roger Dean (1944), and others. We see the legacy of the Columbia University digital studio effort going on to create such projects as the digital Guggenheim, New City, and has even influenced Hollywood world building see examples from Alex McDowell and the World Building Media LabFootnote 11 at the University of Southern California. Yet this is not the only lineage of computation influencing the future of urban conceptions. We have seen, particularly in the last ten years, the influence of computational methods such as parametric, procedural, and machine learning rework the possibilities of creativity to incorporate computation as a collaborator (Fig. 5).

Beyond those technical aspects driving creativity and innovation, there are other stirrings including more cultural ones such as the post-digital movement which has worked to flatten and colorize predominant conceptions of our future. One could argue that the digital screen is the ultimate flattening, and yet the digital screen offers so much in terms of dimensionality. The imagery produced together with the artist and the computer using NST arguably creates a new aesthetic, a new style entirely, obviously grounded in a strong foundation briefly explored above. Yet this aesthetic quite possibly may have been previously out of reach without computational collaboration.

The work that this group has undertaken is inherently optimistic and is rooted in questions such as: What kind of a world would you like to exist in if some limitation was removed? What possible futures could embody more sustainable circumstances for humanity? What is the ideal community for an individual? For industry? For community? What elements are needed to cultivate the most conducive environments? What methods of digital representation are optimal for expressing these ideas? How can one question traditional typologies and outdated organizational systems for the benefit of humanity? How do diverse family structures, demographics, and living/working styles affect the city?

Consider the work of Shannon Xiaotong Shi 施晓彤 with her work highlighted in Fig. 2 entitled Plant First City. In this work she considers the absurd possibility of a city where plants receive the primary and first consideration. What would such a reality feel like? In her extra-planetary conception, plants are the dominant species and in order to survive humans must live symbiotically. Even the conception of such a reality is quite intensive, let along the instantiation of it as a navigable environment, which is exactly what she was able to accomplish with her partner Luna Yuwen Wang 王宇文.Footnote 12 Of course this fabricated virtual environment is not built, but regardless this may be the fiction that inspires the next generation to do something truly historically altering, in the same way that the pacemaker was inspired by Frankenstein.Footnote 13 Regardless, this effort by the artist has impacted if no one else, her future. Hopefully she will reflect on this part of her architectural training as she is considering her career path. Content creation is a growing sector with promise. Regardless, her ability to conceive of realities outside of conventional human conception has been enhanced, and her confidence entertaining computational collaborations has grown.

4 Virtual Habitation

The technologies necessary to create complex digital worlds using automated computational methods seems to still be an elusive matter. NST and related technologies by themselves fall short of being a creative force in 3D world building alone.Footnote 14 Yet NST can be a part of a collaborative process with designers to directly inform and participate in “world building”Footnote 15 minimally for virtual spaces. As virtual spaces are not limited by the constraints outside of computational environments, the NST results can then be taken by the designer as a prompt, for further speculation and development based on a set of constraints.

NST seems to allow designers/architects to envision realities outside of the constraints of currently understood physics. This allows those who experience the place to engage in potentially impossible phenomena under any other circumstances. For example, “Parkour in Mars City”, an Unreal Engine simulation by Alex Guo Yupeng 郭玉鹏 and Ted Kaiyuan Feng 冯开元, based upon a series of NST compositions.Footnote 16 again deals with an extra-planetary parkour experience designed around the divergence in gravitational properties between our own Earth and what might be; elsewhere. How does our design of space change based on this parameter? This example lets you virtually realize those differences. These landscapes become a “digital shadow”Footnote 17 of a physical landscape that has not yet been realized. These shadows have a life of their own outside of any physical reality. The ambition is that through these virtual prototypes we can iterate faster through possibilities toward a better future.

5 Conclusion

We have explored some pedagogical case studies, strengthening the argument that this process can yield compelling results that inspire and demand questioning, and quite possibly yield unconventional approaches to real world problems. NST has many use cases, but the most interesting may be the misuse of the tool by not simply taking the transfer at face value, but by subverting the labeling methods employed in the training set of the model to create unexpected results that take NST beyond just transferring a style toward honest unique creation. With this use case it is not difficult for the imagery to help in the process of speculating about future possibilities. Synthetic intelligenceFootnote 18 allow us to move past human conceptions considering realities we may not deem feasible. Human imagination is always a precursor to what becomes reality. Thinking beyond our current conceptions of reality is essential to move past barriers that currently hold humanity. Non-probable futures allow humanity to identify aspects worth striving for. Even if a world building exercise is not probable for instantiation within the built environment, people can inhabit these environments virtually. What future do you want? Maybe working with NST can help us to create it.