Abstract
With the fast-growing graphical user interface (GUI) development workload in the Internet industry, some work attempted to generate maintainable front-end code from GUI screenshots. It can be more suitable for using user interface (UI) design drafts that contain UI metadata. However, fragmented layers inevitably appear in the UI design drafts, which greatly reduces the quality of the generated code. None of the existing automated GUI techniques detects and merges the fragmented layers to improve the accessibility of generated code. In this paper, we propose UI layers merger (UILM), a vision-based method that can automatically detect and merge fragmented layers into UI components. Our UILM contains the merging area detector (MAD) and a layer merging algorithm. The MAD incorporates the boundary prior knowledge to accurately detect the boundaries of UI components. Then, the layer merging algorithm can search for the associated layers within the components’ boundaries and merge them into a whole. We present a dynamic data augmentation approach to boost the performance of MAD. We also construct a large-scale UI dataset for training the MAD and testing the performance of UILM. Experimental results show that the proposed method outperforms the best baseline regarding merging area detection and achieves decent layer merging accuracy. A user study on a real application also confirms the effectiveness of our UILM.
摘要
随着互联网行业图形用户界面(GUI)开发工作量的快速增长, 国内外学者试图从GUI图片生成可维护的前端代码。该任务更适合从包含有用户界面元数据的UI设计稿来实现代码生成。然而, 碎片图层不可避免地会出现在UI设计稿中, 这大大降低了生成代码的质量。目前尚未有自动化GUI技术来检测并合并碎片图层以提高生成代码的可维护性。本文提出UI图层合并器(UILM), 一种基于计算机视觉的方法, 可以自动检测并合并碎片图层为完整的UI组件。本文提出的UILM包含合并区域检测器和碎片图层合并算法。合并区域检测器结合边界先验知识来准确地检测出UI组件的边界。基于此, 碎片图层合并算法可以搜索出边界内的相关碎片层并将其合并为整体。此外, 我们提出一种动态数据增强算法来提高合并区域检测器的性能。我们还构建了用于训练合并区域检测器和测试UILM的大规模UI数据集。实验结果表明, 本文提出的方法在合并区域检测方面优于最佳基线, 并达到了有效的图层合并准确率。最后, 对真实应用程序的用户研究也证实了本文方法的有效性。
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
Data availability
The data that support the findings of this study are openly available in Github at https://github.com/zju-d3/UILM.
References
Aşıroğlu B, Mete BR, Yıldız E, et al., 2019. Automatic HTML code generation from mock-up images using machine learning techniques. Scientific Meeting on Electrical-Electronics & Biomedical Engineering and Computer Science, p.1–4. https://doi.org/10.1109/EBBT.2019.8741736
Behrang F, Reiss SP, Orso A, 2018. GUIFetch: supporting app design and development through GUI search. Proc 5th Int Conf on Mobile Software Engineering and Systems, p.236–246. https://doi.org/10.1145/3197231.3197244
Beltramelli T, 2018. pix2code: generating code from a graphical user interface screenshot. ACM SIGCHI Symp on Engineering Interactive Computing Systems, Article 3. https://doi.org/10.1145/3220134.3220135
Bunian S, Li K, Jemmali C, et al., 2021. VINS: visual search for mobile user interface design. CHI Conf on Human Factors in Computing Systems, Article 423. https://doi.org/10.1145/3411764.3445762
Cai ZW, Vasconcelos N, 2018. Cascade R-CNN: delving into high quality object detection. IEEE/CVF Conf on Computer Vision and Pattern Recognition, p.6154–6162. https://doi.org/10.1109/CVPR.2018.00644
Chen CY, Su T, Meng GZ, et al., 2018. From UI design image to GUI skeleton: a neural machine translator to bootstrap mobile GUI implementation. Proc 40th Int Conf on Software Engineering, p.665–676. https://doi.org/10.1145/3180155.3180240
Chen CY, Feng SD, Xing ZC, et al., 2019. Gallery D.C.: design search and knowledge discovery through auto-created GUI component gallery. Proc ACM on Human-Computer Interaction, Article 180. https://doi.org/10.1145/3359282
Chen JS, Xie ML, Xing ZC, et al., 2020. Object detection for graphical user interface: old fashioned or deep learning or a combination? Proc 28th ACM Joint Meeting on European Software Engineering Conf and Symp on the Foundations of Software Engineering, p.1202–1214. https://doi.org/10.1145/3368089.3409691
Chen K, Wang JQ, Pang JM, et al., 2019. MMDetection: open MMLab detection toolbox and benchmark. https://arxiv.org/abs/1906.07155v1
Chen S, Fan LL, Su T, et al., 2019. Automated cross-platform GUI code generation for mobile apps. Proc IEEE 1st Int Workshop on Artificial Intelligence for Mobile, p.13–16. https://doi.org/10.1109/AI4Mobile.2019.8672718
Deka B, Huang ZF, Franzen C, et al., 2017. Rico: a mobile app dataset for building data-driven design applications. Proc 30th Annual ACM Symp on User Interface Software and Technology, p.845–854. https://doi.org/10.1145/3126594.3126651
Fay MP, Proschan MA, 2010. Wilcoxon-Mann-Whitney or t-test? On assumptions for hypothesis tests and multiple interpretations of decision rules. Stat Surv, 4:1–39. https://doi.org/10.1214/09-SS051
Feng SD, Ma SY, Yu JZ, et al., 2021. Auto-icon: an automated code generation tool for icon designs assisting in UI development. Proc 26th Int Conf on Intelligent User Interfaces, p.59–69. https://doi.org/10.1145/3397481.3450671
Ge XF, 2019. Android GUI search using hand-drawn sketches. Proc IEEE/ACM 41st Int Conf on Software Engineering: Companion Proc, p.141–143. https://doi.org/10.1109/ICSE-Companion.2019.00060
Halbe A, Joshi AR, 2015. A novel approach to HTML page creation using neural network. Proc Comput Sci, 45:197–204. https://doi.org/10.1016/j.procs.2015.03.122
He KM, Zhang XY, Ren SQ, et al., 2016. Deep residual learning for image recognition. IEEE Conf on Computer Vision and Pattern Recognition, p.770–778. https://doi.org/10.1109/CVPR.2016.90
Jain V, Agrawal P, Banga S, et al., 2019. Sketch2Code: transformation of sketches to UI in real-time using deep neural network. https://arxiv.org/abs/1910.08930
Li G, Baechler G, Tragut M, et al., 2022. Learning to denoise raw mobile UI layouts for improving datasets at scale. CHI Conf on Human Factors in Computing Systems, Article 67. https://doi.org/10.1145/3491102.3502042
Lin TY, Maire M, Belongie S, et al., 2014. Microsoft COCO: common objects in context. Proc 13th European Conf on Computer Vision, p.740–755. https://doi.org/10.1007/978-3-319-10602-1_48
Lin TY, Dollár P, Girshick R, et al., 2017. Feature pyramid networks for object detection. IEEE Conf on Computer Vision and Pattern Recognition, p.936–944. https://doi.org/10.1109/CVPR.2017.106
Liu Z, Chen CY, Wang JJ, et al., 2020. Owl eyes: spotting UI display issues via visual understanding. Proc 35th IEEE/ACM Int Conf on Automated Software Engineering, p.398–409. https://doi.org/10.1145/3324884.3416547
Liu Z, Chen CY, Wang JJ, et al., 2023. Nighthawk: fully automated localizing UI display issues via visual understanding. IEEE Trans Soft Eng, 49(1):403–418. https://doi.org/10.1109/TSE.2022.3150876
Moran K, Bernal-Cárdenas C, Curcio M, et al., 2020. Machine learning-based prototyping of graphical user interfaces for mobile apps. IEEE Trans Soft Eng, 46(2):196–221. https://doi.org/10.1109/TSE.2018.2844788
Nguyen TA, Csallner C, 2015. Reverse engineering mobile application user interfaces with REMAUI. Proc 30th IEEE/ACM Int Conf on Automated Software Engineering, p.248–259. https://doi.org/10.1109/ASE.2015.32
Ren SQ, He KM, Girshick RB, et al., 2017. Faster R-CNN: towards real-time object detection with region proposal networks. IEEE Trans Patt Anal Mach Intell, 39(6):1137–1149. https://doi.org/10.1109/TPAMI.2016.2577031
Suleri S, Pandian VPS, Shishkovets S, et al., 2019. Eve: a sketch-based software prototyping workbench. CHI Conf on Human Factors in Computing Systems, Article LBW1410. https://doi.org/10.1145/3290607.3312994
Vu T, Jang H, Pham TX, et al., 2019. Cascade RPN: delving into high-quality region proposal network with adaptive convolution. Proc 33rd Conf on Neural Information Processing Systems, p.1430–1440.
White TD, Fraser G, Brown GJ, 2019. Improving random GUI testing with image-based widget detection. Proc 28th ACM SIGSOFT Int Symp on Software Testing and Analysis, p.307–317. https://doi.org/10.1145/3293882.3330551
Xu N, Price B, Cohen S, et al., 2017. Deep image matting. IEEE Conf on Computer Vision and Pattern Recognition, p.311–320. https://doi.org/10.1109/CVPR.2017.41
Zhang XY, de Greef L, Swearngin A, et al., 2021. Screen recognition: creating accessibility metadata for mobile applications from pixels. CHI Conf on Human Factors in Computing Systems, Article 275. https://doi.org/10.1145/3411764.3445186
Zhao TM, Chen CY, Liu YM, et al., 2021. GUIGAN: learning to generate GUI designs using generative adversarial networks. Proc 43rd IEEE/ACM Int Conf on Software Engineering, p.748–760. https://doi.org/10.1109/ICSE43902.2021.00074
Author information
Authors and Affiliations
Contributions
Yunnong CHEN, Liuqing CHEN, and Yankun ZHEN designed the research. Yunnong CHEN and Chuning SHI processed the data. Yunnong CHEN and Liuqing CHEN drafted the paper. Chuning SHI, Jiazhi LI, and Zejian LI helped organize the paper. Tingting ZHOU, Yanfang CHANG, and Lingyun SUN revised and finalized the paper.
Corresponding author
Additional information
Compliance with ethics guidelines
Yunnong CHEN, Yankun ZHEN, Chuning SHI, Jiazhi LI, Liuqing CHEN, Zejian LI, Lingyun SUN, Tingting ZHOU, and Yanfang CHANG declare that they have no conflict of interest.
Project supported by the National Key R&D Program of China (No. 2018AAA0100703), the National Natural Science Foundation of China (Nos. 62006208 and 62107035), the Ng Teng Fong Charitable Foundation in the form of ZJU-SUTD IDEA Grant, and the Alibaba-Zhejiang University Joint Research Institute of Frontier Technologies
Rights and permissions
About this article
Cite this article
Chen, Y., Zhen, Y., Shi, C. et al. UI layers merger: merging UI layers via visual learning and boundary prior. Front Inform Technol Electron Eng 24, 373–387 (2023). https://doi.org/10.1631/FITEE.2200099
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1631/FITEE.2200099