applied

sciences
Article
Method for Constructing a Façade Dataset through Deep
Learning-Based Automatic Image Labeling
Hyeongmo Gu and Seungyeon Choo *

School of Architecture, Kyungpook National University, 80 Daehak-ro, Buk-gu, Daegu 41566, Korea;
[email protected]
* Correspondence: [email protected]; Tel.: +82-53-950-5593

Abstract: The construction industry has made great strides in recent decades by utilizing computer
programs, including computer aided design programs. However, compared to the manufacturing
sector, labor productivity is low because of the high proportion of knowledge-based tasks and simple
repetitive tasks. Therefore, knowledge-based task efficiency should be improved through the visual
recognition of information by computers. A computer requires a large amount of training data, such
as the ImageNet project, to recognize visual information. This paper proposes façade datasets that
are efficiently constructed by quickly collecting façade data through road-view images generated
from web portals and automatically labeled using deep learning as part of the construction of image
datasets for visual recognition construction by a computer. Therefore, we attempted to automatically
label façade images to quickly generate large-scale façade datasets with much less effort than the
existing research methods. Simultaneously, we constructed datasets for a part of Dongseong-ro,
Daegu Metropolitan City, and analyzed their utility and reliability. It was confirmed that the computer
could extract significant façade information from the road-view images by recognizing the visual
information of the façade image. In addition, we verified the characteristics of the building con-
struction image datasets. This study suggests the possibility of securing quantitative and qualitative
Citation: Gu, H.; Choo, S. Method for
façade design knowledge by extracting façade design information from façades anywhere in the
Constructing a Façade Dataset world. Previous studies mainly collected façade images through camera photography to construct
through Deep Learning-Based databases, but in this study, a significant part of the database construction process was shortened
Automatic Image Labeling. Appl. Sci. through automation. In the case of façade automatic image labeling studies, it is the façade-based
2022, 12, 7570. https://doi.org/ automatic 3D modeling which has been primarily studied, but it is difficult to find a study to extract
10.3390/app12157570 data for façade design research.
Academic Editors: Igal M. Shohet
and José Salvador Sánchez Garreta
Keywords: façade; exterior building information; deep learning; image processing; image identification;
image extraction
Received: 1 May 2022
Accepted: 21 July 2022
Published: 27 July 2022

Publisher’s Note: MDPI stays neutral 1. Introduction

with regard to jurisdictional claims in The current construction industry has made great progress through the accumulation
published maps and institutional affil- of various computer programs and construction data, including computer aided design
iations.
(CAD) [1]. However, compared with the manufacturing sector, labor productivity is low,
except in the construction industry [1,2]. Artificial intelligence (AI) must be used to improve
areas of architecture that are considered human aspects by intelligent computer systems to
increase productivity. However, the level of AI research and technology development in
Copyright: © 2022 by the authors.
Licensee MDPI, Basel, Switzerland.
the field of architectural design is only at the level of research for optimizing and stream-
This article is an open access article
lining tasks in the design-construction phase or automation of some simple architectural
distributed under the terms and
designs. The development of artificial related technologies based on design uniqueness is
conditions of the Creative Commons insufficient [3].
Attribution (CC BY) license (https:// In this study, we determined that large scale façade datasets can be constructed
creativecommons.org/licenses/by/ with considerably less effort than any other methods by applying AI. One of the most
4.0/). important causes of the low level of research AI in the field of architecture is the lack of

Appl. Sci. 2022, 12, 7570. https://doi.org/10.3390/app12157570 https://www.mdpi.com/journal/applsci

Appl. Sci. 2022, 12, 7570 2 of 17

quantified constructed datasets, which are essential components of artificial neural network
learning [4]. Currently, the data in the field of architectural design are mixed in various
data formats such as drawings, images, and texts. Most of the available design information
accumulated in the existing field of architectural planning is qualitative and depends on
the expert knowledge of construction workers [5]. Therefore, in order to use construction
datasets for AI research, it is essential to quantify the design knowledge and express it in a
form that can be recognized by a computer. Advanced research on AI-based construction
technology requires the quantitative construction of architectural construction datasets as
the initial step.
Building information modeling (BIM) technology, which is an architectural informa-
tion management platform, is an important vehicle for connecting distributed construction
data to digital models. BIM building information can be managed in a form that can be
recognized by a computer. However, BIM models are limited in their utilization to the
BIM process [6]. However, unlike special processes such as BIM, there is a way to build
and utilize large amounts of data without investing a lot of time and capital through ma-
chine learning and by using public data released by governments in various countries [7].
An example of a machine-learning-based technology is Urbanbase, which automatically
converts 2D drawings to 3D models [8]. In this study, we determined that large scale
façade datasets can be constructed with considerably less effort than any other methods
by applying deep learning technology and utilizing non-quantified street façade images
existing in road-view images generated from web portals such as Google Street View [9]
and Naver Street View [10].
By utilizing the constructed datasets, a façade database can be constructed to identify
the characteristics of the façade designs of buildings on a selected street, and the façade de-
sign information can be extracted directly from this database. Additionally, the constructed
database can be used in various related fields. The façade data were directly collected
through road-view images and automatically labeled using deep learning. Through this
process, a method for efficiently constructing façade datasets was proposed. Simultane-
ously, we examined the applicability of the façade datasets in calculating the amount of
data required by the façade database to determine the design characteristics of the streets
in Dongseong-ro, Daegu Metropolitan City.

2. Background and Related Work

For a computer to be able to recognize visual information at the same level as hu-
mans, it is necessary to construct and learn the same amount of data that humans have
accumulated over the years through their eyes. To this end, Fei-Fei (2009) built an image
dataset called the ImageNet project, which has labeled approximately 1 billion images in
167 countries with 50,000 people. Therefore, the computer was able to provide a basis for
recognizing visual information [11]. However, the ImageNet project had a vast array of
general objects and information but was limited in the number of architectural objects and
information; therefore, it is difficult to apply it in the construction field. The construction of
a construction-specialized image dataset that labels construction information in relation to
construction-related objects is an essential early step in this process. Hence, it is necessary
to build big data from various mixed data formats, such as drawings, images, and text, in
the construction field and convert these into a construction dataset to utilize in construction
AI research. Accordingly, this section examines construction dataset cases and studies
related to façade labeling, and describes the research gap between previous studies and
this study.

2.1. Construction Datasets

1. Park (2006) built a building façade database manually to manage and provide the
color of the façade. Using this, the information of the façade was calculated and the
color characteristics of each façade composition type were summarized and analyzed.
Appl. Sci. 2022, 12, 7570 3 of 17

2. Park (2007) manually built a building façade database to provide quantified data.
Using this, the information of the façade was calculated to interpret the characteristics
of the current color range of the existing street side building façade and the color
range for each street was presented through the setting of the color evaluation model.
As described above, to understand the characteristics of the façade, the existing
research in Table 1 mainly needs to collect the façade image of the building by directly
photographing it with a camera, calculating information through manual operation, and
collecting data to establish a database. However, this method is considered costly and
time-consuming given the need to construct a building façade database of more than block
units; therefore, it is necessary to reduce the effort of a part or all of the existing database
construction process through automation. Therefore, we examine research related to façade
labeling, a method to reduce the existing database construction process.

Table 1. Research related to façade database.

No Author Title
A Study on Types of Elevation Composition and Color
1 Park, Sung-Jin Characteristics of Buildings by Urban Streetscape-Focused on
the Building of Central Aesthetic Districts in Gwang-Ju-(2006)
A Study on Establishing Color Ranges of Façade on Urban
2 Park, Sung-Jin Central Street-Focusing on Buildings of Central Aesthetic
District in Gwangju -(2007)

2.2. Façade Labeling Related Research

1. Teboul, Simon, Koutsourakis, and Paragios [2] improved the façade image labeling
performance by using shape grammar, random walk, map classification, and machine
learning methods to improve urban understanding and façade image-based 3D mod-
eling performance. In particular, it was possible to obtain a right-angled label image
through shape grammar and the labeling performance was improved using machine
learning [12].
2. Martinović (2012) used three layers to improve building façade-labeling performance.
Each layer was algorithmized using (1) an RNN, which is a machine learning tech-
nique; (2) enhanced labeling to enhance the recognition of the façade components;
and (3) structural knowledge to make the label image structurally valid [13].
3. Riemenschneider, Krispel, Thaller, Donoser, Havemann, Fellner, and Bischof [3]
improved façade labeling performance by combining low-level classification models
with medium-level object recognition models to compensate for the complicated
research of labeling building façades using existing shape grammar [14].
4. Jampani, Gadde, and Gehler [4] calculated the accuracy of the average 49.04,
overall 75.23, and IoU 39.57 based on a 2D image by dividing the façade through
automatic object recognition known as auto-context to label the point cloud of the
building façade image [15].
5. Jampani (2017) divided the façade through auto-context for quickly and accurately
dividing the 2D image and 3D point cloud of the façade compared to the research
published in 2015. The overall 84.4, IoU 63.7, and runtime (min) 150 accuracy were
calculated [16].
6. Research has been conducted in the field of computer vision to automatically generate
façade-based 3D modeling or to improve façade-labeling performance. It is difficult
to find cases where data are collected for architectural design studies.
As such, previous studies on architectural elements using computer vision technology
were mainly conducted to construct spatial information indoors and outdoors. In the
case of building façade automatic labeling research, the main area of study in the field
of computer vision was the automatic generation of 3D modeling based on the façade or
improvement of façade labeling performance, and it is difficult to find a data collection for
mance. This
mance. This shows
shows that
that the
the CMP
CMP database
database satisfies
satisfies an
an accurate
accurate and
and large
large amount
amount ofof train-
train-
ing data,
ing data, aa prerequisite
prerequisite for
for improving
improving the
the building
building façade
façade automatic
automatic labeling
labeling perfor-
perfor-
mance.
mance.
Appl. Sci. 2022, 12, 7570 2. Research related to façade segmentation.
Table 4 of 17
Table 2. Research related to façade segmentation.

No Author Title
architectural
H design research. It wassegmentation
confirmed that studies No. 5 andfaçade
6 in Table 2 used (2009)
the
11 Boulaassal,
Boulaassal, H Morphological
Morphological segmentation of building
of building façade images
images (2009)
CMP database as training data, and these two studies recorded the highest performance.
This shows
Segmentation of
Segmentation of building
building façades
façades using
using procedural
procedural shapeshape priors
priors
2 Teboul, O that the CMP database satisfies an accurate and large amount of training data,
a prerequisite for improving the building façade automatic(2010) labeling performance.
33 Martinović, A
Martinović, A A Three-Layered
A Three-Layered Approach
Table 2. Research related to façade segmentation.
Approach to to Façade
Façade Parsing
Parsing (2012)
(2012)
Irregular lattices for complex shape grammar façade parsing
44 Riemenschneider, H
Riemenschneider, H Irregular lattices for complex shape grammar façade parsing
No Author (2012)
Title
(2012)
1 Boulaassal, H Morphological segmentation of building façade images (2009)
5 Jampani, V Efficient Façade Segmentation Using Auto-context (2015)
2 Teboul, O Segmentation of building façades using procedural shape priors (2010)
3 Martinović, A Efficient 2D and 3D Façade
A Three-Layered ApproachSegmentation
to Façade ParsingUsing
(2012) Auto-Context
4 6 Jampani, VH
Riemenschneider, Irregular lattices for complex shape grammar façade parsing (2012)
(2018)
(2018)
5 Jampani, V Efficient Façade Segmentation Using Auto-context (2015)
6 Jampani, V Efficient 2D and 3D Façade Segmentation Using Auto-Context (2018)
Image processing
Image processing methods
methods include
include image
image classification,
classification, localization/detection,
localization/detection, and and
image segmentation,
image segmentation,
Imageasprocessing
as shown in
shown in Tableinclude
methods
Table 33 [17].
[17].image classification, localization/detection, and
image segmentation, as shown in Table 3 [17].

Table 3.
Table 3. Automatic
Automatic labeling
labeling
Table 3.
technology
technology
Automatic
analysis.
analysis.
labeling technology analysis.

Image Processing
Image Method
Processing Method Characteristic
Characteristic Example
Example

Alexnet,
Alexnet,
Predict Predict
Predict the class
the class of of
of
the class
Alexnet,
Classification
Classification
Classification Resnet,
Resnet,
Resnet,
the input
the input
Inception
Inception
Inception

Predict Predict
Predict the class
the class of of
of
Localization/
Localization/
Localization/
the class Yolo,
Yolo,Yolo,
objects and provide
objects and provide
Detection Detection objects and
Detection
provide
location information
R-CNN
R-CNN
R-CNN
location information

Predict the
Predict the class
class
Predict of
theof allof
all
class
Segmentation
Segmentation
Segmentation all pixels
SegNet
SegNet
SegNet
pixels
pixels

In calculating the façade information, the prediction of multiple labels and the width
of the object are necessary. As shown in Figure 1, semantic segmentation is most suitable
for the automatic label of the façade image that requires information such as the area
and number of the same class. Semantic segmentation classifies each pixel according
to a specified class and does not distinguish objects of the same class, whereas instance
segmentation distinguishes each object when the labels are distinguished.
In calculating the façade information, the prediction of multiple labels and the width
of the object are necessary. As shown in Figure 1, semantic segmentation is most suitable
for the automatic label of the façade image that requires information such as the area and
number of the same class. Semantic segmentation classifies each pixel according to a spec-
Appl. Sci. ified class
2022, 12, 7570 and does not distinguish objects of the same class, whereas instance segmenta-5 of 17

tion distinguishes each object when the labels are distinguished.

Figure 1. Concept of Figure

semantic segmentation
1. Concept and
of semantic instance and
segmentation segmentation.
instance segmentation.

2.3. Research Gap 2.3. Research Gap

Previous studies mainly collected façade images through camera photography to
Previous studies mainly
construct collected
databases, but in façade images
this study, through
a significant part ofcamera photography
the database to
construction process
construct databases, but in this study, a significant part of the database construction
was shortened through automation. In the case of façade automatic image labeling studies, pro-
cess was shortenedthethrough
façade-based automatic 3D
automation. In modeling
the casehas of been mainly
façade studied, but
automatic it is difficult
image labeling to find
a study to extract data for façade design research.
studies, the façade-based automatic 3D modeling has been mainly studied, but it is diffi-
cult to find a study3.toResearch
extractMethodology
data for façade design research.
3.1. Research Direction
3. Research Methodology
In this study, we attempted to automatically label façade images to quickly generate
large-scale façade datasets with much less effort than by the existing research methods.
3.1. Research Direction
Therefore, the main direction of this study is to utilize deep learning technology to accu-
In this study, we attempted
rately auto-label to automatically
non-quantified label
street façade
façade imagesimages
existingtoinquickly
road-viewgenerate
images gen-
large-scale façade datasets with much less effort than by the existing research methods.
erated from web portals such as Google Street View [9] and Naver Street View [10]. It is
Therefore, the main also necessaryof
direction to this
verifystudy
that the
isrequired datadeep
to utilize can be obtained technology
learning using the façadeto database
accu- to
identify the characteristics of the building design on the street.
rately auto-label non-quantified street façade images existing in road-view images gener-
The direction of this research, shown in Figure 2, was as follows:
ated from web portals
1.
such as Google Street View [9] and Naver Street View [10]. It is also
Definition of Façade Dataset Information
necessary to verify thatDefining
the required data
the required can be obtained
information in a datasetusing
based the
on thefaçade database
existing to of
characteristics
identify the characteristics of the building design on the street.
the façade.
2. Acquisition
The direction of this research, Method
shown of Dataset
in FigureSources
2, was as follows:
Selection of the source of the dataset from road-view images and image distortion
1. Definition of Façade Dataset
correction Information
technology.
Defining the required
3. information
Conversion Method of inDataset
a dataset based on the existing characteristics
Sources
of the façade. For the selection of converted datasets using deep learning model-based automatic
labeling technology, the road-view images were corrected using image distortion
2. Acquisition Method of Dataset Sources
technology, which was converted into computer-recognizable datasets. The deep
Selection of the source of method
learning the dataset from
included theroad-view
SegNet model images
trainingand
dataimage distortion
acquisition using the CMP
-correction technology.
database [18], training data optimization, and securing a façade automatic labeling
3. Conversion Method of Dataset
algorithm basedSources
on the SegNet model [19].
4. of
For the selection Dataset Acquisition
converted datasets using deep learning model-based automatic
Dataset was secured according to the conversion and acquisition methods
labeling technology,
5.
the road-view images were corrected using image distortion
Reliability Analysis of Dataset
technology, whichA was converted
reliability intoconducted
analysis was computer-recognizable datasets.
by comparing the façade The deep
information obtained
learning method included
with ground thetruth
SegNet
on themodel training(Dominant
four categories data acquisition using the
color, Secondary CMP
Color, Number
of Floors, Number of Buildings) of façade information that
database [18], training data optimization, and securing a façade automatic labeling can be calculated through
the datasets.
algorithm based on the SegNet model [19].
6. Verify Efficiency and Application of Dataset
4. Dataset Acquisition
Dataset was secured according to the conversion and acquisition methods
5. Reliability Analysis of Dataset
A reliability analysis was conducted by comparing the façade information obtaine
with ground truth on the four categories (Dominant color, Secondary Color, Numb
of Floors, Number of Buildings) of façade information that can be calculated throug
Appl. Sci. 2022, 12, 7570
the datasets. 6 of 17
6. Verify Efficiency and Application of Dataset
Dataset efficiency verification and utilization involved validation of façade datas
Dataseton
efficiency based efficiency verification
reliability and utilization
analysis involved of
and derivation validation
a means of façade dataset
to utilize the info
efficiency based on reliability analysis and derivation of a means to utilize the infor-
mation, which canwhich
mation, be calculated from
can be calculated thethedatasets.
from datasets.

Figure 2. Research direction.

Figure 2. Research direction.

3.2. SegNet Model

3.2. SegNet Model
Vijay Badrinarayanan, Alex Kendall, and Roberto Cipolla investigated the deep
Vijay Badrinarayanan,
learning-based imageAlex Kendall,
automatic labeling and
methodRoberto Cipolla
using semantic investigated
segmentation and con-the dee
cluded that among other models, the SegNet model [19] had a higher
learning-based image automatic labeling method using semantic segmentation and co performance in terms
of accuracy. Models primarily used for classification, such as VGG and AlexNet, maintain
cluded that aamong other models, the SegNet model [19] had a higher performance
layer that reduces the number and dimension of parameters and consequently loses
terms of accuracy. Models
some location primarily
information. used
However, thefor classification,
SegNet such
model efficiently as VGG
preserves and AlexNe
the location
maintain a layer that reduces
information using the the number
encoder and dimension
and decoder of parameters
structure. As shown in Figure 3, and consequent
the encoder
and decoder structures output the image compressed through the encoder and are labeled
loses some location information. However, the SegNet model efficiently preserves the l
with each pixel according to the designated class [19].
cation information using the encoder and decoder structure. As shown in Figure 3, th
encoder and decoder structures output the image compressed through the encoder an
are labeled with each pixel according to the designated class [19].
Figure 4 shows the input and output labels according to the specified class.
Appl. Sci. 2022, 12, 7570 7 of 17
Appl. Sci. 2022, 12, x FOR PEER REVIEW 7 of 17

Appl. Sci. 2022, 12, x FOR PEER REVIEW 7 of 17

Figure3.3.The
Figure TheSegNet
SegNetarchitecture.
architecture.

Figure
Figure 3. The4 SegNet
shows architecture.
the input and output labels according to the specified class.

Figure 4. Automatic labeling using SegNet model.

To4.implement
Figure automatic
Automaticlabeling
labeling labeling
using of images based on the SegNet model, a high level
SegNetmodel.
model.
Figure 4. Automatic using SegNet
of programming knowledge is required, but various frameworks, such as Tensorflow,
Pytorchzky,
Toimplement
To and Caffe,
implement are available
automatic
automatic labeling
labeling toofof
make this
images
images easier.
based
based ononAmong these,
theSegNet
the SegNet TensorFlow
model,
model, aahigh is
highlevelthe
level
most
of
of often-used framework
programming
programming knowledgein
knowledge isisimage processing.
required,
required, but various
but various frameworks,
frameworks, such as Tensorflow,
Tensorflow,
In this and
Pytorchzky,
Pytorchzky, study, we implemented
andCaffe,
Caffe, areavailable
are availableto an automatic
tomake
make image-labeling
thiseasier.
this easier. Amongthese,
Among algorithm
these, based on
TensorFlow
TensorFlow isisthe
the
SegNet
most
most model using
often-used
often-used TensorFlow.
framework
framework ininimage
imageprocessing.
processing.
Inthis
In thisstudy,
study,we weimplemented
implementedan anautomatic
automaticimage-labeling
image-labelingalgorithm
algorithmbased
basedon onthethe
3.3.
SegNetCMP
SegNetmodel Database
modelusingusingTensorFlow.
TensorFlow.
The higher the amount of correctly labeled data, the greater are learning results. Ac-
3.3.
3.3.CMP
CMPto
cording Database
Database
previous studies, as shown in Table 2. Automatic labeling of the façade based
The
on machine higher
The higher the
learning amount
showed
the amount ofthat
of correctly
the CMP
correctly labeled data,
database,
labeled data, the the
which greater
areare learning
is considered
greater learning results.
toresults.
possess Ac-a
According
relatively
cording to previous
tolarge amount
previous studies,
of data
studies, as
as and shown
shown in
contains Table
in Table 2. Automatic
information
2. Automatic labeling
on façades, of
labelingwas the façade
used
of the based
as training
façade based
on machine learning showed that the CMP database, which is considered to possess a
relatively large amount of data and contains information on façades, was used as training
Appl. Sci. 2022, 12, 7570 8 of 17

Appl. Sci. 2022, 12, x FOR PEER REVIEW 8 of 17

on machine learning showed that the CMP database, which is considered to possess a
relatively large amount of data and contains information on façades, was used as training
data. Therefore, we used the CMP database as training data to obtain an accurate and large
data. Therefore, we used the CMP database as training data to obtain an accurate and
number of datasets in this study.
large number of datasets in this study.
4. Framework
4. Framework
The ultimate purpose of this study was to propose a construction method that vastly
The ultimate purpose of this study was to propose a construction method that vastly
reduces
reduces the
the build-up periodfor
build-up period forthe
theconstruction
construction of façade
of façade datasets
datasets by applying
by applying artificial in-
artificial
telligence technologies such as deep learning. The framework of the proposed
intelligence technologies such as deep learning. The framework of the proposed construc- construction
method is presented
tion method in Figure
is presented in Figure5 and
5 andconsists
consistsof
of five modules.
five modules.

Figure 5. Framework of the proposed construction method.

Appl. Sci. 2022, 12, x FOR PEER REVIEW 9 of 1

Appl. Sci. 2022, 12, 7570 Figure 5. Framework of the proposed construction method. 9 of 17

4.1. Module 1: Raw Data Collection

4.1. Module 1: study,
In this Raw Data Collection
façade images extracted from road-view images were used as raw dat
In this study, façade images extracted from road-view images were used as raw dat
In this
for the study, façade
construction images
of the extracted
datasets. Thisfrom road-view
is because images
images of were used worldwide,
buildings as raw in
data for the construction of the datasets. This is because images of buildings worldwide,
cluding domestic ones, are being updated regularly, and most façade images overlookin
including domestic ones, are being updated regularly, and most façade images overlooking
the street can be secured. However, because the road-view images were obtained throug
the street can be secured. However, because the road-view images were obtained through
aa360
360° camera,
◦ camera, perspective
perspective distortion
distortion occurred,
occurred, as shown
as shown in Figures
in Figures 6 and 7.6Aand 7. A correctio
correction
method must therefore be devised
method must therefore be devised [9,10]. [9,10].

Figure6.6.Road-View
Figure Road-View Image
Image [10].[10].

Figure7.7.Distortion
Figure Distortion correction
correction [10,20].
[10,20].

4.2.
4.2.Module
Module2: 2:
Image Distortion
Image Correction
Distortion Correction
To use the portal site road view as the façade data for buildings, there are two methods:
To use the portal site road view as the façade data for buildings, there are two meth
labeling after performing distortion correction, as shown in Figure 6, and labeling directly
ods: labeling after performing distortion correction, as shown in Figure 6, and labelin
in a distorted image state. In the latter case, there is the advantage of not having to go
directlythe
through inprocess
a distorted image
of photo state. correction,
distortion In the latter
butcase,
therethere is the advantage
are disadvantages, ofthe
such as not havin
to go through the process of photo distortion correction, but there are disadvantages,
degree of distortion of the façade image being different from the road-view images. Hence, suc
as the
the degree
amount of distortion
of training of thetofaçade
data needs image In
be increased. being
this different
study, as afrom
resultthe
of road-view
testing the images
Hence, accuracy
labeling the amount of training
of both methods,data needs toof
the accuracy bethe
increased. In higher.
former was this study, as a result
Therefore, the of test
former
ing themethod of undergoing
labeling accuracy ofa bothdistortion correction
methods, process was
the accuracy of used [10]. was higher. There
the former
fore,Image distortion
the former correction
method requires an
of undergoing understanding
a distortion of image
correction processing
process and[10].
was used
the overall camera system, but various tools have been provided to make it easier to
Image distortion correction requires an understanding of image processing and th
process images.
overall camera system, but various tools have been provided to make it easier to proces
Adobe’s Lightroom [21] is the most commonly used tool for correcting image dis-
images.In addition, an automatic distortion-correction function is provided that reduces
tortion.
Adobe’stime.
the processing Lightroom [21] isthe
In this study, theimage
mostdistortion
commonly used tool
correction for correcting
process image distor
was performed
tion. Lightroom.
using In addition, an automatic distortion-correction function is provided that reduces th
processing time. In this study, the image distortion correction process was performed us
4.3.
ingModule 3: Image Distortion Correction
Lightroom.
In this study, a database containing both façade information and the corresponding
labeling image
4.3. Module 3:was
Imageused as training
Distortion data. As shown in Table 4, the façade information of
Correction
the building was limited to the dominant color, secondary color, number of floors, and
numberIn of
this study, [22].
buildings a database containing both façade information and the correspondin
labeling image was used as training data. As shown in Table 4, the façade information o
the building was limited to the dominant color, secondary color, number of floors, an
number of buildings [22].
Appl. Sci. 2022, 12, 7570 10 of 17

Appl. Sci. 2022, 12, x FOR PEER REVIEW 10 of 17

Table 4. The Definition of façade information.

Façade Information Definition

Table 4. The Definition of façade information.
Dominant Color The largest color in the Façade
Façade Information
Secondary Color Definition
The second largest color in the façade
Dominant Color Single-storyThebuilding (1st in
largest color floor)/Low-rise
the Façade building (2~3rd floor)
NumberColor
Secondary of Floors Medium-rise building
The second largest color(4–5 floors)/High-rise
in the façade building
(more than six
Single-story building (1st floor)/Low-rise floors)
building (2~3rd floor)
Number of Floors
Number of Buildings
Medium-riseNumber
building of buildings
(4–5 facing the
floors)/High-rise street(more
building in one horizontal
than six floors)block
Number of Buildings Number of buildings facing the street in one horizontal block

The CMP database [23] consists of the 12 most basic (Wall, Molding, Cornice, Pillar,
The CMP database [23] consists of the 12 most basic (Wall, Molding, Cornice, Pillar,
Window, Door, Sill, Blind, Balcony, Shop, Deco, and Background) classes of façades with ap-
Window, Door, Sill, Blind, Balcony, Shop, Deco, and Background) classes of façades with
proximately
approximately 600 labeled
600 labeleddata
datainputs.
inputs. In
In this study,among
this study, amongthe
the1212classes,
classes, except
except forfor
thethe class
with
class little influence
with little on calculating
influence on calculatingthethefaçade
façadeinformation, similarclasses
information, similar classes were
were combined
com-
and re-designated
bined into four
and re-designated into classes. The The
four classes. redesigned classes
redesigned areare
classes shown
shownininTable
Table5 5[18].
[18].
Table 5. Class of training data.
Table 5. Class of training data.
Class of Training Data Class of CMP DB
Class of Training Data
Walls Façade (Wall) Class of CMP DB (Grey)
Window Walls Façade (Wall)
Windows (Grey)
(Red)
Window
Entrances Windows
Shop + Door (Red)
(Green)
Background
Entrances Background
Shop + Door (Blue)
(Green)
Background Background (Blue)
4.4. Module 4: Training Data Optimization
4.4. Module 4: Training Data Optimization
CMP databases with re-designated classes require preprocessing to ensure optimal
CMP databases with re-designated classes require preprocessing to ensure optimal
learning in the SegNet model. Pre-processing operations consist of resizing operations to
learning in the SegNet model. Pre-processing operations consist of resizing operations to
meet the input value conditions of the SegNet model, as shown in Figure 8, crop operations
meet the input value conditions of the SegNet model, as shown in Figure 8, crop opera-
to remove elements that impair the quality of the image to make the image similar to the
tions to remove elements that impair the quality of the image to make the image similar
actual
to the environment, and augmentation
actual environment, operations
and augmentation to increase
operations thethe
to increase effectiveness
effectivenessofoflearning
by increasing
learning trainingtraining
by increasing data by horizontally
data reversing
by horizontally thethe
reversing image
image ororadjusting
adjusting the
the gamma
value
gamma [19].
value [19].

Figure 8. Optimize training data.

Figure 8. Optimize training data.
high
high level
level ofof noise
noise werewere excluded.
excluded.
The
The selected
selected building
building images
images werewere collected
collected according
according to to thethe raw
ra
method, and the collected images were distorted
method, and the collected images were distorted and corrected to obtainand corrected to obtain
Appl. Sci. 2022, 12, 7570 the
the test
test data
data were
were alsoalso needed
needed to to calculate
calculate the the optimal
optimal information
information
11 of 17 fro
fro
façade
façade image labeling algorithm based on the SegNet model, the resize aa
image labeling algorithm based on the SegNet model, the resize
performed
performed during the the preprocessing
preprocessing operation
operation conducted
conducted on on thethe traini
4.5. Module 5:during
Test Data Selection traini
tion, a ground
tion, aInground truth
this study, truth
labeling
labeling
a part of
image
image
Dongseong-ro,
was
was
Daegu
required
requiredCity,
Metropolitan
to measure
to measure the labelin
the labelin
which is a central
test image
commercial
test and
and labeling.
image street, was used as the test data. The building façade information was limited
labeling.
to buildings with low noise, such as street trees. Elements such as trees that generated a
high level of noise were excluded.
5.
5. Results
Results
The selected building images were collected according to the raw data collection
method, and the collected images were distorted and corrected to obtain test data. Because
5.1.
5.1. Façade
theFaçade
Information
test dataInformation
were also needed
Calculation
Calculation Criteria
Criteria
to calculate the optimal information from the automatic
façade imagethelabeling algorithm based on the SegNet in
model,
Using
Using the labeling
labeling image
image obtained
obtained thethetest
in the resize
test and crop
data
data task was the bu
selection,
selection,
performed during the preprocessing operation conducted on the training data. In addition,
the bu
formation
formation was
was
a ground truth calculated
calculated
labeling image wasas follows
asrequired
follows in
in Table
Table
to measure the 6:
6:
labeling accuracy of the test
image and labeling.
1.
1. Dominant
Dominant color: color: Combines
Combines the the area
area ofof the
the label
label ofof the
the same
same class
class to
to cc
of
of the
the corresponding
5. Results corresponding façade façade image
image of of the
the largest
largest class
class asas the
the domin
domin
5.1. Façade Information Calculation Criteria
2. Secondary
2. Secondary color:
color: The area of those labels of the same class is comb
The area of those labels of the same class is comb
Using the labeling image obtained in the test data selection, the building façade
the
the color
information was of
color of the
the corresponding
calculated as follows in Table façade
corresponding façade image
6: image of of the
the second
second largest
largest clcl
ary
1. ary color.
color.
Dominant color: Combines the area of the label of the same class to calculate the color
3. of the corresponding façade image of the largest class as the dominant color. is recogniz
3. 2. Number
Number of
of floors
floors (building
(building height):
height): The The entrance
entrance class class is recogniz
Secondary color: The area of those labels of the same class is combined to calcu-
floor
floor and
late theand the
colorthe level
of thelevel of the window
of the window
corresponding façade image class
class is
of theis recognized
recognized
second as
asaseach
largest class the floo
each floo
number
number of
secondary
of floors.
color.
floors.
3. Number of floors (building height): The entrance class is recognized as the ground
4. Number
4. Numberfloor and the
of buildings:
oflevel
buildings: The number
Theclass
of the window number of
of buildings
is recognizedbuildings was
as each floorwas
divided
divided
to calculate
by th
the by th
classes
number of
classes divided
floors.
divided by
by the
the background
background class. class.
4. Number of buildings: The number of buildings was divided by the number of wall
classes divided by the background class.
Table
Table 6.
6. Extraction
Extraction of
of façade
façade information.
information.
Table 6. Extraction of façade information.
No
No No
Façade Information
FaçadeFaçade
Information
Information
Extraction Method
Extraction
Extraction Method
Extraction Method Method

11 1 Dominant
Dominant Color
Color
Dominant Color

22 2 Secondary
Secondary Color
Color
Secondary Color

33 3 Number
Number of
of Floors
NumberFloors
of Floors

44 4 Number of
of Buildings
Number
Number of Buildings
Buildings

5.2. Training Data Output and Analysis

5.2.
5.2. Training
Training
In the CMPData
Data Output
Output
database, and Analysis
and model-based
the SegNet Analysis automatic façade image-labeling algo-
rithm was tested on images that were not used for learning. Table 7 shows the labeling
In
image
the
Inand CMP
theoutput. database,
CMPTables
database, the
8–10 showthe
SegNet
SegNet
the test
model-based automatic façade i
results. model-based automatic façade i
gorithm
gorithm was was tested
tested onon images
images that
that were
were notnot used
used for
for learning.
learning. Table
Table 77 sh
s
image and output. Tables 8–10 show the
image and output. Tables 8–10 show the test results. test results.
Appl.
Appl. Sci.2022,
Appl. 2022,
Sci. 12,x12,
2022, xFOR
FOR PEER
x FOR REVIEW
PEER REVIEW 12 12
of 17
of 17
Appl. Sci.Sci.
2022, 12, 12,
7570 PEER REVIEW 1212
of of
17 17

Table
Table
Table 7.7.Example
7.Table Example ofoflabeling
7. Example
Example labeling
of labeling
of labeling.
Input
Input
Input Image
Image Ground Truth
Ground Image
Truth Image Output
Output Image
Image
Input Image
Image Ground Truth
Ground Image
Truth Image Output
OutputImage
Image
Input image
Input
Input ofoflabeling
image
image labeling
of labeling Output image
Output imagethrough
throughauto-matic
auto-matic
Ground Truth
Ground Image Output Output
Image
Truth image through auto-matic
image through
Input imageimageimage image Ground
of labeling
image Truth
Ground Image
Truth Image labeling
auto-matic
algorithm
labeling algorithm
labelinglabeling
algorithm
algorithm

Table 8. mIoU of CMP test data.

Table
Table 8.8.mIoU
Table mIoUofofCMP
8. mIoU CMP testdata.
of CMP
test data.
test data.
Class mIoU
Class
Class
Class mIoU
mIoU
mIoU
Walls
Walls
Walls 0.70 0.70
0.70
Walls
Windows 0.660.70
Windows
Windows
Windows
Entrances
0.66
0.510.660.66
Entrances
Entrances
Entrances
Background 0.51
0.720.510.51
Background
Background
Average
Background 0.72
0.630.720.72
Average
Average
Average 0.63
0.630.63
Table 9. Pixel accuracy of CMP test data.
Table9.9.Pixel
Table
Table Pixel accuracy
9. Pixel of CMP
accuracy
accuracy testtest
of CMP data.
data.
Dataof CMP test data. Pixel Accuracy
Data
Data
Average Data Pixel
Pixel
0.77 Accuracy
Pixel Accuracy
Accuracy
Average
Average
Average 0.77
0.770.77
Table 10. Façade information accuracy of CMP test data.
Table
Table 10.Façade
Table
10. Façade
10. information
Façade accuracy
information
information ofofCMP
accuracy
accuracy CMP testdata.
of CMP
test data.
test data.
Façade Information CMP Test Data
Façade
Façade Façade Information
Information
Information CMP
CMP CMP Test TestData Data
Dominant Color 0.92Test Data
Dominant
Dominant
Dominant
Secondary Color Color
Color Color 0.92
0.730.92 0.92
The Number Secondary
Secondary
Secondary of FloorsColor
Color Color 0.73
0.890.73 0.73
The Number
The TheNumberof Buildings
Number of Floors
of Floors 0.81 0.89 0.89
The Number of Floors 0.89
Average 0.84
The
The The Number
Number Number ofofBuildings
Buildings
of Buildings 0.81
0.81 0.81
Average
AverageAverage 0.84
0.840.84
A total of 185 labeling tests were conducted using 37 test images and the results were
analyzed. The time required for automatic labeling of the 37 images was 85.47 s, with each
AAtotal
image being
total
A totalofof185
labeled
185
of 185 labeling
labeling
labeling
in
teststestswerewere conducted
conducted using
using 37 37
2.31 s. tests were conducted using 37 test images and the results were
testtest
images
images and andthetheresults
resultswere were
analyzed.
analyzed.
analyzed. The
The Thetime time required
required forautomatic
automatic
for automatic labeling
labelingofofthe
the
of 3737images
the images
37 images was was 85.47
85.47s,with
with
s, witheacheach
The mIoU intime
Table required
4 is an for index indicating labelingthe accuracy of labeling was for 85.47
eachs,class each
by
image
image
image beingbeing
being labeled
labeled
labeled truth in 2.31
in
in 2.31 2.31s. s.
s. and the extracted labeling image, and was indicated as
comparing the ground image
The ThemIoU mIoU inTable
Table
in Table 4isis4anan
is anindexindex indicating
indicating thethe accuracy
accuracy of labeling
of labeling forforeacheachclass
class by by
1 when the accuracy was 100%. index indicating the accuracy of labeling for each class by
The mIoU in 4
comparing
comparing
comparing the
the theground
ground ground truth
truth
truthofimage imageimage and and
and theandthe extracted
the extracted
extracted labeling
labeling
labeling image,
image,image, and and
and class was wasindicated
was indicated indicated
Therefore, the reliability the number location of objects in each can be
as1as
as 1when
when
1 when thethe
the accuracy
accuracy
accuracy was
was was 100%.
100%. 100%.
checked, and the walls and backgrounds were most successfully labeled with a mIoU of
Therefore,
Therefore,
Therefore, thethe
the reliability
reliability
reliability ofofthethethe
of number
numbernumber andand and location
location
location ofofobjects
objects
of inineach
objects each
in eachclass
class cancan
class
can bebe be
0.70 or higher.
checked,
checked,
checked, and
andof and the
the walls
the walls
walls and and and backgrounds
backgrounds
backgrounds were
were were most most successfully
successfully
most successfully labeled
labeled
labeled with with a
with mIoU
a mIoUa mIoU of of
In the case windows, mIoU 0.66 showed relatively compliant accuracy. However, in of
0.70 or
0.70 higher.
or higher.
the0.70
caseorofhigher.
entries, mIoU 0.51 showed a relatively low accuracy compared to the labeling
Inthe
In thethe
In case
case ofofwindows,
case windows,
of windows, mIoU
mIoU mIoU 0.66
0.66 showed
0.66
showed showed relatively
relatively compliant
compliant accuracy.
accuracy. However,
However,
accuracy of other classes. The average mIoU for all relatively
classes was compliant
0.63. accuracy. However,
in the
in The
the case
in the
case
pixelcaseof entries,
of entries,
ofaccuracy
entries, mIoU
mIoU
listedmIoU 0.51
0.51
in Table0.51showed
showed showed a relatively
a relatively
a relatively
5 is an index showinglow
lowlow accuracy
accuracy
accuracy
similarity compared
compared
compared
between to
tothe the
totwo
the label-
the
label- label-
ing
ing accuracy
ing accuracy
accuracy of of other
of
other other classes.
classes.
classes. The
The Theaverage
averageaverage mIoU
mIoU mIoU
images by comparing all pixels of the ground truth image and the extracted labeling image. for
for all
for
all classes
all
classesclasses was
was 0.63.
was
0.63. 0.63.
Thus, the The
The Thepixel
pixel pixel
reliability accuracy
ofaccuracy
accuracy listed
listed
the entire listedininTable
area Table
in
ofTable
the55extracted
isis5anan index
is index
an index showing
showing
labeling showing
image similarity
similarity
similarity
can bebetweenbetween
between
confirmed. thethe
the two
two two
images
images
images by comparing
by
by comparing
The accuracy comparing
of the all all pixels
all
pixels
façade pixels of the
of
of the ground
information ground
the ground
listed truth
truth truthimage
image10
in Table image and
and and the extracted
the
theindex
is an extracted
extracted labeling
labeling
labeling
indicating im-
theim- im-
age. Thus,
age.
age. Thus,
accuracy Thus,
of thethe the thereliability
reliability
reliability
calculated of the
of
of the entire
information theentire
entire
of the area
area area
output of
of the the
of theextracted
extracted
extracted
labeled labeling
imagelabeling labeling
by comparing image
image
imagethe can can
canoutputbe con-
be
be con- con-
firmed.
firmed.
firmed.
labeling image and the ground truth image. The façade information accuracy was 0.84,
which was The
The Theaccuracy
accuracy
accuracy
higher than ofofthe
the the
of façade
the
façade
pixel façade information
information
information
accuracy and mIoU listed
listedlisted inTable
ofinthe Table
inCMPTable
1010 isan
is10 an
database. index
is index
an index
This indicating
indicating
indicating
is because thethe
the
accuracy
accuracy
theaccuracy
class mIoU of
of thethe
of
of the calculated
the calculated
calculated
walls and information
information
information
windows, of of the
of the
the output
which output
output
occupylabeled labeled
labeled imageimage by
image by comparing
an overwhelming comparing
by comparing
proportion,the the
was out-
out-out-
the
put labeling
put
putamong
high labeling
labeling image
theimage image
classes and
and the
and ground
the
the ground
produced ground
by the truth
truth truthimage.image.
image. The
dominant The Thefaçade
andfaçade façade
secondary information
information
information accuracy
accuracy
colors. accuracy was 0.84, was was0.84,
0.84,
which
which was was higher
higher than thanthe pixel
the pixelaccuracy
accuracy and and
which was higher than the pixel accuracy and mIoU of the CMP database. This is becausemIoUmIoU of the
of CMP
the CMP database.
database. This Thisis because
is because
Appl. Sci. 2022, 12, 7570 13 of 17

5.3. Test Data Output and Analysis

The test was conducted on the façade image (raw data) extracted from the road-view
images and the results were analyzed. The test data were intended to determine if the
pixel accuracy, mIoU, and façade information could be maintained and how it could be
improved through distortion correction, even for characteristics that differed from the
training data. Thirty façade images (referred to as original data) extracted from the original
road views and 30 façade images (referred to as distortion correction data) extracted from
the distortion-corrected road views were used as test data.
As shown in Tables 11 and 12, when the original data and distortion correction data
were compared, the average mIoU and pixel accuracy values of the distortion correction
data were higher by an average of 0.05 and 0.09, respectively. In addition, as shown in
Table 13, the accuracy of the façade information was also 0.07 higher than that of the
original. The results show that the distorted and corrected data can be effectively labeled
using the CMP database compared with the original data.

Table 11. mIoU of original and distortion correction data.

Class Original Data Distortion Correction Data

Walls 0.52 0.69
Windows 0.27 0.53
Entrances 0.338 0.50
Background 0.43 0.43
Average 0.39 0.54

Table 12. Pixel accuracy of original and distortion correction data.

Test Pixel Accuracy(Average)

Original Data 0.66
Distortion Correction Data 0.75

Table 13. Façade information accuracy of original and distortion correction data.

Façade Information Original Data Distortion Correction Data

Dominant Color 0.80 0.83
Secondary Color 0.76 0.80
The Number of Floors 0.83 0.93
The Number of Buildings 0.80 0.90
Average 80 0.87

In this study, we compared the CMP database and the distortion correction data by
viewing the distortion correction data showing higher accuracy of the façade information
as a representative image of the façade image extracted from road views. As shown in
Tables 14 and 15, the average mIoU and pixel accuracy values of the distortion correction
data were lower than the CMP database by 0.09 and 0.02, respectively.

Table 14. mIoU of CMP test and distortion correction data.

Class CMP Test Data Distortion Correction Data

Walls 0.69 0.69
Windows 0.66 0.53
Entrances 0.51 0.50
Background 0.72 0.43
Average 0.63 0.54
Appl. Sci. 2022, 12, 7570 14 of 17

Table 15. Pixel accuracy of CMP test and distortion correction data.

Data Pixel Accuracy (Average)

CMP Test Data 0.77
Distortion Correction Data 0.75

Table 16 shows the accuracy of the façade information obtained by the output and
ground truth images. The accuracy of the façade information of the distorted and corrected
data was lower than that of the CMP database by 0.09, but it was higher by 0.07, 0.04, and
0.09 in the secondary color, number of floors, respectively, and the number of buildings
was 0.03 higher in the accuracy of the average façade information. This is a marginally
different result from the mIoU and pixel accuracy values and is assumed to be the result
of the façade characteristics of Dongsung-ro being tested. The façade characteristics of
Dongseong-ro are as follows.

Table 16. Façade information accuracy of CMP test and distortion correction data.

Façade Information CMP Test Data Distortion Correction Data

Dominant Color 0.92 0.83
Secondary Color 0.73 0.80
The Number of Floors 0.89 0.93
The Number of Buildings 0.81 0.90
Average 0.84 0.87

1. The accuracy of the secondary color calculation was high because the ratio of the area
of the window class, which has a relatively high mIoU, is high and the color of the
window is often calculated as a secondary color because it is included in the façade.
2. Compared to the CMP database, the shape of the window is clear and designed for
each floor; therefore, the number of floors calculation accuracy of the building is high.
3. The boundaries between buildings were ambiguous in the CMP database, but the
test data were clear. Therefore, despite the higher mIoU of the wall and background
classes in the CMP database, the number of buildings calculation accuracy of the
buildings in the test data was higher.

6. Discussion
6.1. Discussion of the Results
The resulting value of the revised version was higher than the original value for all
indicators. This means that if there is distortion in the raw data image, the accuracy of the
façade information can be improved through distortion correction. Conversely, it is clear
that the mIoU and pixel accuracy affect the façade information accuracy, but the façade
information accuracy varies depending on the data situation. It was confirmed that the
factors that affect the accuracy of the façade information include the clarity of the shape of
the class that appears in each image data, the clarity of the boundaries of each floor of the
window class, and the clarity of the boundaries between buildings. Therefore, to increase
façade information accuracy, it is necessary to obtain detailed learning criteria for mIoU
and pixel accuracy and secure training data to increase mIoU and pixel accuracy depending
on the characteristics of the data.

6.2. Limitations
Several limitations are acknowledged in this study. First, there was a limitation
that made automatic labeling difficult, in the case of high noise such as street trees and
streetlights. To this end, we are studying ways to increase the façade information accuracy
calculation using NVDIA Image Inpainting [24], a technology that automatically removes
noise and by securing high-quality readings.
Appl. Sci. 2022, 12, 7570 15 of 17

Secondly, the SegNet model exhibited low quality learning due to only using 600 images
of the CMP database as training data. To this end, we will build large-scale databases by
increasing the accuracy of labeling to a certain level from the road-view images.
These issues in part provide a general perspective to the limitations of the deep
learning approach. The approach requires careful data collection and preparation.

6.3. Practical Implications

Researchers and the government can use façade automatic labeling-based façade
recognition to extract the window ratio at a specific distance from the road-view images
and thereby provide a basis for the window ratio as a demonstration of energy consumption.
This will be comparable to the existing window ratio studies or government regulations.
The architectural industry will be able to automatically collect façade images and use
them to model them to develop the level of architectural services by promoting digital
transformation of architecture.
Through digital transformation, this study will contribute to research that can extract
exterior building information from city-scale architecture digitized on the 3D map in the
opposite case [25]. Furthermore, the application of window recognition methodology
on road-view images using deep learning developed in this study will contribute to the
development of a novel automated method for simulating the wind damage to building
windows at the city-scale architecture [26].

6.4. Directions for Future Work

In this study, we proposed the possibility of obtaining quantitative and qualitative
façade design information by extracting patterns of façade design from anywhere in
the world. In the future, based on improved labeling accuracy, we will build datasets
of automatically labeled images from the road-view images for research on automated
color collection and demonstration of energy consumption according to window area
ratio [1,22,27–29].

7. Conclusions
This study proposed a method for efficiently constructing façade datasets through the
process of collecting façade data from road-view images and automatically labeling them
using deep learning. The goal of this study was to automatically label façade images to
quickly generate large-scale façade datasets with much less effort than the existing research
methods. We tested the possibility of calculating the data required by the façade database
to comprehend the design characteristics of the street.
The semantic segmentation method was used for automatic image labeling and a
SegNet model with high performance in terms of accuracy was used. The labeling subjects
in this study were walls, windows, entrances, and background. The façade information
calculated was the dominant color, secondary color, number of floors, and number of
buildings. Six hundred images of CMP database-based training data were used to train the
SegNet model-based automatic image labeling algorithm and 30 road-view images were
used to test the algorithms. The automatic labeling of the test images showed a result of one
per 2.31 s. Automatic labeling and façade information calculation showed results of 0.54,
0.75, and 0.87 for mIoU, pixel accuracy, and accuracy of façade information, respectively.
This study confirmed that the computer could recognize the visual information of
the façade image and calculate meaningful façade information of the road-view images
at a high speed. In addition, the results of the automatic labeling and façade information
analysis were analyzed using the method of calculating the façade information proposed
in this study. Detailed criteria for the required mIoU and pixel accuracy values based on
the characteristics of the data will be prepared and applied in further studies to increase
the façade information accuracy calculation.
Appl. Sci. 2022, 12, 7570 16 of 17

Author Contributions: H.G. conceived experiments, analyzed data, and wrote papers; S.C. investi-
gated prior research and edited thesis; S.C. supervised the research. All authors have read and agreed
to the published version of the manuscript.
Funding: This work is supported by the Korea Agency for Infrastructure Technology Advancement
(KAIA) grant funded by the Ministry of Land, Infrastructure and Transport (Grant 22AATD-C163269-
02). This research is a basic research project in the field of Ph.D. student research incentive project
that was conducted with the support of the Korea Research Foundation with funding from the
government (Future Creation Science) in 2021. Assignment number: NRF-2021R1A6A3A13045849.
Institutional Review Board Statement: Not applicable.
Informed Consent Statement: Not applicable.
Data Availability Statement: Not applicable.
Conflicts of Interest: The authors declare that they have no known competing financial interest or
personal relationships that could have influenced the work reported in this study.

References
1. Kim, T. Inter-Industry Labor Mobility and Changes of Labor Productivity in Korea. J. Ind. Econ. Bus. 2018, 31, 113–131.
2. Jeong-Wook, S.; Yoon, J.-S.; Paek, J.H. A study on construction productivity measurement method. J. Archit. Inst. Korea 2003,
19, 101–108.
3. Kang, I.S.; Moon, J.W.; Park, J.C. Recent research trends of artificial intelligent machine learning in architectural field-review of
domestic and international journal papers. J. Archit. Inst. Korea Struct. Constr. 2017, 33, 63–68. [CrossRef]
4. Choi, J. A study on the standardization strategy for building of learning data set for machine learning applications. J. Digit.
Converg. 2018, 16, 205–212.
5. Kim, W.Y. A Model for Construction Data Integration Based on Growth of Construction Object throughout the Overall Project
Phases. In Proceedings of the Korean Institute of Construction Engineering and Management; Korea Institute of Construction Engineering
and Management: Seoul, Korea, 2003; pp. 143–150.
6. Kim, H.-J.; Ji, S.Y.; Jun, H. A Study on Application of Artificial Intelligence Technology to BIM Architectural Planning—Focus on
Structural BIM Model in Early Design Phase. Korea Soc. Art Des. 2019, 22, 229–242.
7. Available online: https://cloud.eais.go.kr/ (accessed on 1 May 2022).
8. Available online: https://www.urbanbase.com/ (accessed on 1 May 2022).
9. Available online: https://www.google.com/streetview/ (accessed on 1 May 2022).
10. Available online: https://map.naver.com/v5/search?c=14371764.2994954,4182680.8899432,16,0,0,0,dha (accessed on 1 May 2022).
11. Deng, J. A large-scale hierarchical image database. In Proceedings of the IEEE Computer Vision and Pattern Recognition, Miami,
FL, USA, 20–25 June 2009.
12. Teboul, O.; Simon, L.; Koutsourakis, P.; Paragios, N. Segmentation of building facades using procedural shape priors. In
Proceedings of the 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, San Francisco, CA,
USA, 13–18 June 2020; pp. 3105–3112.
13. Martinović, A.; Mathias, M.; Weissenberg, J.; Van Gool, L. A three-layered approach to facade parsing. In European Conference on
Computer Vision; Springer: Berlin/Heidelberg, Germany, 2012; pp. 416–429.
14. Riemenschneider, H.; Krispel, U.; Thaller, W.; Donoser, M.; Havemann, S.; Fellner, D.; Bischof, H. Irregular lattices for complex
shape grammar facade parsing. In Proceedings of the 2012 IEEE Conference on Computer Vision and Pattern Recognition,
Providence, RI, USA, 16–21 June 2012; pp. 1640–1647.
15. Jampani, V.; Gadde, R.; Gehler, P.V. Efficient facade segmentation using auto-context. In Proceedings of the 2015 IEEE Winter
Conference on Applications of Computer Vision, Waikoloa, HI, USA, 5–9 January 2015; pp. 1038–1045.
16. Gadde, R.; Jampani, V.; Marlet, R.; Gehler, P.V. Efficient 2D and 3D facade segmentation using auto-context. IEEE Trans. Pattern
Anal. Mach. Intell. 2017, 40, 1273–1280. [CrossRef] [PubMed]
17. Kim, B. Trend of Deep Learning-Based Image Classification Technology. J. Korean Inst. Commun. Sci. 2018, 8–14.
18. Tylecek, R. The CMP Facade Database; Research Report CTU–CMP–2012–24; Czech Technical University: Prague, Czech, 2012.
19. Badrinarayanan, V.; Kendall, A.; Cipolla, R. SegNet. A deep convolutional encoder-decoder architecture for image segmentation.
arXiv 2015, arXiv:1511.00561. [CrossRef] [PubMed]
20. Adobe. Lightroom. Available online: https://www.adobe.com/kr/products/photoshop-lightroom.html?sdid=4NM89C54&mv=
search&mv2=CC\T1\textbar{}Naver\T1\textbar{}BrandSearch&s_kwcid=AL!3085!87!cf7b7fd015d18b1dc0ce1795fe9bb75d&
ef_id=XnckVQAAAzhc_6x_:20200814043206:s (accessed on 1 May 2022).
21. Available online: https://www.adobe.com/kr/products/photoshop-lightroom (accessed on 1 May 2022).
22. Park, S.J.; Lim, K.C.; Lee, C.W. A Study on Types of Elevation Composition and Color Characteristics of Buildings by Urban
Streetscape-Focused on the Building of Central Aesthetic Districts in Gwang-Ju. J. Archit. Inst. Korea Plan. Des. 2006, 22, 201–208.
Appl. Sci. 2022, 12, 7570 17 of 17

23. Tyleček, R.; Šára, R. Spatial pattern templates for recognition of objects with regular structure. In German Conference on Pattern
Recognition; Springer: Berlin/Heidelberg, Germany, 2013; pp. 364–374.
24. NVDIA’s IMAGE INPAINTING. Available online: https://research.nvidia.com/publication/2018-09_image-inpainting-irregular-
holes-using-partial-convolutions/ (accessed on 1 May 2022).
25. Shon, D.; Noh, B.; Byun, N. Identification and Extracting Method of Exterior Building Information on 3D Map. Buildings 2022,
12, 452. [CrossRef]
26. Gu, D.; Chen, W.; Lu, X. Automated assessment of wind damage to windows of buildings at a city scale based on oblique
photography, deep learning and CFD. J. Build. Eng. 2022, 52, 104355. [CrossRef]
27. Lee, J.-H.; Choi, J.-Y.; Kim, J.-H. A Study on the Planning Method for Public Street Space Using Analysis of Façade Characteristics;
Improving Project for Public Design in Daegu: Dongseong Street. J. Archit. Inst. Korea Plan. Des. 2010, 26, 29–36.
28. Lee, K.H.; Ryu, J.R.; Choo, S.Y. A correlation analysis on’cooling and heating loads’ and’window to wall ratios’ in accordance
with the form of simulation-based office towers-focused on BIM model at the early design stages. J. Archit. Inst. Korea Plan. Des.
2013, 29, 15–24.
29. Lee, C.; Lee, G.; Won, J. A Comparative Study on Construction Productivity Trends as Analyzed by Various Measures-South
Korea, the US, the UK, and Japan (1995–2015). J. Archit. Inst. Korea Struct. Constr. 2019, 35, 175–184.