Validating data acquired with experimental multimodal biometric system installed in bank branches

The engineered biometric stands were also completed by a hand vein scanner available on the market (Fujitsu Identity Management and PalmSecure 2017), serving for the purpose of reference to the methods implemented by the research team.

In the course of the research, the previous knowledge in the field of multimodal technology of identity verification was used and analyzed in the context of many research challenges. The research work was conducted in real banking environment, since PKO Bank Polski offered access to 100 teller stands located in 60 bank branches, which creates the opportunity to develop the experimental research at a large scale.

The development of mobile biometrics has accelerated sharply in recent years. The present verification solutions are based on fingerprint, voice or face scan. Unfortunately, none of the available methods of bank clients verification provides complete reliability for the whole range of possible cases. Therefore, the authors, on the basis of their experience, proposed the use of several biometric methods and their effective combination. A secure data communication system was also established for the purpose of biometric data transmission inside the bank communication network. The technology had been implemented in the form of a system for acquiring, storing, analyzing, and combining (fusing) biometric data. When using simultaneous biometric verification of many modalities, from the practical and research viewpoint, the significant issue is studying the level of acceptance of individual technology solution by both: banking clients and banking tellers.

Especially, combining the biometric methods should be beneficial for the person being verified, both in terms of improving the safety and convenience of using the services of (lowering the rate of false acceptance and false rejection errors). The central element of this process is to discern information on the convenience of individual methods and their acceptance.

The research on the development of the dynamic signature method (described in Section 2) is significant enough to lead to the development of the most acceptable method of identity verification in the society. A handwritten signature has been for many centuries the most prominent means of expressing one’s approval, and to confirm the identity. Owing to the work carried out on the technology of biometric signature, it will be possible to reduce the risk of signature forgery by adding recorded extra dynamic measures of the signature. Therefore, an important step in our research, consequently leading to a large-scale implementation, is the further advancement of methods of acquisition, analysis and interpretation of graphomotoric data. In particular, the leading role will be assigned to the methods of acquisition and playback of dynamic handwritten signature on the basis of multi-dimensional data obtained from accelerometric and gyroscopic sensors, pen-on-paper or pen-on-screen pressure sensors, as well as from the sensors of the pressure exerted on the handle of the pen. During the broadened research on the developed biometric pen technology (Lech and Czyżewski 2016; Lech et al. 2016; Shanker and Rajagopalan 2007), the action was taken to analyze the functioning of the individual components that make up the pen for the mobile applications. The engineered biometric pen and the whole biometric stand installed in one of bank branches are depicted in Fig. 1.

Open image in new window

Besides the numerous practical applications of the face image analysis methods known from the literature (Braga 2017; Bhele and Mankar 2015; Borade et al. 2016; Papatheodorou and Rueckert 2007) and used for personal identification, the progress and spreading of the image acquisition technology resulted also in the market introduction of widely available lidar technologies (laser scanning to produce a spatial representation of the visual three-dimensional objects). A current example of a commercial incarnation of this type of technology is the Time-of-Flight (ToF) camera. It uses temporal relations of radiated and reflected light in order to visualize spatial objects. This way, an image extracted from the acquired data cloud can be used to restore the contour of the face, which is a strong distinctive feature of individuals. Hence, this modality has been also implemented our project to enable carrying out research in the field of applications of laser face contour as an innovative method of biometric identification (Bratoszewski and Czyżewski 2015). However, at the time of this paper preparation the results of processing of data acquired in this way were not available, yet.

We use also some typical biometric approaches, namely RGB image-based face recognition (as in Section 3) and voice biometry (Section 4). Moreover, a commercially available palm vein scanner was mounted on our experimental biometric stands for collecting additional biometric data.

The main purpose of the research scope presented in this paper is to verify correlation between objective characteristics of biometric samples and samples clusters and subjective assessment of ergonomics, user satisfaction and easiness of use of particular biometric traits.

By applying the Rough Set method, a decision-related sets can be approximated. The following key notions such as satisfied or dissatisfied, proficient or not proficient, reliable or unreliable are of interest. Data quality can be also expressed in terms of reliable or unreliable, stable or unstable data, etc. The rough representation is useful for classifying new cases during the enrolment phase, where the number of samples, stored in this phase, is relatively low. The prospect of repeatedly collecting new sample on each verification attempt should be considered. Therefore, based on initial, limited knowledge modeled from enrollment samples a recommendation should be automatically given, e.g. to reject most unstable trait, to repeat registration, or, in worst cases, to rely only on classical verification methods as an alternative procedure. Therefore, the results of soft computing-based processing of gathered data will serve a very important goal in terms of this project purpose.

While the convergence c for any of the six measures is rescaled by the size of the accumulated cost matrix, the thresholds c_THR could be set to a fixed value. The value was equal to 300 and it was set empirically, providing the best FRR/FAR ratio. The global similarity ratio value is the average value of all the p values.

The speaker identity verification is performed using Gaussian Mixtures (GMM) and Universal Background (UBM) Models (Chen et al. 2012). The Alize framework is used as the speaker recognition back-end (Alize 2017) and mel-frequency cepstral coefficients (MFCC) were employed for speech parametrization. In the first step the Universal Background Model (UBM) was trained based on the recordings prepared in real bank branches environment. Those recordings include speech data recorded by 84 participants in both quiet and noisy environments that were found in real banking outlets environment. Besides the performed recordings the speech material from the MOBIO dataset (McCool et al. 2012) was utilized for the processing in order to increase UBM inner variance.

All speech signals were recorded using a single microphone, with 44 kS/s sampling rate and 16-bit resolution. The MFCCs in 13 cepstral channels were extracted within 10 ms timebase, using window of the size equal to 25 ms. The number of MFCCs used in speaker modelling varies from 10 to 20 in literature (Gupta and Gupta 2013; Mermelstein 1980). This is mainly determined by the speech signal characteristics where most of the information is held in low frequency components of speech spectrum (i.e. formants) what coincides with the highest resolution of the mel filters. In this work the number of coefficients was chosen based on both literature studies and empirical approach and was set to 13. Furthermore, it has been long established that adding dynamic information (Furui 1982) increases the speaker recognition accuracy, therefore, the final acoustic feature vector was formed by combining zero-order MFCCs with delta and delta-delta features, resulting in 39 features in total

At the data acquisition step users were recording their utterances in four trials. First, three of them consisting of 17 seconds of speech were acquired in the users’ enrolment process. The speech model training refers to the creating of the user-specific statistical Gausian Mixture Model (GMM) adapted from the UBM employing the maximum a posteriori criterion (Gauvain and Lee 1994). The fourth speech sample, 7 seconds-long was used for the purpose of users’ identity verification.

Foundations of rough set theory were laid by Polish mathematician Pawlak (1982, 1991), and used with successfully in many domains, including biometric data feature selection (Mazumdar et al. 2010), behavioral patterns analysis (Bazan et al. 2005), discovery of medical knowledge (Tsumoto 2002), dimensionality reduction (Banerjee et al. 2007). The rough set prominent applications involve processing of real-life data, often imprecise, ambiguous and uncertain. Therefore, considering the subjective nature of data collected in the presented experiments, the rough set methodology is regarded adequate as a processing and a data mining tool. The theory is used for the approximation of a set by defining its upper and lower approximation: the first one including objects that may belong to the set, and the latter one including objects that certainly belong to the set. Both approximations are expressed as unions of the so-called atomic sets containing objects that are indiscernible, i.e. they have the same values of attributes (Fig. 3). Two objects x and y can be characterized by a defined set of attributes P ⊆A (P is a subset of all possible attributes in A), and they are related by the indiscernibility relation if: (x,y) ∈IND(P), where IND(P)is the equivalence relation defined as in (12):

$$ \text{IND}(\mathbf{P}) = \{(x,y)\in \mathbf{U}^{2} ~|~ \forall a\in\mathbf{P}, a(x)=a(y)\}, $$

(12)

where a(x)is a value of attribute a of object x. In this work P is a set of features introduced in Section 7, and objects x are particular biometric identities.

Open image in new window

Application of the rough sets theory in decision systems often requires a minimal (the shortest) or the most convenient subset of attributes RED ⊆P resulting in the same quality of approximation as P, called a reduct, therefore introducing the same indiscernibility relations: IND(RED) = IND(P). Numerous algorithms to calculate reducts are available. A greedy heuristic algorithm was applied for this work (Janusz and Stawicki 2012).

Usually, for attributes with continuous values, prior to the reduct calculation a discretization is performed. Maximum discernibility (MD) algorithm is applied, which analyses attribute domain, sorts values present in the training set, takes all midpoints between values and finally returns the midpoint maximizing the number of correctly separated objects of different classes (Bazan et al. 2000; Nguyen 2001). Above procedure is repeated for every attribute.

At the classification phase these rules are applied for every object in the testing set, and then the decision is first predicted, and then compared to the actual one. More information on the rough set theory can be found in the literature (Pawlak 1982; Bazan et al. 2000; Nguyen 2001; Zhong et al. 2001).

Raw biometric signals, such as a voice recording, face image or acceleration and rotation data from the engineered biometric pen, were processed to extract features. From this point biometric samples are described by respective features and they are treated as points in the multidimensional feature space. These points will be characterized employing: 1) their relative distances, 2) average position, i.e. cluster center, and 3) distances of samples to the cluster center. Distances reflect stability of each biometric feature, as well as their changes over time. Appropriate metrics were introduced for each biometric modality: face image, signature, and voice, as is described below.

6.4 Data cluster characteristics based on distances

For every modality, the following data characteristics were collected:

• minimal, mean, and maximal distances between all possible pairs of samples (22–24), reflecting existing similarities and dissimilarities between samples,

• mean value of all samples attribute values, representing the center of the samples cluster in n-dimensional space (25). In case of voice a model based on all samples (C_v)was used instead (26).

• the distance of each sample from the cluster (27–28).

$$ \mathit{dist}_{\min} = \min_{i,j = 1{\ldots} N} \{\|x_{i}, x_{j}\|,~ i\ne j\} $$

(22)

$$ \mathit{dist}_{\mathit{mean}} = 1/(N\cdot(N-1)) \cdot \sum\limits_{i,j = 1{\ldots} N,~ i\ne j} (\|x_{i}, x_{j}\|) $$

(23)

$$ \mathit{dist}_{\max} = \max_{i,j = 1{\ldots} N} \{\|x_{i}, x_{j}\|,~ i\ne j\} $$

(24)

$$ C = \left\{a_{i} = 1/N \cdot \sum\limits_{j = 1{\ldots} N} (a_{i}(x_{j})),~ i = 1{\ldots} M\right\} $$

(25)

$$ C_{v} = \sum\limits_{j = 1{\ldots} K} (w_{i}\, g(x_{N}|\mu_{i},~ {\Sigma}_{i})) $$

(26)

$$ C\mathit{dist}_{\min} = \min_{i = 1{\ldots} N} \{\|C, x_{i}\|\} $$

(27)

$$ C\mathit{dist}_{\max} = \max_{i = 1..N} \{\|C, x_{i}\|\} $$

(28)

where: (x_i,x_j) ∈U², N – number of objects, and M – number of features, w_i are normalized positive scalar weights of the Gaussian mixtures, K is the number of mixtures in the voice model, x_Nis the vector of acoustic features for N objects (recordings), g(x_N | μ_i,Σ_i) are the Gaussian densities with mean vector μ_i and covariance matrix Σ_i (Jiang 2005).

The cluster of all enrolment samples was characterized by (22–28), and then a new cluster of all positive samples, i.e. union of sets from the enrolment and from verification procedure, was created and parameterized alike.

As it was previously stated, distances are calculated in the multidimensional feature space. To explain the purpose of introduced metrics a synthetic 2-dimensional samples and distances are visualized (Fig. 4). It can be observed, that some cluster changes occur as a result of adding new samples. In this case:

• cluster center shifted towards new samples,

• dist_mindecreased as a new sample appeared more similar to existing sample,

• Cdist_mindecreased as new cluster center is closer to existing sample,

• Cdist_maxdecreased as new cluster center was shifted right, better balancing the distribution of samples, and cluster size decreased,

• dist_maxwas not changed as two most dissimilar samples are still in the set.

Open image in new window

Depending on the distribution of enrollment and new samples such changes can be more prominent or rather subtle. The higher the number of samples the less impact new samples have on cluster characteristics, as the cluster based on many samples better estimates real distribution. The important question related to this research is to predict changes of small clusters based on personal subjective characteristics of the user. It is assumed, that unreliable user or user feeling not comfortable with the particular biometric technology will register samples that are not representative during the enrollment, so verification samples may shift the cluster significantly. A reliable user will register representative samples, assuring their repeatability, especially in behavioral biometry traits.

Taking into account uncertainty of subjective data as well as imprecision of signals features the rough set methodology is regarded as an efficient data mining tool, by definition aimed at modeling and processing of decision sets approximations. Finally, the presented work concludes with selection of relevant features, generation of rules for classification of new cases, thus acting as a framework encompassing all phases of data mining, through modeling up to exploitation defined for the application of biometric identity verification.

Other algorithms such as decision trees, and neural networks are not entirely suitable for processing of imprecise and approximate data. Thus in this research the subjective responses were collected by means of questionnaires with discrete answers. Therefore, the rough set theory was chosen as the most appropriate approach.

The goal of the work presented further on is to verify relations between objective characteristics of biometric samples sets and subjective features related to users’ performance, their experience and subjective opinions.

The distribution of answers is presented in the form of statistics below (Fig. 5). 126 persons took part in the pilot exploitation tests, providing their opinions.

Open image in new window

It should be stressed that “0” interpreted as refusal to answer was frequent for almost every question, and it seems that users were not always motivated enough to provide replies. In turn, some answers, although different than “0”, were not reliable, as e.g. user gender expected to be “1” or “2” has also other numerical values. It was expected that biometric pen ergonomics will be critically commented because it differs in bulk from a typical pen, but only one person of 126 reported that it is inconvenient.

Distances for all collected samples are shown in Figs. 6, 7 and 8. The radial plots show distances in the clusters only, but not absolute clusters position. All clusters are centered on the plot, thus in each case Cdist = 0 is situated in the middle of the plot. At each angle around the center a different biometric identity representation is plotted, sorted by an increasing value of Cdist_max. Distances of enrollment samples are shown by black dots, mean of all distances for all identities is shown as a black circle. Distances of new samples are shown by gray dots. It can be observed that in numerous cases new samples are placed significantly further away from the center, thus falling outside the region defined by mean distance, so that generally, thresholds for the similarity measure must not be defined on the basis of enrollment samples only.

Open image in new window

Open image in new window

Open image in new window

Above values allow for an objective characterization of collected data, and for a comparison between enrollment data and validation data (Section 10).

Data processing was performed in R programming environment (Gardener and Beginning 2016) with RoughSets package (Riza et al. 2015). It is a mathematical calculation environment offering data importing, scripted processing, and visualization, extensible by numerous additional libraries and packages.

This process was performed for each biometric trait and for each subjective opinion of clients and bank tellers on this trait to be modeled by the rough set. Results of accuracy determination of subjective opinions on each biometric trait are presented in Fig. 9, whereas the number of reducts used at each step is reported in Table 3 and some selected relations between objective features of voice biometry and subjective user experience metrics for given number of discretization cuts are illustrated in Figs. 10 and 11.

Open image in new window

Table 3

Number of reducts, accuracy and relative size of positive region for given number of used discretization cuts and for defined subjective feature

	1 cut			2 cuts			3 cuts			4 cuts			5 cuts			6 cuts
	reduct	accu-	pos.	reduct	accu-	pos.	reduct	accu-	pos.	reduct	accu-	pos.	reduct	accu-	pos.	reduct	accu-	pos.
	size	racy	reg.	size	racy	reg.	size	racy	reg.	size	racy	reg.	size	racy	reg.	size	racy	reg.
Signature biometry subjective features modelling
uFast	1	43	0	1	48	0	2	59	0.14	1	54	0	1	54	0	1	54	0
uEasyPen	2	57	0	1	59	0	2	64	0.17	2	69	0.23	1	59	0	1	59	0
uReliPen	3	66	0	1	65	0	1	65	0	1	69	0	2	81	0.40	2	84	0.60
uWill	2	46	0	2	50	0.03	2	56	0.01	1	47	0	1	49	0	1	51	0
uPriv	1	50	0	3	62	0.11	1	53	0	2	67	0.17	1	55	0	2	77	0.46
uSecu	2	61	0	2	66	0.04	1	61	0	1	63	0	2	76	0.34	2	74	0.37
uAfra	1	46	0	2	55	0	1	53	0	1	54	0	1	54	0	1	54	0
cFast	1	36	0	1	42	0	1	42	0	1	43	0	1	43	0	2	61	0.29
hardPen	1	99	0.77	1	100	1	1	100	1	1	100	1	1	100	1	1	100	1
cHelp	1	41	0	1	42	0	1	43	0	1	42	0	2	58	0.17	1	47	0
cHardPen	2	67	0	1	70	0	3	80	0.56	1	71	0	1	71	0	1	71	0
cCumberPen	2	83	0.05	3	87	0.28	1	85	0	1	85	0	1	85	0	1	85	0
Most frequent signature features
Feature	No. of		Feature		No. of		Feature		No. of		Feature		No. of		Feature		No. of
	reducts				reducts				reducts				reducts				reducts
sigPosMean	33		sigPosMax		7		sigdmin		4		sigLemax		2		SigLemean		1
SigLedif.mean	17		sigTotle		7		sigCLdri		3		sigdpmean		2		sigCmean		1
sigLemin	15		sigdpmin		5		sigdifmax		2		sigdmax		1
Voice biometry subjective features modelling
uFast	2	55	0	1	56	0	1	56	0	1	61	0	2	74	0.19	1	62	0
uEasyVoi	1	48	0	1	51	0	1	51	0	1	51	0	1	52	0	1	56	0
uReliVoi	1	62	0	1	62	0	1	62	0	1	63	0	1	65	0	1	67	0
uWill	1	42	0	1	47	0	1	52	0	1	52	0	1	53	0	1	53	0
uPriv	2	55	0	1	56	0	1	56	0	1	54	0	1	54	0	1	54	0
uSecu	1	58	0	1	60	0	1	60	0	1	62	0	1	63	0	1	63	0
uAfra	2	49	0	2	53	0.03	1	51	0	1	55	0	1	55	0	1	58	0
cFast	2	42	0	2	49	0.02	1	43	0	1	43	0	1	47	0	1	47	0
hardVoi	2	68	0	1	61	0	1	61	0	2	68	0.33	1	71	0	1	71	0
cHelp	2	41	0	1	45	0	1	43	0	1	49	0	1	51	0	1	46	0
cHardVoi	3	47	0.02	1	45	0	1	45	0	1	47	0	1	47	0	1	47	0
cCumberVoi	1	66	0	2	69	0.13	1	67	0	1	67	0	1	67	0	1	67	0
	1 cut			2 cuts			3 cuts			4 cuts			5 cuts			6 cuts
	reduct	accu-	pos.	reduct	accu-	pos.	reduct	accu-	pos.	reduct	accu-	pos.	reduct	accu-	pos.	reduct	accu-	pos.
	size	racy	reg.	size	racy	reg.	size	racy	reg.	size	racy	reg.	size	racy	reg.	size	racy	reg.
Most frequent voice features
Feature	No. of		Feature		No. of		Feature		No. of		Feature		No. of		Feature		No. of
	reducts				reducts				reducts				reducts				reducts
voidpmin	33		voiPCdmin		13		voidmean		5		voidpmax		3		voiCmean		1
voiCmin	17		voidmin		7		voidmax		4		voiCmax		2
Face biometry subjective features modelling
uEasyFac	1	53	0	1	53	0	2	68	0.03	1	59	0	1	58	0	1	59	0
uReliFac	1	46	0	1	50	0	2	67	0.05	2	76	0.28	2	78	0.34	2	80	0.48
uWill	1	45	0	2	54	0	2	60	0.01	1	47	0	1	47	0	1	48	0
uPriv	1	50	0	1	50	0	2	62	0.04	3	84	0.67	3	90	0.78	2	72	0.39
uSecu	2	60	0	1	61	0	1	61	0	1	62	0	1	62	0	1	63	0
uAfra	2	49	0	1	47	0	1	52	0	1	52	0	1	55	0	1	55	0
cFast	1	39	0	1	44	0	1	44	0	1	44	0	1	44	0	1	44	0
hardFac	1	87	0	3	86	0.69	4	84	0.97	3	90	0.87	3	87	0.95	2	85	0.78
cHelp	3	48	0.03	3	55	0.13	1	41	0	1	43	0	1	45	0	1	46	0
cHardFac	2	40	0	1	40	0	1	42	0	1	44	0	1	43	0	1	46	0
cCumberFac	1	68	0	1	68	0	1	69	0	1	70	0	1	72	0	1	72	0
Most frequent face features
Feature	No. of		Feature		No. of		Feature		No. of		Feature		No. of		Feature		No. of
	reducts				reducts				reducts				reducts				reducts
facdifmax	13		facCLdri		12		facpdmean		9		facpdmin		3		facCmax		1
facPCdmean	15		facCmin		10		facCmean		7		facpdmax		3
facdmax	15		facdmin		9		facPCdmax		4		facPCdmin		3

Open image in new window

Open image in new window

In numerous cases reducts with only 1 or 2 attributes are derived, because other attributes would not introduce further improvement in classes separation. Each classifier maintains up to (c + 1) ⋅|RED| rules (c is the number of discretization cuts, and |RED|is the number of attributes in the reduct). If the number of rules is high (e.g. hardFac feature for 3 cuts has 4 attributes, thus 16 rules are derived), then the modeled knowledge is sufficient to express differences between objects, and the relative size of positive region is close to 1.

As is seen from the results of the study, each enrolment cluster reflects quality and stability of samples during the limited period of time. Wide clusters are expected to be related to inability of the user to provide biometric sample in a repeated manner, due to e.g. biometric pen ergonomics differing from a typical pen, voice issues related to stress or noisy background conditions, or face capture issues due to unstable position in front of a camera etc.

In turn, all positive samples cluster includes enrollment samples as well as all other samples collected after arbitrary time intervals. The cluster width is related to differences between old samples and the newer ones. The cluster center is expected to drift over time to account personal characteristics changes, such as voice harshness due to aging, signature improvements being a result of repeated usage of the biometric pen, face image changes due to aging or facial hair, make-up, etc.

The cluster drift would be more prominent in case of rejection of the oldest samples. The rejection time limit definition should be determined on the basis of more prolonged studies employing the group of users, in order to enable assessing the problem of aging and any time-related changes of biometric traits.

The presented method can be also applied for all biometric features fused together and analyzed in unison. This can potentially reveal cross-modalities relations, such as both behavioral traits (speech and signature) features revealing the dependency on user emotions, or feeling of convenience, or familiarity with the technology.