Modeling temporal aspects of sensor data for MongoDB NoSQL database

The emergence of Web 2.0 systems, the Internet of Things (IoT) and millions of users have played a vital role to build a global society, which generates volumes of data. At the same time, this data tsunami has threatened to overwhelm and disrupt the data centers [1]. Due to this constant data growth the information storage, support and maintenance have become a challenge while using the traditional data management approaches, such as structural relational databases. To support the data storage demands of new generation applications, the distributed storage mechanisms are becoming the de-facto storage method [2]. Scaling can be achieved in two ways, vertical or horizontal, where the former means adding up resources to a single node, whereas in the latter case we add more nodes to the system [3]. For the problems that have arisen due to data proliferation, the RDBMS fail to scale the applications horizontally according to the incoming data traffic [2]; because they require data replication on multiple nodes, so they are not flexible to allow data and read/write operations distributed over many servers. So we need to find systems that would be able to manage big volumes of data.

This flood of data passes challenges not only due to its sheer size but also due to the data types, hence demands more robust mechanisms to tackle different data formats. The Web, e-science solutions, sensor laboratories and industrial sector produce in abundance both structural, semi-structural and non-structural data [4, 5, 6]. This is not a new problem, and can be traced back to the history of object-relational databases, under the name of Object-Relational Impedance Mismatch [7]. This mismatch is natural, when we try to model an object into a fixed relational structure. Similarly, the digital information with different structures, such as natural text, PDF, HTML and embedded systems data, is not simple enough to capture as entities and relationships [8]. Even if we manage to do this, it will not be easy to change afterwards, hence such mechanisms are rigid for schema alteration because they demand pre-schema definition techniques. Several new generation systems do not like to fix their data structure to a single schema; rather they want their schema to evolve in parallel to an entity data type’s adaptation, hence they want flexibility [9, 10].

Besides the data abundance and different formats, the rapid flow of data has also attracted the researchers to find mechanisms to manage the data in motion. Typically this is to consider that, how quickly the data is produced and stored, and its associated rates of retrieval and processing. This idea of data in motion is evoking far more interest than the conventional definitions, and needs a new way of thinking to solve the problem [11]. This is not associated only with the growth rate at the data acquisition end, but also data-flow rate during transmission; as well the speed at which data is processed and stored in the data repositories. Any way, we are aware of the fact that today’s enterprises have to deal with petabytes instead of terabytes; and the increase in smart object technology alongside the streaming information has led the constant flow of data at a pace that has threatened the traditional data management systems [11]. RDBMS use two-dimensional tables to represent data and use multi-join transactional queries for the database consistency. Although they are mature and still useful for many applications, but processing of volumes of data using multi-joins is prone to performance issues [12, 13]. This problem is evident when extensive data processing is required to find hidden useful information in huge data volumes; but such data mining techniques are not in our current focus [14, 15, 16, 17], as we limit our discussion to NoSQL temporal modeling and schema based data integration.

Above discussed problems of prolific, multi-structured heterogeneous data in flow urge the researchers to conduct research to find alternate data management mechanisms, hence NoSQL data management systems have appeared and are now becoming a standard to cope with big data problems [11, 18]. Such new data management systems are being used by many companies, such as Google, Amazon etc. The four primary categories of their data model are: (i) key-value stores, (ii) column-oriented, (iii) document, and (iv) graph databases [19, 20]. For rationality, sanity and demonstrating the storage structure, the researchers follow the database schema techniques without losing the advantages of schema flexibility provided by NoSQL databases. Such schema modeling strategies in NoSQL databases are quite different from the relational databases. Collections, normalization and document embedding are few variants to consider during building schema models because they affect the performance and storage effectively because such databases grow very quickly.

While dealing with real-time data, in continuous or sliced snapshot data streams, the data items possess observations which are ordered over time [21]. During previous years, research efforts had been conducted to capture temporal aspects in the form of data models and query languages [22, 23]. But mostly those efforts were for relational or object-oriented models [23], and can be considered as a conceptual background to solve advanced data management challenges [24]. The emerging applications, such as sensor data [25], Internet traffic [26], financial tickers [27, 28] and e-commerce [29], produce large volumes of timestamped data continuously in real-time [30, 31]. The current methods of centralized or distributed storage with static data impose constraints in addressing the real-time requirements [32], as they inflict pre-defined time convictions unless timestamped attributes are explicitly added [31]. They have limited features to support the latest data stream challenges and demand research to augment the existing technologies [31, 33].

In remote healthcare long term monitoring operations, based on Body Area Networks (BAN), demand low energy consumption due to limited memory, processing and battery resources [34]. These systems also demand communication and data interoperability among sensor devices [35]. Recently a propriety protocol ANT+ provides these low energy consumption features; and strengthens the goals of IoT through the interoperability of devices based on Machine-to-Machine (M2M) mechanisms, which employs use case specific device profile standards [34, 36]. Device interoperability, low energy and miniaturisation features allow the building of large ecosystems, hence enable millions of vendor devices to get integrated and interoperated. IoT ecosystems want general storage mechanisms having structural flexibility to accept different data formats arriving from millions of sensory objects [37]. The non-relational or NoSQL databases are schema-free [2]; and allow storage of different data formats without prior structural declarations [34, 37]. However for the storage we need to investigate the NoSQL models to design and develop [8, 22]; besides flexibly preserving the big data timestamped characteristics for the massive real-time data flow during acquisition processes [24]. Although all NoSQL databases have unique advantages, but document-oriented storage, as MongoDB provides, is considered robust for handling multiple structural information to support IoT goals [38]. This rejects the relational structural storage and favours Java Script Object Notations (JSON) documents to support dynamic schemas; hence provide integration to different data types besides scalability features [39, 40].

This article presents a general approach to model temporal aspects of ANT+ sensor data. The authors develop a prototype for the MongoDB NoSQL real-time platform and discuss the temporal data modeling challenges and decisions. An algorithm is presented which integrates JSON data as hierarchical documents and evolves the proposed schema without loosing flexibility and scalability.

This article is organized as follows. "Data stream and data stream management systems (DSMS)" is about time series data. Different NoSQL databases are discussed in detail in "Limitations of RDBMS". It is followed by a subsection discussing MongoDB as a well-known document oriented database. "Big data management frameworks", discusses the different techniques to model time series data using MongoDB. This follows a middleware description explaining how to store data in the MongoDB. "Modeling aspects" and "Temporal modeling for an ANT+ sensor use case" give future work and a short summary respectively.

A time series is a sequence of numerical measurements from observations collected at regular durations of time. Such successive times can either be continuous or discrete time periods. These sequence of values at particular intervals are common in situations, such as weekly interest rates, stock prices, monthly price indices, yearly sales figures, weather reports, patient’s medical reports and so forth.

Healthcare monitoring systems measure physiological and biological body parameters, using BAN, of a patient’s body in real-time. Because timely information is an important factor to detect immediate situations and to improve decision making processes, based on a patient’s medical history, so considering temporal aspects are vital. Such sequence of values represent the history of an operational context and is helpful in a number of use cases where history or order is required during the analysis. This sequences of data flows in streams of different speeds and also needs proper management.

A big data management framework means the organization of the information according to the principles and practices that would yield high schema flexibility, scalability and processing of the huge volumes of data, but for which traditional RDBMSs are not well suited and becomes impractical. Therefore, there is a need to devise new data models and technologies that can handle such big data. Recent research efforts have shown that big data management frameworks can be classified into three layers that consist of file systems, database technology, and programming models [19]. However in this article we shall focus upon database technologies only in context of the healthcare domain with real-time and temporal perspective.

NoSQL also be interpreted as the abbreviation of “NOT ONLY SQL” or “no SQL at all” [45], whereas this was first used by Carlo Strozzi in 1998 for his RDBMS that did not offer an SQL interface [47]. NoSQL databases are often used for storing of the big data in non-relational and distributed manner, and its concurrency model is weaker than the ACID transactions in relational SQL-like database systems [14]. This is because NoSQL systems are ACID non-compliant by design and the complexity involves in enforcing ACID properties does not exist for most of them [10]. For example some of the ACID compliant NOSQL databases are: Redis [48], Aerospike [49] and Voldemort [50] as key-value stores [51]; where as Neo4jDB [52] and Sparksee are as graph-based data stores [8, 51]. In contrast to this MongoDB is not ACID compliant document-oriented database [8].

NoSQL database categories

Based on the differences in the respective data models, NoSQL databases can be organized into following basic categories as: key-value stores, document databases, column-oriented databases and graph databases [10, 14, 19, 20].

Table 1

Key-value stores

Name	Data model	Scalability	Description	Who uses it
Memcached from http://www.bradfitz.com [55]	Set of key-value in associative array	Auto sharding, no replication and persistency	In-memory cache systems, no disk persistence (MemcacheDB give persistent storage [56]); file system storage, ACID	LiveJournal, Wikipedia, Flickr, Bebo, Craigslist
Aerospike from http://www.aerospike.com [49]	Associate keys with records (i.e rows); namespaces (for dataset) divide into sets (i.e tables); key index records	Auto partition, synch. replication	In-memory very fast database with disk persistence. ACID with relax options	AppNexus, Kayak, blueKai, Yashi, Chango
Cassandra from facebook [57]	Hybrid of key-value and column-oriented models; Cassandra query language: SQL like model	Auto partition, synch. and asynch. replication	In-memory database with disk persistence; highly scalable	CERN, Comcast, eBay, Netflix, GitHub
Redis from S. Sanfilippo [48]	Het of (key, value); complex types (string, binary, list, set, sorted set, hashes, arrays); key can be any binary e.g. JPEG	Auto partitioning, replication, persistent levels	Most popular in-memory with disk persistence; ACID; MapReduce through Jedis [58], \({r}^{3}\) [59]	Twitter, GitHub, Flickr, StackOverflow
Riak [60] from Basho Technologies	Data types (flags, counter, sets, registers, maps, hyperlog)	Sharding, replication, master-less [61], backup and recovery	High available; Dynamo base [62]; in-memory with disk persistence; relax consistency; MapReduce, REST-full; enterprise and cloud versions; Riak KV base Riak TimeSeries [63]	AT and T, Comcast GitHub, UKHealth, weather channel
Voldemort from LinkedIn [50]	Complex key-value compound objects (e.g. lists/maps); supported queries: get, put and delete; no complex query filters; simple api for persistence [64]; schema-evolution	Auto data partitioning and replication, versioning	In-memory with disk persistence, big fault-tolerant hash table; no ACID, pluggable serialization (e.g. avro, java) and storage engine (concurrentHashMap, mysql, BDB JE)	LinkedIn, Gilt
DynamoDB from Amazon [65]	Dynamo [62] based to support both document and key-value models [65]; secondary indexes; DynamoDB Titan: integratable graph database	Replication, partitioning, highly available, versioning	Popular, two of the four ACID properties: consistency and durability; elastic MapReduce for Hadoop; AWS SDK to store JSON	Amazon, BMW, duolingo, lyft, redfin, adroll

Key-value stores

These are systems that store values against the index keys, as key-value pairs. The keys are unique to identify and request the data values from the collections. Such databases has emerged recently and are influenced heavily by Amazon’s Dynamo key-value store database, where data is distributed and replicated across multiple servers [62]. The values in such databases are schema flexible and can be simple text strings or more complex structures like arrays. The simplicity of its data model makes the retrieval of information very fast, therefore supports the big data real-time processing along the scalability, reliability and highly available characteristics. Some of the key-value databases store data ordered on the keys, such as Memcached [55] or Berkeley DB [66]; while others do not, such as Voldemort [50] etc. Whereas some keep entire data in memory, such as Aerospike [49], Redis [48]; others use it after writing it to the disk permanently (like Aerospike, MemcacheDB [56] etc.) with the trade-off replying to the queries in real-time. The scalability, durability and flexibility depends upon different mechanisms like partitioning, replication, object versioning, schema evolution [19]. Sharding, also known as partitioning, is the splitting of the data based upon the keys; whereas the replication, also known as mirroring, is the copying of the data to the different nodes.

Amazon’s Dynamo and Voldemort [50], which are used by Linkedin, apply this data model successfully. Other databases that use this model of data category are such as: Redis [48], Tokyo Cabinet [67] and Tokyo Tyrant [68], Memcached [55] and MemcacheDB [56], Basho Riak [60], Berkeley DB [66] and Scalaris [69]. Whereas Cassandra is a hybrid of key-value and column-oriented database models [57]. Table 1 summarizes the characteristics of some of the Key-value stores.

Column-oriented databases

Relational databases have their focus on rows in which they store the instances or the records and return rows or instances of an entity against a data retrieval query. Such rows posses unique keys against each instance for locating the information. Whereas column-oriented databases store their data as columns instead of the rows and use index based unique keys over the columns for the data retrieval. This supports attribute level access rather than the tuple-level access pattern.

Only query relevant necessary columns are required to be loaded, so this reduce the I/O cost significantly [70]. These are good for read-intensive applications, as they only allow relevant data reads because each column contains contiguous similar values; so calculating aggregate values will also be very fast. More columns are easily addable and a column may be further restructured called super-column, where it contains nested (sub)columns (e.g., in Cassandra) [14]. Super columns are key-value pairs, where the values are columns. Columns and super-columns are both tuples with a name and value. The key difference is that a standard column’s value is a string, whereas a super-column’s value is a map of columns. Super-columns are sorted associative array of columns [71].

Table 2

Column-oriented databases

Name	Data model	Scalability	Description	Who uses it
Cassandra from Facebook, Apache [57]	Multi-dimensional column family is a set of rows; Partition (single or more row) key: identify a partition; Row key: identify row in column family	Partitioning, replication, availability	Multi-master no point of failure; in-memory with disk persistence; masterless; query method: CQL anf Thrift; MapReduce; secondary indexing; eventual consistency	CERN, Comcast, GitHub, GoDaddy, Hulu,eBay Netflix
HBase from Apache [72]	Tables have rows and columns; rows: row key and one or more columns; column: consist of column family	Auto sharding; asynchronous replication; availability	BigTable based; Hadoop Distributed File System (HDFS); MapReduce; consistent read/writes; failover support; Thrift anf REST-ful; ZooKeeper	Adobe, Kakao, Facebook, Flurry, LinkedIn, Netflix, Sears
Druid from http://druid.io	Columns are one of three types: a timestamp, a dimension, or a measure. Nested dimensions	Low latency; replication; sharding	Highly optimized for scans and aggregates ; MapReduce; fault-tolerant; ZooKeeper; index structures, not ACID	Alibaba, Cisco, eBay Netflix, Paypal, Yahoo
Accumulo from Apache [73]	Keys-values both byte arrays, timestamp as long; adds new key element of column visibility; sorts keys by element, secondary indexes	Sharding, replication, persistence, fault tolerant	BigTable based Java technology, top of Hadoop, ZooKeeper and Thrift; Map-Reduce; Zookeeper (multi-master) locks for consistency; cell-level access	US National Security Agency (NSA)

Google’s Bigtable, which played the inspirational role for the column databases [74], is a compressed, high performance, scalable, sparse, distributed multi-dimensional database built over a number of technologies, such as Google File System (GFS) [75], a cluster management system, SSTable file format and Chubby [76]. This provides indexes over rows, columns, as well as a third timestamp dimension. Bigtable is designed to scale across thousands of system nodes and allows to add more nodes easily through automatic configuration.

This was the first most popular column oriented database of its type however latter many companies introduced some other variants of it. For example Facebook’s Cassandra [77] integrates the distributed system technologies of Dynamo and the data model from Bigtable. It distributes multi-dimensional structures across different nodes based upon four dimensions: rows, column families, columns, and super columns. Cassandra was open sourced in 2008, and then HBase [72] and Hypertable [78], based upon a proprietary Bigtable technology, have emerged to implement similar open source data models. Table 2 provides the description about some column-oriented databases in a categorical format.

Graph databases

Graph databases, as a category of NoSQL technologies, represent data as a network of nodes connected with edges and are having properties of key-value pairs. Working on relationships, detecting patterns and finding paths are the best applications to be solved by representing them as graphs. Neo4j [52], Allegro Graph [79], ArangoDB [80] and OrientDB [81] are few examples of such systems, and are described along their characteristics in a categorical format in Table 3. Neo4j is the most popular open source, embedded, fully transactional with ACID characteristics graph-based database. This is schema flexible to store data as a network of nodes, edges and their attributes. This also supports custom data types with its Java persistence engine. Neo4j does not support graph sharding on different nodes, rather it supports in memory cache sharding [52, 82]. The reason having that, the mathematical problem of optimally partitioning a graph across a set of servers is near-impossible (NP complete) to do for large graphs [82]. Whereas AllegroGraph is a Resource Description Framework (RDF) [83] triple store for linked data and widely used by different organizations, such as Stanford University, IBM,Ford, AT&T, Siemens, NASA and United States Census department.

Table 3

Graph databases

Name	Data model	Scalability	Description	Who uses it
neo4j from Neo Technology [52]	Flexible network structure of nodes; data stored in: edges, nodes, or attributes; neo4django: an Object Graph Mapper [84]; custom data types	No direct sharding but cache [82], no replication and persistency	Most popular high performance [85], ACID, monitoring:Neo4j Metrics; query methods: Cypher, SparQL, nativeJavaAPI, JRuby	42talents, ActiveState, Cisco Securus, Apptium, BISTel [52]
AllegroGraph from http://franz.com	Triplestore, resource description framework (RDF) and graph database	Data replication and synchronization; Partitioning with Federation	Linked data format; brings semantic Web to Twitter; Common Lisp: dialect of Lisp; eventual consistency; ACID	Stanford, IBM, Ford, Novartis, AT and T, Siemens, NASA, US Census
ArangoDB from ArangoDB GmbH [80]	Native multi-data models: key/value, document, and graph data to be stored together and queried with a common language [86]	Synchronous replication, tripple store sharding	Most popular having open source license; ACID-compliant for the master; eventualy consistent [87]; annotation query language (AQL) for RDF	DemonWare, Douglas, Craneware, ictual, mobility, egress
OrientDB from OrientDB Ltd [81]	Multi-data models: graph and document database; custom data types	Multi-master replication; supports sharding	Highly available; SQL using pattern matching to support MapReduce, eventual consistency; ACID; Schema-less, Schema mix	Progress, UltrDNS proteus, Enel Flux Gtech, NIH

Table 4

Document-oriented databases

Name	Data model	Scalability	Description	Who uses it
MongoDB from http://10gen.com, MongoDB Inc. [38]	JSON-like hierarchical documents with or without schemas, object mapping, BSON	Sharding, replication and persistency	Most popular JSON document store, ACID, MapReduce, primary secondary indexing, eventual consistency, RESTful	Expedia, Bosh, MetLife, Facebook, comcast, sprinklr
CouchDB from Apache [88]	Native JSON-document store; types: strings, numbers, dates, ordered lists and associative arrays	Multi-master replication,	JavaScript as query language using MapReduce, and HTTP for an API, multi-version concurrency, MapReduce, ACID, eventual consistency	meebo, AirFi Sophos, BBC, npm CANAL+
Couchbase from Couchbase, Inc. [89]	Multi-model: key/value store, document-store; JSON documents	Sharding, master-master and master-slave replication	CouchDB based with Memcached-compatible interface, eventual consistency, limited ACID, eventual consistency, RESTful HTTP API	Informatica, Joyent, intel, Wipro, Google, Simba
RethinkDB [90]	JSON documents with dynamic schemas	Sharding, master-slave replication	Push real-time data; RethinkDB Query Language (ReQL);Hadoop-style MapReduce; primary&secondary indexes; Not ACID	Jive SW, Mediafly, Pristine Platzi, CMUNE, Wise.io
Cloudant from IBM, Apache [91]	JSON based flexible documents	Sharding, master-master & master-slave replication	CouchDB based; primary and secondry indexes; MapReduce; Eventual Consistency; RESTful HTTP/JSON API	Samsung, IBM, Expedia, DHL, Microsoft, Pearson

Document databases

These are the most general models, which use use JSON (JavaScript Object Notation) or BSON (Binary JSON) format to represent and store the data structures as documents for the data management. Document stores provide schema flexibility by allowing arbitrarily complex documents, i.e. sub-documents within document or sub-documents; and documents as lists. A database comprises one or more collections, where each collection is a named group of documents. A document can be a simple or complex value, a set of attribute-value pairs, which can comprise simple values, lists, and even nested sub documents. Documents are schema-flexible, as one can alter the schema at the run time hence providing flexibility to the programmers to save an object instances in different formats, thus supporting polymorphism at the database level [92]. Figure 1 illustrates two collections of documents for both students and course within an academic management System. It is evident that a collection can have different formats of documents in JSON format and they have hierarchies among themselves, such as courses has an attribute books which contains a list of sub-documents of different formats.

Open image in new window

These databases store and manage volumes of collections of textual documents (e.g. emails, web pages, text file books), semi-structure, as well as no structure and de-normalized data; that would require extensive usage of null values as in RDBMS [93]. Unlike key-value stores, the document databases support secondary indexes on sub-documents to allow fast searching. They allow horizontal scaling of the data over multiple servers called shards. MongoDB [38], CouchDB [88], Couchbase [89], ReThinkDB [90], and Cloudant [91] are some of the most popular document-oriented databases, as shown in the Table 4 in a categorical format along their different characteristics. Among these MongoDB is the most popular one due to its efficiency, in memory processing and complex data type features [94]. The other databases such as Couchbase, ReThinkDB, Cloudant and CouchDB do not offer in-memory processing features; although the former three offer a list of data types. MongoDB query languages more like the SQL of RDBMS, so is easy to use for the programmers. MongoDB is good for the dynamic queries, which the other document-oriented databases lack, such as CouchDB or Couchbase [95]. Besides this there are different object relational mapping middlewares available [96], to define out of the box multiple schemas depending upon the application requirements. The object nature of MongoDB documents makes this mapping even more fluid and fast, such as while using Mongoose [96] or Morphia [97].

Document databases: MongoDB

MongoDB, created by 10gen in 2007, is a document oriented database for today’s applications which are not possible to develop using the traditional relational databases [98]. It is an IoT database which instead of tables (as in RDBMS) provides one or more collection(s) as main storage components consisted upon similar or different JSON or BSON based documents or sub documents. Documents that tend to share some of the similar structure are organized as collections, which can be created at any time, without predefinitions. A document can simply be considered as a row or instance of an entity in RDBMS, but the difference is that, in MongoDB we can have instances within instances or documents with in documents, even lists or arrays of documents. The types for the attributes of a document can be of any basic data type, such as numbers, strings, dates, arrays or even a sub-document.

Open image in new window

MongoDB provides unique multiple storage engines within a single deployment and automatically manages the movement of data between storage engine technologies using native replication. MongoDB 3.2 consists of four efficient storage engines as shown in Fig. 2, all of which can coexist within a single MongoDB replica set [99]. The default WiredTiger storage engine provides concurrency control and native compression with best storage and performance efficiency. MongoDB allows both the combinations of in-memory engine for ultra low-latency operations with a disk-based engine for persistence altogether.

It allows to build large-scale, highly available, robust systems and enables different sensors and applications to store their data in a schema flexible manner. There is no database blockage, such as we encounter during alter table commands in RDBMS during schema migrations. However in rare cases, such as during the write-intensive scenarios in master-slave nature of MongoDB there may be blockage at the document level or bottleneck to the system if sharding is not used, but these cases are avoidable. MongoDB enables horizontal scalability because table joins are not as important as they are in the traditional RDBMS. MongoDB provides auto-sharding in which more replica server nodes can easily be added to a system. It is a very fast database and provides indexes not only on the primary attributes rather also on the secondary attributes within the sub-documents even. For the cross comparison analysis between different collections we have different technologies, such as aggregation framework [100], MapReduce [101, 102], Hadoop [14, 19] etc. These processing techniques will be targeted in future and are not currently focused in this paper. Next we discuss the data modeling methodologies, which is followed by a data model description we used for the storage of the real-time temporal data of ANT+ sensors in the healthcare domain.

The representation of the concepts of data structures required by a database is called a data model. A data model determines the logical structure of a database and basically decides in which manner the data can be stored, organized, and manipulated. Such data structures decide the data objects and their relationships and the semantic data rules they govern. In software engineering data modeling is a process to create data models for an information system. There are mainly three types of data model instances, i.e. conceptual, logical and physical [103]. Conceptual data modeling defines the semantics of an application domain, to elaborate the scope of the model, in order to identify the various entities and relationships needed in an application. Logical data model is similar to conceptual, but with more details, which defines the structure of a domain of information. It is also called as a schema model, as it describes the objects with their properties and relationships to determine the structure of the data, whereas the physical data model concerns the physical storage of the data.

Open image in new window

Although NoSQL approaches provide schema flexibility to the developers which allow them to store different formats of same entity’s instances in a single storage collection, but for the rationality, sanity and demonstrating the storage structure they still have to follow some sort of format logically, at least at the application level and then to store the data in the selected format(s). In this way one can define as many schema formats for a single entity and the same collection will allow the storage to all of them, given that a logical rationality has been maintained through out the definitions, as shown in Fig. 3. Applications treat each schema differently based upon different logics using advanced programming, such as object oriented principles in the form of polymorphism, classes inheritance or object arrays etc. Hence developers eagerly define and model the schemas that yield robust characteristics for their problem domain in general. The initial research regarding NoSQL database modeling and insights is provided in [104, 105]. Bugiotti et al. deal with NoSQL modeling approaches in general [106]. How to have NoSQL related approaches inside the relational databases is discussed in [107]. For the initial thoughts about NoSQL modeling and its comparison between traditional relational models and NoSQL models is provided in [102]. A study about normalization and de normalization during the data modeling is done in [108].

Note that because the document-oriented data models allow sub-documents embedding, so hierarchical principles are their core features; and this create opportunity for the sub-entities to get embedded and to play the role. A collection can refer to one or more models, as shown in Fig. 3. A schema can have a single model representing an entity or can have multiple models representing more entities (may be hierarchical). For example in Fig. 1, we model two course documents for a course entity. The document titled as “Programming” is quite simple, whereas the document titled as “data structures” has a tree structure where a sub-entity books is shown at the second level. Similarly, two structures are shown for the student collection, where the first with name “Scott” is simple and the second with the name “Neo” contains a hierarchy. Therefore the collections having different schemas are actually having different format logics and must be treated differently.

How good is a schema design?

To be rational about a schema and speaking in context of the RDBMS, during the mapping process, we still require some formal efforts to analyse why a relational schema, with a grouping of attributes, may be better than another. This goodness or measurement of the quality of a structural or relational schema design follows the procedures of checking data redundancy and data dependency features extensively [110]. For a technical application domain, we should investigate and compare the available data modeling approaches to chose the best.

In general, we should consider all the requirements and constraints related to a problem domain, such as temporal, streaming, distributed aspects, multiple messages formats and their attributes. Modeling for the schema flexibility this investigation needs intensive care; to allow the storage of multiple structural schemas. This way one main goal of big data can be achieved i.e. variety. It is to note that schema flexibility, such as provided by MongoDB, can permit all vendor devices to store data simultaneously with the time-stamp regardless of different message formats. In context of the IoT perspective, it supports storage independence of data related to any “thing(s)”.

Goals of normalization

Katsov in his online guide [104], discusses NoSQL data modeling techniques for all the NoSQL management frameworks in general. He says that denormalization and aggregation are the main fundamental design drivers, beside thinking about how the application would address the end user queries. RDBMS do normalization because of three goals, i.e (i) To free the database of modification anomalies, i.e. to update only one row in one table against a wrong instance entry, (ii) to minimize the redesign of the database, and (iii) to avoid biasness towards any particular access patterns. For the third point MongoDB do not care at all, therefore with MongoDB we should not worried about being biased to a particular access pattern.

One of the big advantage of RDBMS is that they keep the data relational, normalized and enforce the foreign Keys for the data consistency. But in MongoDB it is only upto the programmer to keep such a consistency, therefore MongoDB do not provide such a guarantee. Whereas on the other side, MongoDB allow the embedding of the entire document and in this case do not have to care about even about the foreign keys at all. With normalization RDBMS usually perform transactions based upon ACID principles, which MongoDB does not have to care about. However it allows atomic operations at a document level.

Normalization or embedding

Normalization was first introduced by E.F. Codd in the field of rational databases and is explained in many of the database related text, such as [110, 111]. In RDBMS the traditional practices are to properly normalize the database because it increases the data integrity and efficiency of the space utilization [103]. In denormalization we reject the splitting of the physical tables, and keep the data together in a single database table which have frequent access, to reduce the query processing time, as in this way we can avoid joins as well and the disk scanning at different location to answer the queries. Following can be the database schemas in normalized forms; for the de-normalized student and course collections depicted in Fig. 1.

Open image in new window

Kanade et al. [108] did a study for NoSQL databases with both normalized and denormalized forms using a similar dataset, and have found that the embedded MongoDB data model provides a much better efficiency as compared to a normalized model. Therefore, Database designers have confronted that tight normalization degrades the system performance, hence they recommend to denormalize in many cases. Denormalization supports the data read performance, as the spinning disks have a very high latency; therefore the idea is to keep the data close and together to help avoid disk read overheads and to decrease the latency. Therefore, denormalization is a key feature of document databases to support efficient data retrieval and to optimize the data storage capacity by allowing dataset hierarchies.

Time series data modeling in MongoDB

As streaming data contains timestamp values within each sequenced message element. MongoDB is a great fit to handle timestamped data and many organizations are using it [128]. Some proprietary services are also available and popular in the community, like MongoDB Management Service (MMS) which models and displays the time-series data as a dashboard [129]. This facilitates a developer to develop visualization and alerts on different datasets using metrics.

Schema design for time series: a case study

It is beneficial to have a schema mapping out of the box; thus facilitating the programmers in developing scalable distributed systems more rapidly and flexibly. One can design as many schemas as possible for MongoDB as NOSQL databases are flexible for this purpose. But to be rational there is a tradeoff between different schemas, and as we learned that the level of normalization or de-normalization is important. This section discusses an approach described in a blog, which models time-series data for MongoDB with respect to different levels of normalization [128, 130]. Before explaining the design approach for a NoSQL database; first let us observe, how an RDBMS handles time-series data. This is quite straightforward, i.e. by storing each event as a row within a Table. For example, a monitoring system stores a temperature sensor measurement in relational format, as shown in Table 5: But, this approach will save a single document in each row for MongoDB.

Table 5

TimeSeries data storage in a traditional RDBMS

Timestamp	Temperature
2016-04-10T22:04:23.000Z	24.1
2016-04-10T22:04:24.000Z	24.1
2016-04-10T22:04:25.000Z	24.0

Open image in new window

This doc-per-event approach does not take advantage from the document embedding, and there will be lot of latency in reading the data; for example around 3600 reads for 1 min data, if the sensor transmits one message per second [130]. An alternative is, to have a single document-per-minute and to save the temperature value against each second. Such an approach, as shown bellow, is better and more optimized in the context of the storage as well as data retrieval.

Open image in new window

This model contains sub-documents as values, which are embedded in the main document. To store one value per second (against a minute), we can simply represent each second as separate fields between 0 and 59, as shown in the above code listing. Such kind of approaches affect the efficiency of systems, in context of the number of inserts and updates. For example, for one hour read 60 total documents need to scan; and some systems copy the entire data to a new location against an update, which is not robust when the data is in large quantity. If the database also has to perform indexing after the copying process, it will affect adversely. A feature of MongoDB is that it can manage in place field-level updates as long as the size of the document does not grow more than a certain limit (i.e, 16 megabytes), which may also contain sub-documents [131]. However GridFS API of MongoDB provides flexibility to store a document more than the maximum size [132]. Conclusively, because field level updates are more efficient, so the document-embedding is better than the relational design. Writes will be faster as: for updates (one per second) than for inserts (one per minute) [128]. Because instead of writing a full document at a new place we will actually request a much smaller update that can be modeled as described below for the temperature collection:

Open image in new window

A more compact approach is to store per second data at hourly level, i.e. doc-per-hour. It has two variants: (i) by second, and (ii) by minute. In the former case seconds are stored from 0 to 3599 for an hour. This approach cause extra workload during update operations, because to update the last second 3599 steps are required. To avoid document movements pre allocation of the structure space is recommended. However the later case stores per-second data at the hourly level but with nesting documents for each minute. This approach is also update driven, but to update the last second it requires 59 + 59 steps. The following MongoDB query depicts the later model, and it is followed by an update query to modify the last second.

Open image in new window

and, to update the last second:

Open image in new window

In the next section we select a real time use case regarding sensor network data and shall model the temporal properties of the message generated from different sensor devices.

To preserve the NoSQL schema flexible goals, i.e, data variety; the model should reside outside the database and should not be tied with it; and should provide a mirror to the programming language objects to get mapped seamlessly for persistent storage. The less energy resourced sensor devices do not communicate directly with the web servers rather they use a gateway, display device, PC, service or a running application. In our system, a monitoring Windows service transmits the ANT+ sensor data in near real-time over the WebSockets protocol, where as the NodeJS [96] context data server accepts, processes and stores the JSON messages in MongoDB database according to the defined schema models. The context data server is a NodeJS based JavaScript engine, which provides a WebSockets based interface for the communication middleware layer and data processing. The server uses the Mongoose based schemas for the direct document mapping to persistently store the JSON messages. Figure 4 depicts a general view of the real-time distributed system architecture for the prototype; whereas the complete architecture components are explained in [133]. For temporal storage, data integration and schema evolution the server executes relevant ANT+ device profile related Mongoose schema based storage algorithms against the specific event notifications.

Open image in new window

To illustrate the approach through a prototype, we deal with three ANT+ wearable sensor categories, such as heart rate, foot pod and temperature to monitor a patient’s body parameters [34, 133]. Next we shall rationally observe the temporal modeling perspectives of the ANT+ sensor data and will learn the Mongoose schema based algorithm to evolve the schema during the data integration process.

Collection identification vs. entity and entity relationships’ identification: What to model?

An important question during document-oriented schema flexible modeling is what to model for an application. The ANT+ message protocol and usage document guides us to work with sensor nodes and to identify the device’s messages in general [134]; but for more detailed understanding of each device type’s data semantics we should follow the message format principles described in specific ANT+ Device Profile documents, such as Environment profile for temperature sensor [135], Stride Based Speed and Distance Monitor profile for footpod sensor [136] and Heart Rate Monitor profile for heartrate sensor [137]. A device profile is a defined standard, which confines all the vendors to manufacture their devices according to the rules and principles defined by ANT+ Alliance. Therefore, if a profile indicates about transmitting a message it means that all the vendor devices will transmit it, hence we can perform in general similar operations to the data of a specific type of all the sensors that meet a particular profile. ANT+ devices transmit monitoring data in the form of data messages which are specific to a profile type. Next we discuss the software engineering process of data modeling for the ANT+ sensors. We shall be rational during the time-series sensor data modeling for the MongoDB document oriented database, and shall also discuss the relational modeling (i.e., denormalization) with respect to our healthcare use case for the comparison purpose. The following descriptions expose the attributes of the chosen sensors.

Temperature It transmits current, minimum (in previous 24 h) and maximum (in previous 24 h) temperature in Centigrade (C\(^{\circ }\)). With each measurement it also transmits the transmission information, timestamp and the number of event counts—which increments with each measurement.

Footpod A footpod measures speed, cadence, distance, duration, strides accumulated and distance accumulated in meters/second, strides/minute, meters, seconds, number of strides and meters as measurement units respectively.

Heart rate monitor It measures and transmits the heart rate in beats per minutes (bpm), with heart beat variability, event count and timestamp.

If we consider the profile type as an entity then such data messages will be the attributes of that actually. The three sensors are actually the instances of the device entity, but should be treated separately because they do not share attributes. So we allocate a separate MongoDB collection to each sensor category and define its schema for the document storage. It is to note that a sensor will always have a unique serial number, manufacturer and model number, although the devices can share manufacturer ids, device types etc.

Open image in new window

However, there are messages which most devices transmit, and those are not specific to a profile standard; therefore they can be treated as separate common entity with a referenced instances to the relevant sensors. Such common messages are called as common data pages or background messages [138]. For example devices transmit their information such as battery status, message rate, manufacturer or hardware information etc. We treat such messages differently during processing and storage depending on the schema model. Our storage server accepts two types of JSON messages from the sensors: (i) per-minute timestamped holding sensor data, and (ii) per-hour timestamped holding device information and other healthcare data depending on the use case.

Because a user can use more than one sensor at a time, so user ID will be an attribute of each sensor instance. Based on these guidelines an entity relationship diagram is depicted in Fig. 5, which shows the possible attributes, relationships, relationship types and primary keys. Then a relational model is drawn in Fig. 6 in 1st and 2nd Normal Form (1NF) [103].

Open image in new window

Discussion w.r.t RDBMS In context to the RDBMS, this model although supports sensor data storage but it has many flaws, like many joins are required while querying the data for a patient at a particular time. There will be many null values because sensors will not transmit all the attribute values altogether. If we want to resolve this issue then we will end-up only with three columns in each of the sensor table i.e. SensorTable (timestamp, value, event count). This is similar to saving every event message, but this will not be robust for SQL as well as for NoSQL because there will be insertion and update issues besides the computation processing issues, such as aggregates etc.

Discussion w.r.t MongoDB Considering each entity as a possible document collection will require joins between them which document-oriented databases do not support natively. However it is possible either within the application or by using data processing analysis tools such as aggregation framework [100], MapReduce [101, 102], Hadoop [14, 19] etc. With document-oriented databases we do not have problems with the null values because of schema flexibility, however interestingly, in such a case we will again end-up with the same every event storage sort of structure, as discussed in the previous discussion. Such a scenario although supports the event message storage but is not robust with respect to processing and efficiency. We shall discuss it in the next sections.

Timestamped based cardinalities

A message rate is measured in cycles per second and its unit for frequency is Hertz (Hz). We discuss the data messages of the three device profiles with respect to their allowable message rates. The discussion provides us to calculate that how many instances of a certain sensor data message will appear against a time value, such as seconds, minutes and hours; and Table 6 summarizes the discussion.

Table 6

Data messages cardinalities with respect to allowable message rates

Device profile	Message rate (Hz)	Seconds	Minutes	Hours
Temperature	0.5	0	30	1800
	4.0	4	240	14,400
Foot pod	2.01	2	120	7236
	4.03	4	241	14,508
Heart rate	1.02	1	61	3672
	2.03	2	121	7308
	4.06	4	244	14,616

Temperature This environmental device broadcasts only at minimum 0.5 Hz or the maximum 4.0 Hz message periods therefore once per 2 s or once per 0.25 s respectively.

Footpod A device can receive data at the full rate ( 4.03 Hz) or at half of this rate, therefore once per 0.248 s or once per 0.498 s respectively.

Heart rate monitor It transmits at the full rate ( 4.06 Hz) or at one half or one quarter of this rate; therefore the data can be received four times per second, twice per second, or once per second.

Common messages or background messages appear after 64 data messages of each device profile type [138]. In context to the time it depends upon the message rate, for example, heartrate with a message rate of  2.03 Hz will transmit almost 2 messages per second, therefore in this case a background message will appear after 30 seconds approximately, whereas with the message rate of  1.02 Hz (which is the minimum possible message rate) we shall never receive two background messages in a single same minute roughly. It is also to notice that such information do not change rapidly, so we may not require to store every received message, hence it is better to store it either after every 10, 15 or 20 minutes or only once per hour.

Normalization or embedding: How good is the model?

Designing a better model is the main goal, and its process encounters the tradeoff between normalization and denormalization. For this we observe the examples for three different approaches to model the time-series data in MongoDB, i.e. document-per-event, document-per-minute and document-per-hour. All these three approaches are different in context of document embedding, and result in different storage space and hence data retrieval. We have already discussed the document-per-event scenario previously in the same section during the discussion w.r.t RDBMS. We learned that in such a case the outcome will be the three columns in each of the sensor table i.e. SensorTable (timestamp, value, event count).

Data Stream Management Systems (DSMS) are prone to the message arrival rates, and need adjustments upon any change in the rate. The current prototype does not support this automatic adjustment. Therefore, to be simple in the current prototype the least message rates of the sensors are used. We used the document-per-hour approach with the nested minute documents carrying 0-59 second documents. This approach requires one insert initially for the hour document, and then only updates are required for each new second value. To update the last second it requires maximum 59 + 59 steps. Next section presents the Mongoose based schema model based upon the chosen approach for the heart rate profile.

Since the above three versions of the embedded document storage differentiate the data into hour, minute and seconds, therefore before going to present the Mongoose schema, the first step is to separate the hourly, minute and second level data from the heart rate sensor payload. As stated already, background messages mostly relate with the hardware information of the sensor device itself, and appear only after 65 other data messages [139], therefore we keep them at the hourly level for the initial prototype. Since hour-level document will be a separate document therefore it will also keep a reference of the patient, as well as the start and end timestamps.

The minutes document will keep seconds’ sub-document inside it. Each second’s sub-document will have the heart rate data, i.e heart beat, variability and event count. The ERD in Fig. 5 and data model in Fig. 6 depict these attributes. The Fig. 7a shows an in general denormalized model for the heart rate sensor, in which the main SchemaSensor (for hours) contains the SchemaMinute (for minutes), which further contains SchemaSecond as the second’s sub-document schema. Alongside the Fig. 7b shows an in general Mongoose schema pseudocode for the denormalized schema. We can observe that most of the values will be at the seconds level, i.e, Heart beat, heart rate variability and event count as the measurements.

Open image in new window

Mongoose schema base temporal data storage and evolution

We require to define a finite set of sequenced steps, to store a JSON message instance related to an ANT+ profile type. They must perform logically correct updates to a database, leaving it consistent and concurrent after the manipulations. Following is the description of the storage algorithm which accepts JSON formatted sensor data messages to store and evolve the chosen schema model for the MongoDB. This is based upon the Mongoose object mapping middleware, which is first used to define the time-series schema model, as in Fig. 7 depicting the rationality and storage structure. The algorithm will use the object of the Mongoose schema model to perform the inserts and the updates to the document database, and such manipulations must be performed in a manner that the new data integration would result into a storage formation as defined and desired in the model. For this purpose the manipulations need to be done in a controlled logical manner and require an algorithm for this purpose. For the above stated Mongoose schema i.e, SchemaSensor, the set of tasks which the NodeJS Data Server will execute once for each new specific sensor event, for the JSON formatted sensor payload, is provided in the flowchart depicted in Fig. 8. This flowchart shows the logic of the algorithm and emphasises the individual steps and their interactions in a controlled manner from one action to another. This also depicts how the data flow will take place within the doc-per-hour schema based upon the Mongoose object mapping middleware. Figure 8 is preceded with JavaScript code presented in listing , which is the implementation of the data integration and schema evolver algorithm defined in the flowchart.

Open image in new window

Open image in new window

The temporal storage algorithm will use a list of input variables, such as Schm: a Mongoose Schema; args: a set of JSON arguments; rq: the request object; qryHr: the query criteria to find the hour document; hrDoc: the JSON object representing the Hour document object; and similarly the criteria for minutes along the minutes and second sub-document variables. The sensor schema in JavaScript will be imported first and will initialize the main schema object variable i.e., Schm (lines 1–2). The Schm object will be the mapper between the language constructs and the persistency of data in the MongoDB, and will help also in evolving the database according to the desired temporal schema logic. It first queries the database for the hours-level document based upon the time-interval provided in the request object (i.e, rq in line 4). The (qryHr) is a MongoDB query (in line 5) having the parameter and the projection parameters to fetch the relevant hour’s document. For example the following code constructs the query parameter to fetch the document where the hrmHour is equal to dateHours and patient is userid.

Open image in new window

In Asynchronous JavaScript programming the functions return the results in the last argument i.e., the callback; therefore the result will be returned in the hrDoc object. The listing is self explanatory and is having correspondent with the flowchart. If the hrDoc is found in line 6, the algorithm will go on searching for the minutes document, as in line 7; otherwise will move to line 18 for a new declaration and creation of the hours document.

Some related work in context of data modeling for the NoSQL databases is already pointed out very briefly during the previous subsections. A lot of data modeling studies has already been conducted by different researchers for NoSQL databases, but non has provided in depth study for the ANT+ sensor data especially to preserve the data based on temporal properties, therefore this research is novel in context to the schema flexible time series of the ANT+ data. The authors in [140] and [8] make models for the NoSQL databases, such as MongoDB; and present both relational and non-relational database queries to have a comparison between them. While using simple select queries, the former researchers present that joins are not required during NoSQL based retrieval, whereas they are required during SQL based approach. More using MongoDB queries the data is stored in single document or if needs to store in different documents then documents are related by using reference fields. The latter approach present a schema modeling case study for both relational and non-relational databases. They explain that each NoSQL database has its own query language, such as CQL (Cassandra Query Language) for Cassandra, MongoDB Query Language for MongoDB, Cypher Query Language for Neo4j etc. The latter research presents more general results and show query syntax for three databases, i.e PostgreSQL for relational database, MongoDB query language for MongoDB and Cypher query language for Neo4j. They explain that these NoSQL databases may require extra storage space, because of denormalized data, but results in overall improvements in performance, flexibility and scalability. However they do not deal with modeling for the temporal aspects.

Similarly Bugiotti et al. [106] design for a selected NoSQL framework, is based on best practices and guidelines. They provide a methodology, independent of the specific target system, which depends upon the initial activities of software design. They make novel data model for NoSQL databases, named as NoAM (NoSQL Abstract Model). After outlining the commonalities of various NoSQL systems, they specify a system-independent representation of the application data. They treat collections separately as abstract model or table, as in this system. However we have discussed and used the approach to model time series schema design in MongoDB with different variants during the research and the prototype development [128]. Same approach is used for different application domains in [141] and [142].

Vera et al. [143] proposes a general data modeling standard in the form of ERD diagrams for document-oriented databases. Parker et al. [144] compare the performance of NoSQL database, MongoDB, with one of the relational database, SQL Server. Their study shows that for a modest amount of data the performance of MongoDB is equal or better than the relational database. During this study they consider the three main aspects for performace i.e.,insert speed, update speed, and select operation speed. In [12] the authors perform a comparison of 14 different NoSQL Databases based on their data models, query possibilities, partitioning, and replication opportunities. They recommend to use NoSQL databases for fast operations over very large datasets. Following section guides us how we can improve our work in future.

For the temporal storage of the ANT+ sensor data messages there is a lot of potential for one to extend this work, especially in context to the adjustment of the different arrival rates and by offering different different storage models to each of them. This may require different schema models for each message rate. The other schema techniques also need attention (i.e., doc-per-minute, etc.) along the algorithm development for the data integration and the schema evolution. In future, we plan to have a comparison between the the exact storage measurements with respect to different schema models. For both relational and NoSQL models, such query based data measurement and comparison would reveal valuable results in context to the optimized data storage, retrieval and performance measurements. In the NoSQL domain and while using denormalized schema modeling, the temporal factor is a significant dimension that has not been addressed by the researchers too much. The approach presented in the form of schema model and algorithm is not limited to the ANT+ sensor data but is applicable to other domains also. Using this fundamental study and the referenced information we can define in general modeling standards for the document-oriented databases.

The data-driven monitoring applications rapidly and continuously transmit the sensor data, which is meaningful only when is processed and analysed while considering the temporal characteristics. Traditional RDBMS has given in managing such variety of prolific data and the new technologies, such as big data has promised to offer robust management frameworks to handle such continuous data streams. There are different NoSQL management frameworks, such as key-value, column-oriented, graph and document-oriented. A way to manage this is by defining out of the box optimized storage schemas and then to store the data into NoSQL databases while abiding by the format principles. This paper presents the usage of Mongoose middleware, to define document oriented hierarchical schemas for the temporal modeling of the ANT+ sensor data. The NodeJS data server uses these schemas to run a sequence of particular operations to be executed upon a specific event for the data integration and schema evolution. The algorithm automatically integrates the sensor data into a hierarchical structure based upon the temporal properties. This out of the box schema is modeled for NoSQL using the traditional ERD data modeling techniques. There are many possible schema variants, such as (i) document-per-event, (ii) document-per-minute, and (iii) document-per-hour. In this research, we define denormalized schema to have a document for each hour, which contains minutes as sub documents containing sensor data in an array of seconds’ sub-documents. The normalization and denormalization of the document hierarchy decides the quality of a schema with respect to number of reads, updates and storage space utilization.