1 Introduction

DNA is the molecular basis of life. It stores all the information required for life to reproduce, sustain itself and adapt to the environment. The human genome consists of approximately three billion DNA base pairs [1, 2], which would total about three meters if linearly untangled. Therefore, to fit in the cell nucleus, which has an approximate diameter of six micrometers, DNA has to fold itself several times.

The study of DNA folding is a current biological research frontier and only recently it was possible to reliably probe its characteristics. A major technological milestone, which enabled for the first time to see the whole genome three-dimensional organization, was the invention of the Hi-C technique [3], which maps 3D contacts between all genomic locations. For the first time it was possible to reveal that genomes fold into high-level structures called Topologically Associating Domains (TADs) [4], which are genomic regions of high self-interaction and low interactions across them. TADs are of great biological relevance, since they are speculated to be associated with functional compartmentalization of genomic features like genes and their regulatory elements. For this reason, understanding the mechanisms of TADs and chromatin loops formation is a major biological research problem.

Studying chromatin conformation and Hi-C data requires high computational resources and complex data analyses pipelines. This is due to both the intrinsic noise present in the data and to the big size of the experimental datasets [5]. Hi-C experiments can produce billions of reads and their storage can require several terabytes. For this reason, it is necessary to develop scalable and interactive software for the exploration of big genomic datasets like Hi-C. In addition, studying Hi-C data usually requires the integration of other heterogeneous data sources (like, for example, ChIP-seq and RNA-seq), which implies that any software pipeline designed for complex biological analyses on chromatin conformation has to be able to work with multiple data formats as well as be able to easily integrate and manipulate them.

In the first part of this chapter, we present the design and implementation of a software for interactive and scalable data exploration for genomic data. This system, named PyGMQL, is a easy-to-use Python extension of the GMQL big genomic data engine, developed by the Genomic Computing team at Politecnico di Milano. PyGMQL is a Python package which enables the user to build complex genomic pipelines by relying on the Spark big data engine. Using a carefully designed set of functions, it allows the user to inspect and manipulate arbitrarily big genomic datasets and design the flow of execution of the program. Once the researcher has defined the complete data analysis pipeline, it can then be fully executed on the complete dataset. It also allows the loading and manipulation of heterogeneous data, coming from different experimental procedures, through the adoption of the Genomic Data Model (GDM), developed initially for the GMQL system. We demonstrate the qualities and performance of this software through a series of data analysis scenarios, showing its usability, requirements and execution times. PyGMQL is extremely versatile and can be used both a downstream analysis data analysis tool and as a development framework on top of which bioinformaticians can build new specialized software tools.

In the second part, we apply this software stack to the study of chromatin conformation data. We focus on the study of the epigenetic drivers and determinants of TADs, and we explore the relationship between the positions and motif orientations of CTCF binding sites. CTCF is a structural protein known to be associated with chromatin looping [6], but the underlying rules determining TAD topologies have never been highlighted. We propose a set of spatial rules, having as elements the binding sites of CTCF, which correlate with TADs and their topological characteristics. We argue that genome conformation can be explained by a set of “grammatical rules” acted out by CTCF.

We finally inspect the relationship between chromatin conformation and gene expression. We ask to what extent chromatin networks determined by Hi-C can explain co-expression relations between genes, both intra- and inter-chromosomically. We model this research question as a machine learning problem, designing a graph representation learning model for the encoding of gene chromatin topological features. The learnt gene embeddings are used then as inputs to a Random Forest classifier. We demonstrate that this model outperforms a set of baselines based on previous works.

2 Interactive and Scalable Data Analysis for Genomics

The analysis of biological data is conventionally divided into three macro-steps, depending on the data processing and the biological question being asked and answered. This categorization is particularly important in the case of next-generation sequencing data.

  • Primary data analysis: The first step in the analysis of biological data is usually the production and quality assessment of the raw data generated from the sequencing machine. It produces the raw nucleotide sequences for each read obtained from the machine.

  • Secondary data analysis: Reads are then filtered and aligned to a reference genome, which gives the researcher the information about the position of a sequencing read in the target genome.

  • Tertiary data analysis: Researchers then deepen into the biological problem they are studying and ask high-level questions about the data. This is the most important and case-specific step of biological data analysis and it usually requires custom pipelines and result evaluation, as well as the integration of different data sources.

2.1 The Genometric Query Language and Its Ecosystem

The Genomic Computing (GeCo) team at Politecnico di Milano decided to address these issue of managing, extracting and analysing tertiary biological data by proposing a paradigm shift in genomic data management [7].

The Genomic Data Model (GDM) represents genomic information as the combination of region data and metadata [8]. The Genometric Query Language (GMQL) [7, 9] was designed to answer complex biological questions using a declarative query language. GMQL queries are compiled into a computation graph. This enables the implementation of different backends of the GMQL system [9]. GMQL has support for Spark [10], Flink [11] and SciDB [12] implementations. The GMQL repository [13] hosts data coming from The Cancer Genome Atlas [14], Roadmap Epigenomics [15], ENCODE [16] and others. The main interface of the GMQL system is through a web application, where the user can browse through the publicly available datasets in the GMQL repository, write and compile genomic queries through the query editor, and inspect the results of queries in his/her private storage.

2.2 PyGMQL: Scalable Programmatic Data Analysis of Genomic Data

Footnote 1 GMQL operates in a batch fashion, which means that the scientists has to write the query, run it and wait for the outcome. This approach does not fit very well the data exploration paradigm. This is due the fact that, in particular at the beginning of a study, scientists do not know the specific research question they want to ask to their data. This approach requires a shift of our framework towards interactive computation, where parameters are learnt during the exploration process through trial and error.

With the aim of solving this issue, we designed and implemented PyGMQL [17, 18], a Python library which embeds the GMQL engine.

The library adopts a client-server architecture, where the Python front-end exposes to the user all the dataset manipulation functions and utilities, while a Scala back-end implements all the query operators. As depicted in Fig. 1a, the back-end relies on the implementation of GMQL on Spark.

PyGMQL offers a set of methods which wrap and extend the GMQL language operators. PyGMQL offers methods to manipulate genomic datasets as variables, each associated to structures called GMQLDataset. The library stores all the operations performed on a variable through a GMQL directed acyclic graph (DAG) (see [9]). This design enables the back-end to apply query optimizations to the DAG structure [19]. Conforming to the lazy execution model of Spark, no actual operation is executed until the user explicitly triggers the computation.

PyGMQL can access data stored in the local machine, like most of the Python libraries. In addition, PyGMQL can also interface with an external GMQL system to interact with his/her private datasets or the public repository. Therefore, queries in PyGMQL can be composed of genomic operations acting both on local and remote datasets. The library can also “outsource” the query computation to an external GMQL service, and then download the result to the user machine (Fig. 1b). The location of the datasets used during the query is therefore orthogonal with respect to the mode of execution. The library keeps tracks of the used datasets and their dependencies during the whole Python program execution, minimizing the data transmission between local and remote systems.

Fig. 1
figure 1

Figure adapted from [17]

a Schematic representation of the software components of PyGMQL. In the front-end, the GMQLDataset is a data structure associated with a query. The GDataframe stores the query result and enables in-memory manipulation of the data. The RemoteManager module is used for message interchange between the package and an external GMQL service. The back-end is implemented in Spark. b Deployment modes and executor options of the library.

Full size image

We demonstrated the flexibility of the PyGMQL library through three data analysis workflows, available in the PyGMQL GitHub repository.Footnote 2 For a progressive introduction to PyGMQL usage, the applications are increasingly complex both for what concerns the biological analysis and the data size. We also provide a performance evaluation on increasingly large datasets of a significantly complex genomic query, highlighting the scalability of the system [17].

3 The Grammar of Genome Folding

Footnote 3 We then decided to apply our novel software stack to a concrete biological research problem, having as focus the study of the mechanisms of chromatin folding.

The mechanisms underlying the storage and packing of DNA is a current knowledge frontier in biology. Thanks to the Hi-C technique it was possible to identify genomic regions of having an high level of self-interaction, which were named Topologically Associating Domains (TADs) [4]. TADs are divided by boundaries that are detected as positions in the genome where there is a sharp break from preferential left-ward interactions to preferential right-ward interactions [4, 21]. Scientists also tried to understand the underlying factors determining chromatin topological organization. It was shown that the CTCF protein is a fundamental actor in this context. The CTCF protein binds to an asymmetric motif on the DNA [22]. This means that it is possible to find these motifs in two possible orientations (> and <). It has been shown that convergent CTCF binding sites (\(> <\)) are located at the two extremities of long range interacting DNA regions (loops), explaining multiple features of Hi-C data sets [6, 23].

Fig. 2
figure 2

Figure adapted from [20]

a Enrichment of the four CTCF pattern classes around highly conserved TAD boundaries in 7 cell types. b Enrichment of the same classes along the length of TADs, as a function of percentage of TAD size. c Enrichment of the same classes around the center of TADs. d Schematic representation of the “grammatical rules” induced by CTCF and its orientation, and their correlation with TAD features.

Full size image

With the aim to reconcile the notion of chromatin loops and TADs, we created a classification scheme for sets of more than two adjacent CTCF sites. Each cluster is classified based on the relative orientation of the CTCF binding sites composing it[20]. We identified four categories: same (\(>>>\), \(<<<\)), convergent (\(>> <\), \(> <<\)), divergent (\(<>>\), \(<<>\)) and convergent-divergent (\(> <>\), \(<> <\)). We therefore analysed the distributions of the sizes of CTCF clusters the human genome, revealing that, at length scales ranging from 5 to 100 kb, divergent CTCF site clusters are enriched, while convergent CTCF site clusters are depleted. This suggests that divergent CTCF sites code for TAD boundaries (Fig. 2a, b) and that convergent CTCF sites are involved in the definition of left and right TAD sections (Fig. 2c). We validated this orientation-based grammar as a function of CTCF site strength and TAD boundary strength (Fig. 2a).

These results suggest the presence of a linear “grammar” dictating the relative orientations of CTCF binding sites motifs, and that these simple rules play an important role in the definition of key chromatin conformation features, like TADs, their inside and their boundaries (Fig. 2d).

4 Chromatin Conformation and Gene Expression

A key element left out from our previous analysis of chromatin conformation is the role of genes. Gene expression is the key biological mechanism which produces phenotypic differences across cell types, tissues and biological conditions. Gene expression data can be summarized through the computation of co-expression between pairs of genes, building therefore a co-expression network.

The relationship between gene co-expression and chromatin conformation is of great biological interest [24,25,26]. Given the high complexity of Hi-C data and the difficult definition of gene coexpression networks [27], the development of proper computational tools to investigate such relationship is rapidly gaining the interest of researchers. One of the most fascinating questions in this context is how chromatin topology correlates with gene coexpression and which physical interaction patterns are most predictive of coexpression relationships [28, 29].

Fig. 3
figure 3

Figure adapted from [30]

Schematic representation of the workflow. We generate a Gene Chromatin Network by summarizing Hi-C information of genes and their neighborhood for each gene (1). We then produce a reduced vector representations of genes through network embedding techniques (2). Finally, we take the combined pairs of gene vectors as input for a Random Forest classifier, trained on a subset of the gene co-expression network (3).

Full size image

We explored the relationship between chromatin conformation and gene expression using a predictive modeling approach. Specifically, we designed a model to predict co-expression between two genes from the physical set of interactions derived from an Hi-C experiment (see Fig. 3). We used a representation learning approach [31] for embedding the topological features of genes. In our work, the features of the nodes are learnt by solving an optimization problem, which defines the embedding strategy of the physical interaction network extracted from Hi-C data. Therefore, the proper choice of the optimization method is critical.

We compared two different node embedding strategies. The first method is based on Matrix Factorization [32], while the second exploits a random walk procedure to find similar embeddings for genes in the same neighborhood [33]. We then used the learnt embeddings to train a non-linear classifier, based on Random Forest [34]. We compared the performances between the two embedding strategies and against a set of baselines: the first one is based on the computation of a set of predefined measures for each gene/node in the Hi-C network to be used as input of the classifier [24]; the second is a simple distance-based predictor using only the linear distance between genes in the genome as input.

Fig. 4
figure 4

Figure adapted from [30]

Accuracy measure across the 22 single-chromosome networks of the proposed gene embedding strategies and comparison with baselines. Each box plot is derived from the accuracy measures for each cross validation fold and for each chromosome.

Full size image

We validated our models on a comprehensive collection of datasets, where Hi-C data and matching gene expression is available, showing consistent results across conditions, cell line and tissues. Our results (see Fig. 4) show that both our embedding methods outperform the considered baselines. This finding shows that it is indeed possible to encode the topological properties of genes, and to use them to predict their co-expression. It is also important to point out that, despite the significant improvement in prediction performance of our proposed methods, we were able to predict correctly only a portion of the co-expression relationships. A possible interpretation of this result is that chromatin conformation and gene expression are linked phenomena, but they also maintain a degree of independence: the gene expression machinery is a very complex system, influenced by several factors, like histone modifications and DNA methylation, together with chromatin conformation.