---
title: "Using `HDTD` to Analyze High-Dimensional Transposable Data: An Application in Genetics"
author: Touloumis, A., Marioni, J.C., and Tavaré, S.
date: "`r format(Sys.time(), '%d %B %Y')`"
output: rmarkdown::html_vignette
vignette: >
    %\VignetteIndexEntry{HDTD to Analyze High-Dimensional Transposable Data}
    %\VignetteEngine{knitr::rmarkdown}
    %\VignetteEncoding{UTF-8}
---
```{r, echo=FALSE}
    knitr::opts_chunk$set(
    tidy = TRUE,
    collapse = TRUE, 
    comment = "#>")
```

# Introduction
The R/Bioconductor package `HDTD` is designed to analyze high-dimensional transposable data. The term transposable data implies the following structural information in the dataset: 

* the data for each sampling unit (e.g., subject/patient) can be written in a matrix,
* the rows and the columns in each matrix correspond to two distinct sets of variables,
* dependencies might occur among and/or between the row and column variables.

The term high-dimensional implies that the sample size (e.g., the number of subjects/patients) is a lot smaller than the total number of row and column variables.

Since the statistical methods implemented in `HDTD` were primarily motivated by studies in genetics, a microarray dataset is utilized to illustrate the functionality of the package. However, the use of `HDTD` is not limited to gene-expression experiments and we emphasize that it is suitable for analyzing datasets that satisfy the high-dimensional transposable data definition.

# Mouse aging dataset
`HDTD` includes a subset of the tissue study described in Zahn et al. (2007). This dataset contains expression levels for $40$ mice. For each mouse, the expression levels of $46$ genes that belong to the vascular endothelial growth factor signalling pathway were measured across $9$ tissues (adrenal gland, cerebrum, hippocampus, kidney, lung, muscle, spinal cord, spleen and thymus). The experimental design satisfies the definition of transposable data because: 

* the data for each mouse can be written in a matrix form, 
* the rows correspond to genes and the columns to multiple tissues, and 
* we expect dependencies among genes, among tissues and between genes and tissues. 

The dataset is formatted as a single matrix with rows the $46$ genes and columns the $9 \times 40=360$ tissues.

```{r}
library("HDTD")
data("VEGFmouse")
dim(VEGFmouse)
rownames(VEGFmouse)
```

Further, every 9 consecutive columns belong to the same mouse and the tissues are ordered in the same way for each mouse. For example, we can check the column variables for the first two subjects:

```{r}
colnames(VEGFmouse)[1:18]
```

It is crucial to import the data in a particular format when using `HDTD`. In particular, first write the data for each subject (mouse) in a matrix form while preserving the order of the row (genes) and column (tissues) variables within each subject-specific matrix. Then, create a single matrix by stacking column-wise the subject-specific matrices the one after the other. Finally, read the datafile in `HDTD`.

# Mean relationship of genes across tissues
The user can determine the mean relationship of the genes across tissues by testing and estimating the mean matrix. One interesting hypothesis to be tested is conservation of the gene expression levels across tissues, i.e., if the mean gene expression levels vector in the VEGF signaling pathway changes across the $9$ tissues:
```{r}
meanmat.ts(VEGFmouse, N = 40, group.sizes = 9)
```
Since $p$-value $<0.001$, we have strong evidence against the null hypothesis that there is no tissue effect in the gene expression levels. To explore further the mean gene expression level pattern across tissues, additional tests can be carried out. We illustrate a more complicated hypothesis that also requires data manipulation. Consider testing the hypothesis that the mean gene expression levels vector is constant only across the adrenal glands, the spleen, the kidney and the lung. To do this, we first need to place these $4$ tissues in a successive order in the dataset,   
```{r}
colnames(VEGFmouse)[1:9]
columnorder <- c(1, 4, 5, 8, 2, 3, 6, 7, 9)
VEGForder <- orderdata(VEGFmouse, N = 40, order.cols = columnorder) 
colnames(VEGForder)[1:9]
```
and then to perform the test using the ordered dataset
```{r}
meanmat.ts(VEGForder, N = 40, group.sizes = c(4, 1, 1, 1, 1, 1))
```
Note that we have included $5$ additional column groups of size one in the `group.sizes` argument to reflect the fact that the mean gene expression levels vector in each of the remaining $5$ tissues remained unspecified. The null hypothesis is rejected, and hence we may conclude that the mean gene expression levels vector is not constant in the adrenal glands, the spleen, the kidney and the lung.

Apart from hypothesis testing, one can estimate the mean relationship between the genes and the tissues. In this example, the mean matrix seems to be unstructured and thus the mean gene expression levels in the $9$ tissues can be estimated via the sample mean matrix 
```{r}
sample_mean <- meanmat.hat(VEGFmouse, N = 40)
sample_mean
```
Note that the output preserves the order of the genes and the columns. For example, $0.8399$ is the average $\log_2$ intensity for gene "Akt1" in the adrenal gland based on $40$ mice. The mean matrix for the first 10 genes across the $9$ tissues is
```{r}
head(round(sample_mean$estmeanmat, 4), n = 10)
```

# Dependence structure of the genes and of the tissues
The user can estimate two covariance matrices, one for the genes (rows) and the other for the multiple tissues (columns). We have developed shrinkage estimators for both covariance matrices but we let the user decide if shrinkage is required to both, one or neither of these matrices. In principle, we recommend shrinking both covariance matrices in order to obtain well-defined and invertible covariance matrix estimators. The `covmat.hat` function provides the corresponding covariance estimators: 
```{r}
est_cov_mat <- covmat.hat(datamat = VEGFmouse, N = 40)
est_cov_mat
```
The output summarizes the results but the user can recover the full covariance matrix estimators. For example, the covariance matrix of the tissues is
```{r}
round(est_cov_mat$cols.covmat, 3)
```
Moreover, the user can study the gene-wise or tissue-wise correlation by using the `covmat.ts` function. For example, the results from the identity, sphericity and diagonality hypothesis tests applied on the column variables
```{r}
covmat.ts(datamat = VEGFmouse, N = 40, voi = "columns")
```
suggest that the tissues might be uncorrelated at a $5\%$ significance level. 

# How to cite
```{r}
print(citation("HDTD"), bibtex = TRUE)
```



