too-many-cells

too-many-cells can be installed using the functional package manager nix . While you will need sudo to install, no sudo is required after the correct setup. First, install nix following the instructions on the website. Then, with an unset LD_LIBRARY_PATH,

git clone https://github.com/GregorySchwartz/too-many-cells.git
cd too-many-cells
nix-env -f default.nix -i too-many-cells

Different computers have different setups, operating systems, and repositories. Do put the entire program in a container to bypass difficulties (with the other methods above), we user docker. So first, install docker.

To get too-many-cells (replace 2.0.0.0 with any version needed):

docker pull gregoryschwartz/too-many-cells:2.0.0.0

To run too-many-cells in a docker container:

sudo docker run -it --rm -v "/home/username:/home/username" gregoryschwartz/too-many-cells:2.0.0.0 -h

Now you can follow the tutorial below with the addition of the docker paths and commands. If you add yourself to the docker group, sudo is not needed. For instance:

docker run -it --rm -v "/home/username:/home/username" \
    gregoryschwartz/too-many-cells:2.0.0.0 make-tree \
    --matrix-path /home/username/path/to/input \
    --labels-file /home/username/path/to/labels.csv \
    --draw-collection "PieRing" \
    --output /home/username/path/to/out \
    > clusters.csv

Make sure to increase the memory that can be used by docker containers if you use macOS or Windows. Also, docker won’t be able to find your files by default. You need to mount the folders with -v in order to have docker read and write from and to the filesystem, respectively. Read the documentation about volumes for more information. You can simply mount your entire relevant path as in the above example to handle both input and output, or just mount your entire user directory as above. Specifically, -v "/home/username:/home/username" for the whole directory or each individual -v /path/to/matrix/on/host:/input_matrix with -m /input_matrix is what you want, where before the : is on the host filesystem while after the : is what the docker program sees. Then you can write the output in the same way: -v /path/to/output/on/host:/output will write the output to the folder before the :.

To build the too-many-cells image yourself if you want:

nix-build docker.nix
docker load < /nix/store/${NAME_OF_OUTPUT_IMAGE}.tar.gz

Be sure to have LD_LIBRARY_PATH unset when running nix-env to make sure the linked libraries are in /nix/store.

Try upgrading stack with stack upgrade. The new installation will be in ~/.local/bin, so use that binary.

stack and too-many-cells assume system libraries and programs. To solve this issue, first install the dependencies above at the system level, including system R. Then to every stack and too-many-cells command, prepend PATH="$HOME/.local/bin:/usr/bin:$PATH" to all commands. For instance:

• PATH="$HOME/.local/bin:/usr/bin:$PATH" stack install
• PATH="$HOME/.local/bin:/usr/bin:$PATH" too-many-cells make-tree -h

If your shared libraries are abnormal and use libR.so from non-system locations, be sure to also have LD_LIBRARY_PATH=/usr/lib/:$LD_LIBRARY_PATH when installing (and / or the location of R libraries, such as /usr/local/lib/R/lib/).

Open an issue! While working on the issue, try out the docker for too-many-cells, it requires no installation at all (other than docker).

Docker containers may run into this issue if the memory given to the containers is insufficient. Make sure to increase the memory that can be used by docker containers.

For some computers, you may need to change the command to single quotations for the argument: --draw-leaf 'DrawItem (DrawContinuous [\"Cd4\"])'

too-many-cells make-tree generates a binary tree using hierarchical spectral clustering. We start with all cells in a single node. Spectral clustering partitions the cells into two groups. We assess the clustering using Newman-Girvan modularity: if \(Q > 0\) then we recursively continue with hierarchical spectral clustering. If not, then there is only a single community and we do not partition – the resulting node is a leaf and is considered the finest-grain cluster.

The most important argument is the –prior argument. Making the tree may take some time, so if the tree was already generated and other analysis or visualizations need to be run on the tree, point the --prior argument to the output folder from a previous run of too-many-cells. If you do not use --prior, the entire tree will be recalculated even if you just wanted to change the visualization!

The main input is the --matrix-path argument. When a directory is supplied, too-many-cells interprets the folder to have matrix.mtx, genes.tsv, and barcodes.tsv files (cellranger outputs, see cellranger for specifics). If a file is supplied instead of a directory, we assume a csv file containing feature row names and cell column names. This argument can be called multiple times to combine multiple single cell matrices: --matrix-path input1 --matrix-path input2.

The second most important argument is --labels-file. Supply with a csv with a format and header of “item,label” to provide colorings and statistics of the relationships between labels. Here the “item” column contains the name of each cell (barcode) and the label is any property of the cell (the tissue of origin, hour in a time course, celltype, etc.). You can also now use -Z as a list for each matching -m in order to manually give the entire matrix that label (useful for situations like -m ./t-all -Z T-ALL -m ./control -Z Control). To get the newly generated labels with =-Z into a labels.csv file, specify --labels-output and the labels.csv will be in the output folder.

To see the full list of options, use too-many-cells -h and -h for each entry point (i.e. too-many-cells make-tree -h).

The interactive entry point has a basic GUI interface for quick plotting with a few features. We recommend limited use of this feature, however, as it can be quite slow at this stage, has fewer customizations, and requires specific dependencies.

too-many-cells interactive \
    --prior out \
    --labels-file labels.csv

A main use of single cell clustering is to find differential genes between multiple groups of cells. The differential aids in this endeavor by allowing comparisons with edgeR. Let’s find the differential genes between the liver group and all other cells. Consider our pruned tree from earlier:

We can see the id of each group with --draw-node-number.

We need to define two groups to compare. Well, it looks like node 98 defines the liver cluster. Then, since we don’t want 98 to be in the other group, we say that all other cells are within nodes 89 and 1. As a result, we end up with a tuple containing two lists: ([89, 1], [98]). Then our differential genes for (liver / others) can be found with differential (sent to stdout):

too-many-cells differential \
    --matrix-path input \
    --prior out_pruned \
    --filter-thresholds "(250, 1)" \
    -n "([89, 1], [98])" \
    > differential.csv

If we wanted to make the same comparison, but compare the liver subtree with liver cells from all other subtrees, we can use the --labels argument:

too-many-cells differential \
    --matrix-path input \
    --prior out_pruned \
    --labels-file labels.csv \
    --filter-thresholds "(250, 1)" \
    -n "([89, 1], [98])" \
    --labels "([\"Liver\"], [\"Liver\"])" \
    > differential_liver.csv

We can also look at the distribution of abundance for individual genes using the --features and --plot-output arguments.

Furthermore, we can compare each node to all other cells by specifying no nodes at all. The output file will contain the top --top-n genes for each node. We recommend using multiple OS threads here to speed up the process using +RTS -N${NUMOSTHREADS} (no number to use all cores). The following example will compare all nodes to all other cells using 8 OS threads:

too-many-cells differential \
    --matrix-path input \
    --prior out_pruned \
    --filter-thresholds "(250, 1)" \
    -n "([], [])" \
    --normalization "UQNorm" \
    +RTS -N8

Diversity is the measure of the “effective number of entities within a system”, originating from ecology (See Jost: Entropy and Diversity). Here, each cell is an organism and each cell label or cluster is a species, depending on the question. In ecology, the diversity index measures the effective number of species within a population such that the minimum is a diversity of 1 for a single dominant species up to maximum of the total number of species (evenly abundant). If our species is a cluster, then here the diversity is the effective number of cell states within a population (for labels, make-tree generates these results automatically in “diversity” columns). Say we have two populations and we generated the trees using make-tree into two different output folders, out1 and out2. We can find the diversity of each population using the diversity entry point.

too-many-cells diversity\
    --priors out1 \
    --priors out2 \
    -o out_diversity_stats

We can then find a simple plot of diversity in diversity_output. In addition, we also provide rarefaction curves for comparing the number of different cell states at each subsampling useful for comparing the number of cell states where the population sizes differ.

“Pseudotime” refers to the one dimensional relationship between cells, useful for looking at the ordering of cell states or labels. The implementation of pseudotime in a too-many-cells point-of-view is by finding the distance between all cells and the cells found in the longest path from the root in the tree. Then each cell has a distance from the “start” and thus we plot those distances.

too-many-cells paths\
    --prior out \
    --labels-file labels.csv \
    --bandwidth 3 \
    -o out_paths

For more information, check out the too-many-peaks walkthrough.

scATAC-seq is a powerful technology for quantifying chromatin accessibility for individual cells. too-many-cells now supports scATAC-seq to generate cell clade relationships from chromatin state information through too-many-peaks. All of the previous analyses used with gene-product features now work with genomic regions in the form chrN:START-END, where N is the chromosome number, START is the start of the region and END is the end base of the region.

Matrices in this format can be read from either CSV or matrix-market as above but with the correctly formatted features, or you can load in directly from a fragments.tsv.gz file in Cellranger format (tab delimited with each row being chrN\tSTART\tEND\tBARCODE\tCOUNT) making sure that the filename contains the fragments ending, such as t-all_fragments.tsv.gz. For example:

too-many-cells make-tree\
    -m ./t-all_fragments.tsv.gz \
    -Z "T-ALL" \
    -m ./control_fragments.tsv.gz
    -Z "Control" \
    --filter-thresholds "(1000, 1)" \
    --binwidth 5000 \
    --lsa 50 \
    --normalization NoneNorm \
    --blacklist-regions-file Anshul_Hg19UltraHighSignalArtifactRegions.bed.gz \
    --draw-node-number \
    --draw-mark "MarkModularity" \
    --fragments-output \
    --labels-output \
    -o out \
    > out_leaves.csv

Note: We use --lsa and --normalization NoneNorm for latent semantic analysis dimensionality reduction as there are many features in scATAC-seq, so we try to overcome a potential issue where all cells are considered outliers. To blacklist known biased regions in the genome, we can call --blacklist-regions-file. The --fragments-output and --labels-output go hand-in-hand with -Z in order to keep the renamed barcodes and labels (found in the output folder). too-many-cells will binarize the data by default unless --no-binarize is specified. Lastly, we choose a binwidth using --binwidth to conform to a set of standard features across cells and samples.

With scATAC-seq, we want to identify enriched locations in the genome for each newly found subpopulation of cells. The peaks entrypoint can collect the appropriate fragments for quantification and visualization of peaks.

too-many-cells peaks \
    -f ./out/fragments.tsv.gz \
    --prior ./out \
    --genome human.hg19.genome \
    --bedgraph \
    --labels-file ./out/labels.csv \
    --all-nodes \
    --peak-node "1" \
    --peak-node "5" \
    --peak-node-labels "(1, [\"Control\"])" \
    --peak-node-labels "(5, [\"T-ALL\"])" \
    -o out_peaks \
    +RTS -N6

Here, we will have our peaks in the specified output folder, along with many other files and folder:

--bedgraph enabled the cluster_bedgraphs folder, while --all-nodes specified to find peaks for all nodes, not just the leaves. However, when paired with --peak-node, we just look at the peaks for each node in the list (but --all-nodes is still required if looking at non-leaf nodes as well). Without --peak-node, this command would have found peaks for every node. Furthermore, --peak-node-labels allows the filtration based on the label of cells in of the requested node. --genome tells the peak finding program where the genome file is (containing the effective genome sizes of chromosomes in tab-delimited format of chrN\tSIZE used in the MACS2 program). Here, the -f fragments.tsv.gz and labels.csv was from the previous scATAC-seq section, where we automatically generated the correctly renamed barcodes and labels. Lastly, +RTS -N6 tells too-many-cells to use six cores for the calculation. These output files, especially the merged peak files, can be used for differential accessibility analysis as in scRNA-seq. This entrypoint is highly customizable, down to the exact command used for peak calling, so check out too-many-cells peaks -h for more information.

After differential accessibility using peaks, the result can be used to find motifs enriched in each node.

too-many-cells motifs \
    --diff-file ./diff_out.csv \
    --motif-genome hg19.fa \
    --top-n 1000 \
    --motif-command "homer/homer-4.9/bin/findMotifs.pl %s fasta %s" \
    -o motifs

In this example, we use the output from a differential expression analysis using too-many-cells differential from our merged peaks. Using a complete genome file used by our motif program of choice (here HOMER, but defaults to MEME) with --motif-genome, we want to provide the motif program with the top 1000 most differential peaks using --top-n. Lastly, while the default uses MEME, we find HOMER to be much faster. The prior command shows the use of another program to find the motifs, making sure the %s for input and output are in the right locations (check too-many-cells motifs -h).

To identify potential cell type candidates from sorted bulk data, too-many-cells classify uses cosine-similarity to provide scores for each bulk population. For example, we have a scATAC-seq experiment in ./mat. We also have known bulk ATAC-seq peak data of B cells in bedgraph files. We can score each cell with:

too-many-cells classify \
    --reference-file ./proB.bdg \
    --reference-file ./preB.bdg \
    --reference-file ./memoryB.bdg \
    --reference-file ./plasmaB.bdg \
    -m ./mat \
    --normalization "NoneNorm" \
    --blacklist-regions-file "mm9-blacklist.bed.gz" \
    > labels.csv

--reference-file is a list of bedgraphs here for each population. You can also specify a single reference file as an input matrix with each barcode being the label for the population, as bulk just has one sample. To use a single matrix, use --single-reference-matrix in addition to --reference-file to specify the file as a single reference matrix. The output will be identical to a normal too-many-cells labels.csv file, but with an additional column score which provides the value of the highest cosine-similarity label.

Spatial single-cell technologies allow us to measure not only the features of cells such as cell surface markers or transcriptomes, but also the spatial location of each individual cell in situ. These technologies, such as imaging mass cytometry and Visium, allow us to use various methods to quantify the spatial relationships between cell features and cell types. too-many-cells can report these relationships with the spatial entrypoint, making use of both AnnoSpat for cell type classification and spatstat for relationship quantification.

As an example, consider an imaging mass cytometry output containing two files, features.csv and spatial.csv. features.csv (can be any matrix format that too-many-cells accepts with -m) here is a matrix of cell rows and feature columns:

item,CD20,CD4,CD8,Foxp3,...
barcode1,0.1095368741640727,0.013183117496457954,0.19233368842522866,0.05579191206063343,...
barcode2,0.08268388046574766,0.003996753797330361,0.007560142177239592,0.0008473833902161547,...
...

spatial.csv is a file containing the locations of each cell, of the format item,sample,x,y, where item is the cell barcode, sample is the sample the barcode came from (for bulk processing to make sure there is segregation by sample), and x and y are the cell coordinates:

item,sample,x,y
barcode1,donor1,-493.99,496.08
barcode2,donor1,-479.629,496.641

Using this information, we can relate the cells by their marker expression:

too-many-cells spatial \
  --matrix-transpose \
  -m total_normalized_features.csv \
  -j total_spatial.csv \
  -o tmc_mark_output \
  --mark "CD4" \
  --mark "CD20"

We use --matrix-transpose to make sure the barcodes for the feature matrix becomes the columns in this case, -o denotes the output folder for the analyses, and --mark denotes each feature we want to relate. If you want to see every pairwise comparison between all marks, instead just use --mark "ALL".

too-many-cells will output results into the tmc_mark_output folder containing a folder for each sample. Within each sample folder, there will be projections and relationships folders, the former containing an interactive visualization of the cells locations on the left with the cumulative distribution functions of each mark on the right. You can click and drag on these distributions to filter the cells on the left plot by their mark.

The relationships folder contains additional folders for pairwise comparisons of marks. Within each of these folders, there are the following files (for more information, check out spatstat:

The stats.csv file contains multiple measures to summarize the functions:

The mark cross-correlation function may be used with discrete values as well, so instead of, for instance, cell surface expression, you could use cell types by passing in a labels file (used by any too-many-cells entrypoint) with -l:

too-many-cells spatial \
  --matrix-transpose \
  -m total_normalized_features.csv \
  -j total_spatial.csv \
  -o tmc_mark_output \
  -l labels_celltypes.csv \
  --mark "Helper T Cell" \
  --mark "B Cell"

You can even use AnnoSpat to predict cell types to use instead of a labels file with --annospat-marker-file (see the AnnoSpat documentation for this format).

A simple entrypoint to output the transformed matrix too-many-cells uses before clustering. Saves to --mat-output.