Question: what are the setting optimised for speed rather than memory use?

[vitkl]

Hi

I am using the following workflow to read and quantify individual 10x scATAC batches:

def load_atac_tiles(
    sample,
    fragment_file,
    path_to_reference='.../refdata-cellranger-arc-mm10-2020-A-2.0.0/',
    bin_size=500,
):

    chrom_size = {
        str(k): v.values[0]
        for k, v in pd.read_csv(f"{path_to_reference}star/chrNameLength.txt", sep="\t", header=None, index_col=0).T.items()
    }

    data = snap.pp.import_data(
        fragment_file,
        # genome=snap.genome.mm10,
        gff_file=f"{path_to_reference}genes/genes.gtf.gz",
        chrom_size=chrom_size,
        # file="pbmc.h5ad",
        sorted_by_barcode=False,
        chunk_size=100000,
    )
    data.obs['sample'] = sample
    data.obs_names = (
        data.obs['sample']
        + pd.Series("#", index=data.obs_names)
        + data.obs_names
    )

    snap.pl.tsse(data, interactive=False)
    plt.show()

    # quantify ATAC
    snap.pp.add_tile_matrix(
        data, bin_size=bin_size, chunk_size=100000, # exclude_chroms=['chrM', 'chrY', 'M', 'Y'],
    )

    return data

It takes a long time (7 min for 500bp and 45min for 10bp windows => 10s of hours for all samples) - are there any settings that help improve the speed? I can easily get 100-500GB RAM so memory requirements should not be an issue.

[kaizhang]

7 min per sample is actually not bad. There is not much you can do on your side. You may be able to dramatically cut down the running time by running all sample in parallel, e.g., using multiprocessing, if you have a fast file system and a sufficient number of CPU cores.

Here is an example run time for 92 samples when you process them in parallel on a Linux computer with 8 cores, 64G memory.

[kaizhang]

One more thing, setting low_memory=False in import_data may speed up things a little bit in your case.

[vitkl]

Thanks for explaining and for the suggestions. Speed for 500bp tiles is fine but 10bp is more challenging. I should probably revise how 10bp is computed - rather than tiling everything tile only peaks.

[TingTingShao]

Could I please ask what is your rational to select the bin size of 500bp and 10bp? I also heard from a seminar, but difficult to find the reference on this.

Thanks,
tingting

[TingTingShao]

Hi,

From here, I ddin't see the option to set low_memory: https://kzhang.org/SnapATAC2/api/_autosummary/snapatac2.pp.import_data.html

regards,
tingting

[kaizhang]

Hi,

From here, I ddin't see the option to set low_memory: https://kzhang.org/SnapATAC2/api/_autosummary/snapatac2.pp.import_data.html

regards, tingting

low_memory was removed as it did not significantly improve the speed but used more memory.