Hi @mschubert - yes, I think that should be relatively easy to do. In your case, do you use :

i) a single fixed offset into all the reads
ii) a fixed range (e.g. 10-12bp in)
iii) different or the same values for all samples?

In my case that would be i) same offset for all samples, but if the offset was determined per sample that would also work

Upon reading the guide-counter count --help page again, I realized that I misunderstood what the --offset-min-fraction does. In fact, this option already makes it possible to set a minimum amount of matches for an offset to be considered. So this already works and provides an option to limit off-target matches.

However, the value is given as a fraction of the total number of matches, not a fraction of the 100,000 reads sampled. This means that the desired value of this number changes whenever the off-target number changes. In my case, with 100,000 reads I have 75,000 matches at my desired offset. But when I run with --offset-min-fraction 0.5, I get no counts. And I don't know what to set this value to before running guide-counter, because I don't know how many off-targets I will get.

Wouldn't it make more sense to specify this parameter per reads sampled?

@mschubert My thought in having the denominator be the number of matches (instead of number of reads) is that that fraction should be consistent/predictable even with sequencing error or other problems causing the actual fraction of matching reads overall to drop.

Looking at the code, one issue I see is that it currently counts all matches from a read, so to your point, with short kmers you can end up with multiple matches per read. This is definitely not something I had anticipated. Something that is perhaps exacerbating this is that when looking for the prefixes, it is also tolerating mismatches, which obviously expands the possibility of finding multiple.

I think there are probably a few ways to fix this and I'd appreciate your input:

1. Add the ability to just specify the prefixes you want to use and skip auto-detection
2. Add the ability to restrict auto-detection to just a smaller range of the read, but still do auto-detection
3. Make auto-detection more intelligent/complicated. I think what this would look like would be: i) tallying all the possible matched offsets for a single read, ii) if a single offset is found use it, iii) if multiple offsets are found, and there are a mixture of no-mismatch and one-mismatch offsets, reduce to just the no-mismatch set, iv) instead of counting each offset as 1 match, count it instead as 1 / num_selected_matches_for_read.

(3) would make the prefix auto-detection a bit slower, but also more robust. Whereas (1) is obviously simplest. Thoughts?

@mschubert I'm not sure if you're up for checking out a branch and building, but I took a shot at implementing (3) in #14. Alternatively if you're able to share the first 25-100k of your fastq I can given it a shot too.

Thanks for your quick answer @tfenne!

I should be able to build this locally. Be aware that I'm already using --exact-match, so the no-mismatch before 1-mismatch will not affect my counts. So before I look into building this I've got a conceptual question. The way I see it, we've got two expectations of the --offset-min-fraction parameter:

• (Yours) Obvious sequencing errors should not be counted in this fraction (let's say, bacterial contamination or primer dimers)
• (Mine) Offsets should not be competitive (i.e., if half of the reads have the barcode on position 35 that should correspond to a --offset-min-fraction of 0.5, irrespective of whether the same amount of matches occurs elsewhere)

I think there are probably a few ways to fix this and I'd appreciate your input

Your proposed changes in #14 would still not align with my expectation because the counting is still competitive. On the contrary, it makes the fraction that I expect harder to estimate because I don't know where a match will be counted as 1 and where as 1/n. What about:

4. Still count each match as 1, but have a separate counter with how many reads contain a barcode at any position. Then, for computing the --offset-min-fraction divide by this read-based counter instead of the matches-based counter.

(This is your tool of course; so feel free to ignore my opinion here. Maybe there is a reason to use competitive counts that I don't yet see.)