Image Sorting

Introduction

The Sorting.py script passes and fails images on the basis of whether or not they can be used for digitization. For example, say you had the following pictures of the bluestriped fangblenny (taken from the Fishbase species page):

While the first one shows off his winning smile, most of the fish's body is obscured, it's at an angle, and the image is instead dominated by rocks and coral. In the second image, the fish is clearly laid out, takes up most of the image, and is on a fairly uniform background; this is much better for collecting morphometric data.

Regionalization

Because fish have such a variety of colors and body forms, it isn't feasible to have the computer recognize all types of fishes and sort images that way. Instead, the script uses HSV color histograms of five basic regions of the image - four corners and the center:

The corners are defined as the four quadrants of the image masked by a central ellipse that is 80% of the height and width of the image. The center is defined by an ellipse that is 50% of the height and width of the image; these parameters can be adjusted in the colordescriptor.py script.

1. Corner uniformity

Many species pictures on Fishbase have relatively uniform, matte backgrounds, especially for posed dead specimens. One easy way to check this is with color histograms of the four corners:

Because the natural background has much more variation, there's a much wider spread of values in the second histogram. By contrast, in the first histogram, almost all the values are exactly clustered in a single bin, even for all four corners. This is checked by counting the number of peaks within a given threshold of the highest peak for every corner. On a perfectly uniform background, each corner will have only 1 high peak per corner; on a background with natural variation, there will be many peaks. Even on a relatively uniform background with some variation there are still very few peaks, although there is more than one:

2. Corner similarity

Another check is whether the corners are very similar, since fish are sometimes photographed against relatively mottled or variable backgrounds (such as against a paper towel or checkerboard). If all four corners have highly correlated color histograms, then the image is likely usable, even if that background is not perfectly uniform.

3. Corner-center comparison

Since the fish is expected to be roughly in the center of the image, the color histogram for the center of the image should look different from the color histograms on the corners. If the center has a very low correlation with the corners, there is probably a fish in the middle of the image. If the center has a very similar color histogram, we're probably not looking at a picture with a fish taking up most of the center of the image; instead, the fish could be taking up a very small portion of the image, or we're looking at something like a school of fish, neither of which is useful for digitization. For example, in the images above, the good image has a very different central color histogram (purple triangles):

While in the other image the histograms overlap a lot:

Similarly, in this image where the fish is very small, they overlap because much of the center consists of the background:

Decision tree

Behold, the dreaded flowchart:

An image has to pass a mix of strict and lenient checks based on color uniformity, corner-corner similarity, and corner-center dissimilarity. The more checks it passes, the better. An image can pass either one strict check and one lenient check, or several lenient checks. For example, the good image above scores high on corner uniformity (an average of fewer than 2 peaks per corner), so all it has to pass is a check that the center of the image is somewhat different from the corners (in this case, that the corner-center correlation coefficient is less than 0.998). This image passes one strict and one lenient check. On the other hand, we could have an image like this:

The image fails the check for uniformity (each corner has a fairly broad color histogram), but the corners are somewhat similar, and they are also fairly dissimilar from the center (purple triangles), which has minimal overlap with any of the four corners. This image passes several lenient checks but none of the strict checks.

Tweaking

The thresholds for what counts as 'very high', 'fairly high', 'fairly low,' etc, are different for each checkpoint, and have been tuned using several different fish families. If the image classifier has a high error rate, try tweaking these parameters in Sorting.py, lines 67-77 (values as of 8/16/2016):

strictMean = 1.5
softMean = 4
peakThresh = 0.5

baseCtr = 0.998
strictCtr = 0.25
softCtr = 0.11
maxCtr = 0.15

softCorr = 0.8
failCorr = 0.9

• If the classifier is too strict: try increasing any of 'Mean' or 'Ctr' parameters or decreasing the 'Corr' parameters.
• If the classifier is too lenient: try decreasing any of 'Mean' or 'Ctr' parameters or increasing the 'Corr' parameters. It is recommended that you only adjust one parameter at a time, since subtle thresholding can produce dramatic results, unless the classifier is seriously in error.