Fix size mismatch error when CatToNumTransform sees only a subset of labels at test time
#2036
| Job | Run time |
|---|---|
| 1m 43s | |
| 1m 48s | |
| 1m 47s | |
| 1m 49s | |
| 1m 48s | |
| 8m 55s |