How to solve error: Error in nnet.default(X, Y, w, mask = mask, size = 0, skip = TRUE, softmax = TRUE, : too many (1666) weights?

[Bigsealion]

I have a matrix that shape is 7000*165, and I want to impute it by mice. The max NA rate in columns is 47%, and in row is 49% (I have del col and row that NA rate > 50%). Total NA rate in data is 7%. There is 32 factor variable and another is numeric.
When mice is running, an error occurred. Can you tell me what should I do?
Another question, when most variable class are 'factor', the speed of imputation will be very very slow, even error. Can it be solved? Thanks!

 Error in nnet.default(X, Y, w, mask = mask, size = 0, skip = TRUE, softmax = TRUE,  : 
  too many (1666) weights 
10.
stop(gettextf("too many (%d) weights", nwts), domain = NA) 
9.
nnet.default(X, Y, w, mask = mask, size = 0, skip = TRUE, softmax = TRUE, 
    censored = censored, rang = 0, ...) 
8.
nnet::multinom(formula(xy), data = xy[ry, , drop = FALSE], weights = w[ry], 
    maxit = nnet.maxit, trace = nnet.trace, MaxNWts = nnet.MaxNWts, 
    ...) 
7.
mice.impute.polyreg(y = structure(c(6L, 1L, 1L, 1L, 1L, 5L, 2L, 
1L, 4L, 5L, 1L, 1L, 7L, 7L, 2L, 2L, 1L, 2L, 1L, 2L, 4L, 7L, 3L, 
7L, 2L, 1L, 4L, 7L, 7L, 7L, 3L, 3L, 7L, 2L, 4L, 4L, 6L, 1L, 4L, 
7L, 7L, 6L, 7L, 1L, 6L, 4L, 1L, 6L, 4L, 1L, 1L, 2L, 3L, 1L, 7L,  ... 
6.
do.call(f, args = args) 
5.
sampler.univ(data = data, r = r, where = where, type = type, 
    formula = ff, method = theMethod, yname = j, k = k, calltype = calltype, 
    user = user, ...) 
4.
sampler(data, m, where, imp, blocks, method, visitSequence, predictorMatrix, 
    formulas, blots, post, c(from, to), printFlag, ...) 
3.
mice(df_imp, m = imput_num, seed = imp_seed, predictorMatrix = pred) at mice_imputation_v1.R#223

[stefvanbuuren]

Related #111.

It might be that the number of categories is a little too much for polyreg. As an alternative try pmm for the categorical variables, e.g., set the argument method = "pmm". Any luck with that?

[stefvanbuuren]

See also the discussion in https://stefvanbuuren.name/fimd/sec-categorical.html#evaluation

[Bigsealion]

Related #111.

It might be that the number of categories is a little too much for polyreg. As an alternative try pmm for the categorical variables, e.g., set the argument method = "pmm". Any luck with that?

Thank you for your quick reply!
It's so unfortunately that the speed of impute is still slow even thought I have set method as 'pmm'. Maybe the factor variable is too much (there are 365 factor variables in total 382 variables, and the case number is 4900). Can I set factor variables as num to impute (in fact, most of them is level num, but class level is slowly also)? Would that be an unacceptable mistake?
another question, I find when the imputation is running, some variables name will have a '*', what does that means? Thanks!

[stefvanbuuren]

1. Using pmm for factors is easy. Do something like:

library(mice)
str(boys)
mice(boys, method = "pmm", print = FALSE)

2. Perhaps you don't need all variables in your analysis, and could trim the problem. Imputing 400 variables is not a routine problem.

3. Or, use quickpred() to trim imputation models.

[Bigsealion]

1. Using pmm for factors is easy. Do something like:

library(mice)
str(boys)
mice(boys, method = "pmm", print = FALSE)

1. Perhaps you don't need all variables in your analysis, and could trim the problem. Imputing 400 variables is not a routine problem.
2. Or, use quickpred() to trim imputation models.

Maybe I need to adjust my data. And, I want to know, dose mice have potentially difficult in large data (like 5000*400)? I would be very grateful if you could provide me with a literature about performance of mice. Thanks!

[stefvanbuuren]

Blindly running mice(data) on a dataset of 5000 * 400 is not recommended. Blind imputation is slow and the results will be disappointing. It is slow because it runs 5 (maxit) * 5 (number of imputations) * 400 (variables) regressions, each probably having a 1000+ predictors (because categorical data are expanded). The results will be disappointing: mice will try very hard at each iteration to solve the inherent multicollinearity problems (it will print a star next to each variable name), but there's a limit how far you can go.

In general, it pays huge benefits, both in terms of speed and quality, if you spend some time in building the imputation model. Probably the most important thing you can do is downsize the number of predictors in each model to the most important, say, 20-30 variables. See https://stefvanbuuren.name/fimd/sec-modelform.html#sec:predictors for some guidelines on how to do that.

Your reward will be a high-quality and speedy model without warning stars that will easily scale to larger datasets.