Imputation in high dimensional data by mice package

[bugravarol]

Hello everybody. In the mice package, I simulated a high-dimensional data and generated missing structures in the first 50 variables (25% per variable). Later, when I want to imput these missing values by the methods in mice, I get warnings as follows. I searched for this but couldn't figure it out. I added the term (ls.meth="ridge") inside the function but I keep getting the warning

I couldn't find where I made a mistake. How can I make correct imputations using the mice package for a high-dimensional dataset that I have simulated in this way?

                                                         out

1
df set to 1. # observed cases: 63 # predictors: 120

2 V2, V3, V4, V6, V9, V14, V15, V17, V19, V23, V25, V26, V28, V30, V31, V34, V36, V38, V39, V46, V49, V52, V53, V54, V64, V66, V67, V68, V70, V73, V79, V82, V84, V89, V91, V93, V94, V95, V99, V100, V101, V106, V107, V108, V109, V111, V117, V120

3 mice detected that your data are (nearly) multi-collinear.\nIt applied a ridge penalty to continue calculations, but the results can be unstable.\nDoes your dataset contain duplicates, linear transformation, or factors with unique respondent names?

4
df set to 1. # observed cases: 69 # predictors: 120

5 V1, V3, V4, V5, V14, V21, V22, V25, V27, V30, V35, V37, V41, V43, V44, V47, V51, V53, V55, V58, V59, V60, V61, V68, V71, V72, V81, V82, V84, V90, V93, V94, V97, V101, V105, V108, V111, V117

6 mice detected that your data are (nearly) multi-collinear.\nIt applied a ridge penalty to continue calculations, but the results can be unstable.\nDoes your dataset contain duplicates, linear transformation, or factors with unique respondent names?

## MY CODES

install.packages("MASS")
install.packages("stats")
install.packages("mice")

library(MASS)
library(stats)
library(mice)

######################################################
# DATA FUNCTION
######################################################
#  rm(list = ls())
generateData<- function(n,p) {
pr <- seq(0.80, 0.40, length.out = p)
pr[1] <- 1
covmat <- toeplitz(pr)
mu= rep(0,p)
X_ <- data.frame(mvrnorm(n, mu = mu, Sigma = covmat))
X <- unname(as.matrix(sample(X_)))
vCoef = rnorm(ncol(X))
vProb =exp(X%*%vCoef)/(1+exp(X%*%vCoef))
Y <- rbinom(nrow(X), 1, vProb)
mydata= data.frame(cbind(X,Y))
return(mydata)
}

######################################################
# SIMULATED DATA
######################################################
n <- 100
p <- 120
data <- generateData(n , p)
# table(data[ncol(data)])
X <- data[-ncol(data)]
Y <- data[ncol(data)]

###### PATTERN #######
myfreq <- 0.25
pstar <- 50
npat <- 120
mypatterns <- matrix(1, nrow = npat, ncol = p)
for(i in 1:npat){
  idx <- sample(x = 1:pstar, size = myfreq * n, replace = F)
  mypatterns[i,idx] <- 0
}

####### MCAR MECHANISM #########
my_bycases <- TRUE
my_prop <- 0.5
data_mcar <- ampute(X, prop = my_prop, pattern=mypatterns, mech ='MCAR', type="RIGHT", bycases=my_bycases)
data_mcar_missing <- data_mcar$amp

####### IMPPUTATION #########
complete_datasets <- mice(data_mcar_missing, m = 2, defaultMethod = "pmm")
all_imp_data <- mice::complete(complete_datasets, "all")

[hanneoberman]

My hunch is that the covariances in the covmat are too high. What happens if you rerun the simulation with lower correlations between variables?

[bugravarol]

First of all, thank you for your interest in my question. I changed the ranges for covmat from 0.10 to 0.30 but I keep getting the same warnings. Could it be a problem with high dimensionality?

[stefvanbuuren]

Trying to reproduce but getting

> data_mcar <- ampute(X, prop = my_prop, pattern=mypatterns, mech ='MCAR', type="RIGHT", bycases=my_bycases)
Error in ampute(X, prop = my_prop, pattern = mypatterns, mech = "MCAR",  : 
  object 'X' not found

Please check your script in a fresh session.

[bugravarol]

I am so sorry. I revised my codes.

[stefvanbuuren]

Please do not cross-post #477. There is now also an answer there.