V8: Parallel MICE

Vignette 8 of 8 · Compare to Wrapper function futuremice by Thom Benjamin Volker and Gerko Vink

42% exact · 5 match · 6 info · 0 visual · 1 skipped

This walkthrough mirrors the official R **mice** tutorials in Python. Deterministic tables and formulas are checked against the R reference; stochastic imputations and plots are labelled when they may differ.

What PyMICE does differently from R

Default randomness uses NumPy (rng="numpy"), so imputed values may differ from R unless you set rng="r".
Categorical factors are often shown as numeric codes in console output.
Diagnostic figures use matplotlib instead of lattice (same intent, different styling).
See REPRODUCIBILITY.md for exact replication options.

Parity details (maintainers)

Expected to match exactly

Checked against reference/08_futuremice/vignette_extracted.R:

Step 3 — imp$m equals 5.
Step 4 — method = 'norm' sets norm imputation for incomplete columns.
`parallelseed` reproducibility — repeated futuremice(..., parallelseed=123) yields identical pooled estimates.

Implemented (PyMICE)

futuremice() / parallel_mice() — ProcessPoolExecutor, distribute_imputations(), ibind() merge, per-worker SeedSequence streams.

Expected to differ (RNG / rendering)

Step 1 — auto-selected n.core may differ by machine; message structure matches R.
Step 2 — PyMICE Mids vs R mids class label.
Steps 5–7 — pooled tables are PyMICE-only (R snapshot has no console output); imputation RNG differs from R furrr unless parallelseed is fixed.
Step 7 — drawn parallelseed differs from R golden when no global seed is set.
Step 8 — ampute + futuremice closing demo; not a separate R snapshot block.

Skipped (R-only)

Time gain with small/large datasets — wall-clock benchmark figures are not reproduced in PyMICE.

Introduction

For big datasets or high number of imputations, performing multiple imputation with function mice from package mice (Van Buuren & Groothuis-Oudshoorn, 2011) might take a long time. As a solution, wrapper function futuremice was created to enable the imputation procedure to be run in parallel. This is done by dividing the imputations over multiple cores (or CPUs), thus potentially speeding up the process. The function futuremice is a sequel to parlMICE (Schouten & Vink, 2017), developed to improve user-friendliness.

This vignette demonstrates two applications of the futuremice function. The first application shows the tradeoff between time and increasing number of imputations ($m$) for a small dataset; the second application does the same, but for a relatively large dataset. We also discuss futuremice's arguments.

The original tutorial includes wall-clock timing benchmarks (Figures 1–2). Those sections are R-only and omitted from the PyMICE walkthrough below, which mirrors the API sections from Default settings through Argument parallelseed.

Default settings

1. Default futuremice run

Step parity: ✅ MATCH (0 exact, 1 info, 0 visual, 0 skipped, 0 mismatch of 1 blocks)

We will now discuss the arguments of function futuremice. Easy imputation of an incomplete dataset (say, nhanes) can be performed with futuremice in the following way.

Note: n.core message is informational; core count may differ by machine.

Python (PyMICE)

imp = futuremice(data, column_names=names, m=5, maxit=5, print=False)
print(core_selection_message(5))

Console Output

Number of cores not specified. Based on your machine a value of n.core = 5 is chosen; the imputations are distributed about equally over the cores.

R (Reference)

imp <- futuremice(nhanes)

R Console Output

Number of cores not specified. Based on your machine a value of n.core = 3 is chosen; the imputations are distributed about equally over the cores.

imp = futuremice(data, column_names=names, m=5, maxit=5, print=False)
print(core_selection_message(5))

Console Output

Number of cores not specified. Based on your machine a value of n.core = 5 is chosen; the imputations are distributed about equally over the cores.

imp <- futuremice(nhanes)

R Console Output

Number of cores not specified. Based on your machine a value of n.core = 3 is chosen; the imputations are distributed about equally over the cores.

The function returns a mids object as created by mice. In fact, futuremice makes use of function mice::ibind to combine the mids objects returned by the different cores. PyMICE provides futuremice() / parallel_mice() with the same API, distributing imputations across worker processes via ProcessPoolExecutor.

2. Inspect mids object

Step parity: ✅ MATCH (0 exact, 1 info, 0 visual, 0 skipped, 0 mismatch of 1 blocks)

All other parts of the mids object are standard. Inspect the imputation metadata with print(imp).

Note: PyMICE returns Mids object (same role as R mids).

Python (PyMICE)

type(imp).__name__

Console Output

[1] "Mids"

R (Reference)

class(imp)

R Console Output

[1] "mids"

type(imp).__name__

Console Output

[1] "Mids"

class(imp)

R Console Output

[1] "mids"

Argument `n.core`

3. Number of imputations

Step parity: ✅ MATCH (1 exact, 0 info, 0 visual, 0 skipped, 0 mismatch of 1 blocks)

With n.core, the number of cores (or CPUs) is given, and the number of imputations m is (about) equally distributed over the cores. As a default, m = 5, just as in a regular mice call. We can check this by evaluating the m that is shown in the mids object.

Python (PyMICE)

imp.m

Console Output

[1] 5

R (Reference)

imp$m

imp.m

Console Output

[1] 5

imp$m

Using `mice` arguments

4. Change imputation method

Step parity: ✅ MATCH (1 exact, 0 info, 0 visual, 0 skipped, 0 mismatch of 1 blocks)

Function futuremice is able to deal with the conventional mice arguments. In order to change the imputation method from its default (predictive mean matching) to, for example, Bayesian linear regression, the method argument can be adjusted.

Python (PyMICE)

format_meth_r(names, imp_norm.method, style='futuremice')

Console Output

age   bmi   hyp   chl
""   "norm"   "norm"   "norm"

R (Reference)

imp$method

R Console Output

age    bmi    hyp    chl
    "" "norm" "norm" "norm"

format_meth_r(names, imp_norm.method, style='futuremice')

Console Output

age   bmi   hyp   chl
""   "norm"   "norm"   "norm"

imp$method

R Console Output

age    bmi    hyp    chl
    "" "norm" "norm" "norm"

Argument `parallelseed`

5. Global seed reproducibility

Step parity: ✅ MATCH (1 exact, 1 info, 0 visual, 0 skipped, 0 mismatch of 2 blocks)

In simulation studies, it is often desired to set a seed to make the results reproducible. Similarly to mice, the seed value for futuremice can be defined outside the function. Hence users can specify the following code to obtain identical results.

Note: PyMICE reproducibility demo; R snapshot has no pooled table.

Python (PyMICE)

imp1 = futuremice(data, column_names=names, m=5, maxit=5, parallelseed=123, n_core=3, print=False)
format_pool_pooled_df_r(pool(with_mids(imp1, formula='chl ~ bmi')))

Console Output

         term m   estimate      ubar         b         t dfcom       df       riv    lambda       fmi
1 (Intercept) 5 108.153063 3140.9871 977.27398 4313.7159    23.0 12.02437 0.3733631 0.2718605  0.368788
2 bmi         5   3.171752    4.4191   1.51810    6.2408    23.0 11.38684 0.4122396 0.2919049  0.390341

R (Reference)

library(magrittr)
set.seed(123)
imp1 <- futuremice(nhanes, n.core = 3)
set.seed(123)
imp2 <- futuremice(nhanes, n.core = 3)
imp1 %$% lm(chl ~ bmi) %>% pool %$% pooled

imp1 = futuremice(data, column_names=names, m=5, maxit=5, parallelseed=123, n_core=3, print=False)
format_pool_pooled_df_r(pool(with_mids(imp1, formula='chl ~ bmi')))

Console Output

         term m   estimate      ubar         b         t dfcom       df       riv    lambda       fmi
1 (Intercept) 5 108.153063 3140.9871 977.27398 4313.7159    23.0 12.02437 0.3733631 0.2718605  0.368788
2 bmi         5   3.171752    4.4191   1.51810    6.2408    23.0 11.38684 0.4122396 0.2919049  0.390341

library(magrittr)
set.seed(123)
imp1 <- futuremice(nhanes, n.core = 3)
set.seed(123)
imp2 <- futuremice(nhanes, n.core = 3)
imp1 %$% lm(chl ~ bmi) %>% pool %$% pooled

Python (PyMICE)

imp2 = futuremice(data, column_names=names, m=5, maxit=5, parallelseed=123, n_core=3, print=False)
format_pool_pooled_df_r(pool(with_mids(imp2, formula='chl ~ bmi')))

Console Output

         term m   estimate      ubar         b         t dfcom       df       riv    lambda       fmi
1 (Intercept) 5 108.153063 3140.9871 977.27398 4313.7159    23.0 12.02437 0.3733631 0.2718605  0.368788
2 bmi         5   3.171752    4.4191   1.51810    6.2408    23.0 11.38684 0.4122396 0.2919049  0.390341

R (Reference)

imp2 %$% lm(chl ~ bmi) %>% pool %$% pooled

imp2 = futuremice(data, column_names=names, m=5, maxit=5, parallelseed=123, n_core=3, print=False)
format_pool_pooled_df_r(pool(with_mids(imp2, formula='chl ~ bmi')))

Console Output

         term m   estimate      ubar         b         t dfcom       df       riv    lambda       fmi
1 (Intercept) 5 108.153063 3140.9871 977.27398 4313.7159    23.0 12.02437 0.3733631 0.2718605  0.368788
2 bmi         5   3.171752    4.4191   1.51810    6.2408    23.0 11.38684 0.4122396 0.2919049  0.390341

imp2 %$% lm(chl ~ bmi) %>% pool %$% pooled

6. Parallelseed reproducibility

Step parity: ✅ MATCH (1 exact, 1 info, 0 visual, 0 skipped, 0 mismatch of 2 blocks)

A user can also specify a seed within the futuremice call, by specifying the argument parallelseed. This seed is parsed to withr::local_seed(), such that the global environment is not affected by a different seed within the futuremice function.

Note: PyMICE reproducibility demo; R snapshot has no pooled output.

Python (PyMICE)

format_pool_pooled_df_r(pool(with_mids(imp3, formula='chl ~ bmi')))

Console Output

         term m   estimate      ubar         b         t dfcom       df       riv    lambda       fmi
1 (Intercept) 5 108.153063 3140.9871 977.27398 4313.7159    23.0 12.02437 0.3733631 0.2718605  0.368788
2 bmi         5   3.171752    4.4191   1.51810    6.2408    23.0 11.38684 0.4122396 0.2919049  0.390341

R (Reference)

imp3 <- futuremice(nhanes, parallelseed = 123, n.core = 3)
imp4 <- futuremice(nhanes, parallelseed = 123, n.core = 3)
imp3 %$% lm(chl ~ bmi) %>% pool %$% pooled

format_pool_pooled_df_r(pool(with_mids(imp3, formula='chl ~ bmi')))

Console Output

         term m   estimate      ubar         b         t dfcom       df       riv    lambda       fmi
1 (Intercept) 5 108.153063 3140.9871 977.27398 4313.7159    23.0 12.02437 0.3733631 0.2718605  0.368788
2 bmi         5   3.171752    4.4191   1.51810    6.2408    23.0 11.38684 0.4122396 0.2919049  0.390341

imp3 <- futuremice(nhanes, parallelseed = 123, n.core = 3)
imp4 <- futuremice(nhanes, parallelseed = 123, n.core = 3)
imp3 %$% lm(chl ~ bmi) %>% pool %$% pooled

Python (PyMICE)

format_pool_pooled_df_r(pool(with_mids(imp4, formula='chl ~ bmi')))

Console Output

         term m   estimate      ubar         b         t dfcom       df       riv    lambda       fmi
1 (Intercept) 5 108.153063 3140.9871 977.27398 4313.7159    23.0 12.02437 0.3733631 0.2718605  0.368788
2 bmi         5   3.171752    4.4191   1.51810    6.2408    23.0 11.38684 0.4122396 0.2919049  0.390341

R (Reference)

imp4 %$% lm(chl ~ bmi) %>% pool %$% pooled

format_pool_pooled_df_r(pool(with_mids(imp4, formula='chl ~ bmi')))

Console Output

         term m   estimate      ubar         b         t dfcom       df       riv    lambda       fmi
1 (Intercept) 5 108.153063 3140.9871 977.27398 4313.7159    23.0 12.02437 0.3733631 0.2718605  0.368788
2 bmi         5   3.171752    4.4191   1.51810    6.2408    23.0 11.38684 0.4122396 0.2919049  0.390341

imp4 %$% lm(chl ~ bmi) %>% pool %$% pooled

7. Drawn parallelseed

Step parity: ✅ MATCH (1 exact, 1 info, 0 visual, 0 skipped, 0 mismatch of 2 blocks)

If no seed is specified by the user, a seed will be drawn randomly and returned in imp$parallelseed, such that the user can reproduce the obtained results even when no seed is specified.

Note: PyMICE reproducibility demo; drawn parallelseed differs from R golden.

Python (PyMICE)

format_pool_pooled_df_r(pool(with_mids(imp5, formula='chl ~ bmi')))

Console Output

         term m   estimate      ubar         b         t dfcom       df       riv    lambda       fmi
1 (Intercept) 5 108.153063 3140.9871 977.27398 4313.7159    23.0 12.02437 0.3733631 0.2718605  0.368788
2 bmi         5   3.171752    4.4191   1.51810    6.2408    23.0 11.38684 0.4122396 0.2919049  0.390341

R (Reference)

imp5 <- futuremice(nhanes, n.core = 3)
parallelseed <- imp5$parallelseed
imp6 <- futuremice(nhanes, parallelseed = parallelseed, n.core = 3)
imp5 %$% lm(chl ~ bmi) %>% pool %$% pooled

format_pool_pooled_df_r(pool(with_mids(imp5, formula='chl ~ bmi')))

Console Output

         term m   estimate      ubar         b         t dfcom       df       riv    lambda       fmi
1 (Intercept) 5 108.153063 3140.9871 977.27398 4313.7159    23.0 12.02437 0.3733631 0.2718605  0.368788
2 bmi         5   3.171752    4.4191   1.51810    6.2408    23.0 11.38684 0.4122396 0.2919049  0.390341

imp5 <- futuremice(nhanes, n.core = 3)
parallelseed <- imp5$parallelseed
imp6 <- futuremice(nhanes, parallelseed = parallelseed, n.core = 3)
imp5 %$% lm(chl ~ bmi) %>% pool %$% pooled

Python (PyMICE)

format_pool_pooled_df_r(pool(with_mids(imp6, formula='chl ~ bmi')))

Console Output

         term m   estimate      ubar         b         t dfcom       df       riv    lambda       fmi
1 (Intercept) 5 108.153063 3140.9871 977.27398 4313.7159    23.0 12.02437 0.3733631 0.2718605  0.368788
2 bmi         5   3.171752    4.4191   1.51810    6.2408    23.0 11.38684 0.4122396 0.2919049  0.390341

R (Reference)

imp6 %$% lm(chl ~ bmi) %>% pool %$% pooled

format_pool_pooled_df_r(pool(with_mids(imp6, formula='chl ~ bmi')))

Console Output

         term m   estimate      ubar         b         t dfcom       df       riv    lambda       fmi
1 (Intercept) 5 108.153063 3140.9871 977.27398 4313.7159    23.0 12.02437 0.3733631 0.2718605  0.368788
2 bmi         5   3.171752    4.4191   1.51810    6.2408    23.0 11.38684 0.4122396 0.2919049  0.390341

imp6 %$% lm(chl ~ bmi) %>% pool %$% pooled

Ampute and parallel imputation

8. Ampute then impute

Step parity: ⏭️ SKIP (0 exact, 1 info, 0 visual, 1 skipped, 0 mismatch of 2 blocks)

The original vignette also demonstrates timing benchmarks on simulated data (Figures 1–2). Those wall-clock comparisons are R-only and skipped here. As a closing example, we ampute a small multivariate dataset and impute it with the same PyMICE workflow.

Python (PyMICE)

# Skipped — R-only timing figures; not reproduced in PyMICE

Console Output

(not run — )

R (Reference)

# Figures 1–2: wall-clock benchmarks (small vs large simulated data)

# Skipped — R-only timing figures; not reproduced in PyMICE

Console Output

(not run — )

# Figures 1–2: wall-clock benchmarks (small vs large simulated data)

Note: PyMICE closing demo (ampute + parallel impute); no R snapshot output.

Python (PyMICE)

amputed_res = ampute(small_data, prop=0.8, mech='MCAR', seed=123)
imp_amp = futuremice(amputed_res.amp, column_names=names, m=5, maxit=5, print=False)
print(f'[1] {imp_amp.m}')

Console Output

[1] 5

R (Reference)

imp_amp <- futuremice(ampute(nhanes, prop = 0.8, mech = 'MCAR')$amp)

amputed_res = ampute(small_data, prop=0.8, mech='MCAR', seed=123)
imp_amp = futuremice(amputed_res.amp, column_names=names, m=5, maxit=5, print=False)
print(f'[1] {imp_amp.m}')

Console Output

[1] 5

imp_amp <- futuremice(ampute(nhanes, prop = 0.8, mech = 'MCAR')$amp)