Reading and storing check points (restart files)

readCheckPoints.f90

Description

This module contains the functions that can help reading and mapping of the restart files

Quick access

Variables

bytes_allocated, l_axi_old, lreadr, lreads, lreadxi, n_start_file, ratio1, ratio1_old, ratio2, ratio2_old

Routines

finish_start_fields(), getlm2lmo(), mapdatahydro(), mapdatamag(), mapdatar(), maponefield(), maponefield_mpi(), read_map_one_field(), read_map_one_field_mpi(), read_map_one_scalar(), read_map_one_scalar_mpi(), readstartfields(), readstartfields_mpi(), readstartfields_old()

Needed modules

Variables

  • readcheckpoints/bytes_allocated [integer,private/optional/default=0]
  • readcheckpoints/l_axi_old [logical,private]
  • readcheckpoints/lreadr [logical,private]
  • readcheckpoints/lreads [logical,private]
  • readcheckpoints/lreadxi [logical,private]
  • readcheckpoints/n_start_file [integer,private]
  • readcheckpoints/ratio1 [real,private]
  • readcheckpoints/ratio1_old [real,private]
  • readcheckpoints/ratio2 [real,private]
  • readcheckpoints/ratio2_old [real,private]

Subroutines and functions

subroutine readcheckpoints/readstartfields_old(w, dwdt, z, dzdt, p, dpdt, s, dsdt, xi, dxidt, phi, dphidt, b, dbdt, aj, djdt, b_ic, dbdt_ic, aj_ic, djdt_ic, omega_ic, omega_ma, domega_ic_dt, domega_ma_dt, lorentz_torque_ic_dt, lorentz_torque_ma_dt, time, tscheme, n_time_step)

This subroutine is used to read the old restart files produced by MagIC. This is now deprecated with the change of the file format. This is still needed to read old files.

Parameters
  • w (ulm-(llm)+1,n_r_max) [complex ,out]

  • dwdt [type_tarray ,inout]

  • z (ulm-(llm)+1,n_r_max) [complex ,out]

  • dzdt [type_tarray ,inout]

  • p (ulm-(llm)+1,n_r_max) [complex ,out]

  • dpdt [type_tarray ,inout]

  • s (ulm-(llm)+1,n_r_max) [complex ,out]

  • dsdt [type_tarray ,inout]

  • xi (ulm-(llm)+1,n_r_max) [complex ,out]

  • dxidt [type_tarray ,inout]

  • phi (ulm-(llm)+1,n_r_max) [complex ,out]

  • dphidt [type_tarray ,inout]

  • b (ulmmag-(llmmag)+1,n_r_maxmag) [complex ,out]

  • dbdt [type_tarray ,inout]

  • aj (ulmmag-(llmmag)+1,n_r_maxmag) [complex ,out]

  • djdt [type_tarray ,inout]

  • b_ic (ulmmag-(llmmag)+1,n_r_ic_maxmag) [complex ,out]

  • dbdt_ic [type_tarray ,inout]

  • aj_ic (ulmmag-(llmmag)+1,n_r_ic_maxmag) [complex ,out]

  • djdt_ic [type_tarray ,inout]

  • omega_ic [real ,out]

  • omega_ma [real ,out]

  • domega_ic_dt [type_tscalar ,inout]

  • domega_ma_dt [type_tscalar ,inout]

  • lorentz_torque_ic_dt [type_tscalar ,inout]

  • lorentz_torque_ma_dt [type_tscalar ,inout]

  • time [real ,out]

  • tscheme [real ]

  • n_time_step [integer ,out]

Called from

getstartfields()

Call to

abortrun(), getlm2lmo(), mapdatahydro(), scatter_from_rank0_to_lo(), get_single_rhs_imp(), get_pol_rhs_imp(), get_entropy_rhs_imp(), get_tor_rhs_imp(), get_comp_rhs_imp(), mapdatamag(), get_mag_rhs_imp(), get_mag_ic_rhs_imp(), finish_start_fields()

subroutine readcheckpoints/readstartfields(w, dwdt, z, dzdt, p, dpdt, s, dsdt, xi, dxidt, phi, dphidt, b, dbdt, aj, djdt, b_ic, dbdt_ic, aj_ic, djdt_ic, omega_ic, omega_ma, domega_ic_dt, domega_ma_dt, lorentz_torque_ic_dt, lorentz_torque_ma_dt, time, tscheme, n_time_step)

This subroutine is used to read the restart files produced by MagIC.

Parameters
  • w (ulm-(llm)+1,n_r_max) [complex ,out]

  • dwdt [type_tarray ,inout]

  • z (ulm-(llm)+1,n_r_max) [complex ,out]

  • dzdt [type_tarray ,inout]

  • p (ulm-(llm)+1,n_r_max) [complex ,out]

  • dpdt [type_tarray ,inout]

  • s (ulm-(llm)+1,n_r_max) [complex ,out]

  • dsdt [type_tarray ,inout]

  • xi (ulm-(llm)+1,n_r_max) [complex ,out]

  • dxidt [type_tarray ,inout]

  • phi (ulm-(llm)+1,n_r_max) [complex ,out]

  • dphidt [type_tarray ,inout]

  • b (ulmmag-(llmmag)+1,n_r_maxmag) [complex ,out]

  • dbdt [type_tarray ,inout]

  • aj (ulmmag-(llmmag)+1,n_r_maxmag) [complex ,out]

  • djdt [type_tarray ,inout]

  • b_ic (ulmmag-(llmmag)+1,n_r_ic_maxmag) [complex ,out]

  • dbdt_ic [type_tarray ,inout]

  • aj_ic (ulmmag-(llmmag)+1,n_r_ic_maxmag) [complex ,out]

  • djdt_ic [type_tarray ,inout]

  • omega_ic [real ,out]

  • omega_ma [real ,out]

  • domega_ic_dt [type_tscalar ,inout]

  • domega_ma_dt [type_tscalar ,inout]

  • lorentz_torque_ic_dt [type_tscalar ,inout]

  • lorentz_torque_ma_dt [type_tscalar ,inout]

  • time [real ,out]

  • tscheme [real ]

  • n_time_step [integer ,out]

Called from

readstartfields_mpi()

Call to

abortrun(), print_info(), getlm2lmo(), read_map_one_scalar(), read_map_one_field(), maponefield(), scatter_from_rank0_to_lo(), finish_start_fields(), get_single_rhs_imp(), get_pol_rhs_imp(), get_entropy_rhs_imp(), get_tor_rhs_imp(), get_comp_rhs_imp(), get_mag_rhs_imp(), get_mag_ic_rhs_imp()

subroutine readcheckpoints/read_map_one_scalar(fh, tscheme, nexp_old, nimp_old, nold_old, tscheme_family_old, dscal_dt)

– Input variables

Parameters
  • fh [integer ,in]

  • tscheme [real ]

  • nexp_old [integer ,in]

  • nimp_old [integer ,in]

  • nold_old [integer ,in]

  • tscheme_family_old [character(len=*),in]

  • dscal_dt [type_tscalar ,inout]

Called from

readstartfields()

subroutine readcheckpoints/read_map_one_field(fh, tscheme, wold, work, scale_w, r_old, lm2lmo, n_r_max_old, n_r_maxl, dim1, nexp_old, nimp_old, nold_old, tscheme_family_old, w, dwdt, l_map, l_transp)

— Input variables

Parameters
  • fh [integer ,in]

  • tscheme [real ]

  • wold (*,*) [complex ,inout]

  • work (*,*) [complex ,inout]

  • scale_w [real ,in]

  • r_old (*) [real ,in]

  • lm2lmo (lm_max) [integer ,in]

  • n_r_max_old [integer ,in]

  • n_r_maxl [integer ,in]

  • dim1 [integer ,in]

  • nexp_old [integer ,in]

  • nimp_old [integer ,in]

  • nold_old [integer ,in]

  • tscheme_family_old [character(len=*),in]

  • w (ulm-(llm)+1,dim1) [complex ,out]

  • dwdt [type_tarray ,inout]

  • l_map [logical ,in]

  • l_transp [logical ,in] :: do we need to transpose to lm-loc

Called from

readstartfields()

Call to

maponefield(), scatter_from_rank0_to_lo()

subroutine readcheckpoints/readstartfields_mpi(w, dwdt, z, dzdt, p, dpdt, s, dsdt, xi, dxidt, phi, dphidt, b, dbdt, aj, djdt, b_ic, dbdt_ic, aj_ic, djdt_ic, omega_ic, omega_ma, domega_ic_dt, domega_ma_dt, lorentz_torque_ic_dt, lorentz_torque_ma_dt, time, tscheme, n_time_step)

This subroutine is used to read the restart files produced by MagIC using MPI-IO

Parameters
  • w (ulm-(llm)+1,n_r_max) [complex ,out]

  • dwdt [type_tarray ,inout]

  • z (ulm-(llm)+1,n_r_max) [complex ,out]

  • dzdt [type_tarray ,inout]

  • p (ulm-(llm)+1,n_r_max) [complex ,out]

  • dpdt [type_tarray ,inout]

  • s (ulm-(llm)+1,n_r_max) [complex ,out]

  • dsdt [type_tarray ,inout]

  • xi (ulm-(llm)+1,n_r_max) [complex ,out]

  • dxidt [type_tarray ,inout]

  • phi (ulm-(llm)+1,n_r_max) [complex ,out]

  • dphidt [type_tarray ,inout]

  • b (ulmmag-(llmmag)+1,n_r_maxmag) [complex ,out]

  • dbdt [type_tarray ,inout]

  • aj (ulmmag-(llmmag)+1,n_r_maxmag) [complex ,out]

  • djdt [type_tarray ,inout]

  • b_ic (ulmmag-(llmmag)+1,n_r_ic_maxmag) [complex ,out]

  • dbdt_ic [type_tarray ,inout]

  • aj_ic (ulmmag-(llmmag)+1,n_r_ic_maxmag) [complex ,out]

  • djdt_ic [type_tarray ,inout]

  • omega_ic [real ,out]

  • omega_ma [real ,out]

  • domega_ic_dt [type_tscalar ,inout]

  • domega_ma_dt [type_tscalar ,inout]

  • lorentz_torque_ic_dt [type_tscalar ,inout]

  • lorentz_torque_ma_dt [type_tscalar ,inout]

  • time [real ,out]

  • tscheme [real ]

  • n_time_step [integer ,out]

Called from

getstartfields()

Call to

mpiio_setup(), abortrun(), readstartfields(), print_info(), getlm2lmo(), read_map_one_scalar_mpi(), getblocks(), read_map_one_field_mpi(), maponefield(), scatter_from_rank0_to_lo(), get_single_rhs_imp(), get_pol_rhs_imp(), get_entropy_rhs_imp(), get_tor_rhs_imp(), get_comp_rhs_imp(), get_mag_rhs_imp(), get_mag_ic_rhs_imp(), finish_start_fields()

subroutine readcheckpoints/read_map_one_scalar_mpi(fh, tscheme, nexp_old, nimp_old, nold_old, tscheme_family_old, dscal_dt)

– Input variables

Parameters
  • fh [integer ,in]

  • tscheme [real ]

  • nexp_old [integer ,in]

  • nimp_old [integer ,in]

  • nold_old [integer ,in]

  • tscheme_family_old [character(len=*),in]

  • dscal_dt [type_tscalar ,inout]

Called from

readstartfields_mpi()

subroutine readcheckpoints/read_map_one_field_mpi(fh, info, datatype, tscheme, wold, lm_max_old, n_r_max_old, nrstart_old, nrstop_old, radial_balance_old, lm2lmo, r_old, n_r_maxl, dim1, scale_w, nexp_old, nimp_old, nold_old, tscheme_family_old, w, dwdt, disp, l_map, l_transp[, l_read_field_only])

— Input variables

Parameters
  • fh [integer ,in]

  • info [integer ,in]

  • datatype [integer ,in]

  • tscheme [real ]

  • wold (lm_max_old,nrstop_old-nrstart_old+1) [complex ,in]

  • lm_max_old [integer ,in,]

  • n_r_max_old [integer ,in]

  • nrstart_old [integer ,in]

  • nrstop_old [integer ,in]

  • radial_balance_old (n_procs-1+1) [load ,in]

  • lm2lmo (lm_max) [integer ,in]

  • r_old (*) [real ,in]

  • n_r_maxl [integer ,in]

  • dim1 [integer ,in]

  • scale_w [real ,in]

  • nexp_old [integer ,in]

  • nimp_old [integer ,in]

  • nold_old [integer ,in]

  • tscheme_family_old [character(len=*),in]

  • w (ulm-(llm)+1,dim1) [complex ,out]

  • dwdt [type_tarray ,inout]

  • disp [integer ,inout]

  • l_map [logical ,in]

  • l_transp [logical ,in] :: Do we need to transpose d?dt arrays?

Options

l_read_field_only [logical ,in,]

Called from

readstartfields_mpi()

Call to

maponefield_mpi()

subroutine readcheckpoints/getlm2lmo(n_r_max, n_r_max_old, l_max, l_max_old, m_min, m_min_old, m_max, m_max_old, minc, minc_old, lm_max, lm_max_old, lm2lmo)

— Input variables

Parameters
  • n_r_max [integer ,in]

  • n_r_max_old [integer ,in]

  • l_max [integer ,in]

  • l_max_old [integer ,in]

  • m_min [integer ,in]

  • m_min_old [integer ,in]

  • m_max [integer ,in]

  • m_max_old [integer ,inout]

  • minc [integer ,in]

  • minc_old [integer ,in]

  • lm_max [integer ,in]

  • lm_max_old [integer ,out]

  • lm2lmo (lm_max) [integer ,out] :: data found in startfile

Called from

readstartfields_old(), readstartfields(), readstartfields_mpi()

subroutine readcheckpoints/maponefield_mpi(wold, lm_max_old, n_r_max_old, nrstart_old, nrstop_old, radial_balance_old, lm2lmo, r_old, n_r_maxl, dim1, lbc1, l_ic, scale_w, w)

— Input variables

Parameters
  • wold (lm_max_old,nrstop_old-nrstart_old+1) [complex ,in]

  • lm_max_old [integer ,in,]

  • n_r_max_old [integer ,in]

  • nrstart_old [integer ,in]

  • nrstop_old [integer ,in]

  • radial_balance_old (n_procs-1+1) [load ,in]

  • lm2lmo (lm_max) [integer ,in]

  • r_old (*) [real ,in]

  • n_r_maxl [integer ,in]

  • dim1 [integer ,in]

  • lbc1 [logical ,in]

  • l_ic [logical ,in]

  • scale_w [real ,in]

  • w (ulm-(llm)+1,dim1) [complex ,out]

Called from

read_map_one_field_mpi()

Call to

mapdatar()

subroutine readcheckpoints/maponefield(wo, scale_w, r_old, lm2lmo, n_r_max_old, n_r_maxl, dim1, lbc1, l_ic, w)

— Input variables

Parameters
  • wo (*,*) [complex ,in]

  • scale_w [real ,in]

  • r_old (*) [real ,in]

  • lm2lmo (lm_max) [integer ,in]

  • n_r_max_old [integer ,in]

  • n_r_maxl [integer ,in]

  • dim1 [integer ,in]

  • lbc1 [logical ,in]

  • l_ic [logical ,in]

  • w (lm_max,dim1) [complex ,out]

Called from

readstartfields(), read_map_one_field(), readstartfields_mpi()

Call to

mapdatar()

subroutine readcheckpoints/mapdatahydro(wo, zo, po, so, xio, r_old, lm2lmo, n_r_max_old, lm_max_old, n_r_maxl, lbc1, lbc2, lbc3, lbc4, lbc5, w, z, p, s, xi)

— Input variables

Parameters
  • wo (lm_max_old,n_r_max_old) [complex ,in]

  • zo (lm_max_old,n_r_max_old) [complex ,in]

  • po (lm_max_old,n_r_max_old) [complex ,in]

  • so (lm_max_old,n_r_max_old) [complex ,in]

  • xio (lm_max_old,n_r_max_old) [complex ,in]

  • r_old (*) [real ,in]

  • lm2lmo (lm_max) [integer ,in]

  • n_r_max_old [integer ,in,]

  • lm_max_old [integer ,in,]

  • n_r_maxl [integer ,in]

  • lbc1 [logical ,in]

  • lbc2 [logical ,in]

  • lbc3 [logical ,in]

  • lbc4 [logical ,in]

  • lbc5 [logical ,in]

  • w (lm_max,n_r_max) [complex ,out]

  • z (lm_max,n_r_max) [complex ,out]

  • p (lm_max,n_r_max) [complex ,out]

  • s (lm_max,n_r_max) [complex ,out]

  • xi (lm_max,n_r_max) [complex ,out]

Called from

readstartfields_old()

Call to

mapdatar()

subroutine readcheckpoints/mapdatamag(wo, zo, po, so, r_old, n_rad_tot, n_r_max_old, lm_max_old, n_r_maxl, lm2lmo, dim1, l_ic, w, z, p, s)

— Input variables

Parameters
  • wo (lm_max_old,n_r_max_old) [complex ,in]

  • zo (lm_max_old,n_r_max_old) [complex ,in]

  • po (lm_max_old,n_r_max_old) [complex ,in]

  • so (lm_max_old,n_r_max_old) [complex ,in]

  • r_old (*) [real ,in]

  • n_rad_tot [integer ,in]

  • n_r_max_old [integer ,in,]

  • lm_max_old [integer ,in,]

  • n_r_maxl [integer ,in]

  • lm2lmo (lm_max) [integer ,in]

  • dim1 [integer ,in]

  • l_ic [logical ,in]

  • w (lm_maxmag,dim1) [complex ,out]

  • z (lm_maxmag,dim1) [complex ,out]

  • p (lm_maxmag,dim1) [complex ,out]

  • s (lm_maxmag,dim1) [complex ,out]

Called from

readstartfields_old()

Call to

mapdatar()

subroutine readcheckpoints/mapdatar(datar, r_old, n_rad_tot, n_r_max_old, n_r_maxl, lbc, l_ic)

Copy (interpolate) data (read from disc file) from old grid structure to new grid. Linear interploation is used in r if the radial grid structure differs

called in mapdata

Parameters
  • datar (*) [complex ,inout] :: old data

  • r_old (*) [real ,in]

  • n_rad_tot [integer ,in]

  • n_r_max_old [integer ,in]

  • n_r_maxl [integer ,in]

  • lbc [logical ,in]

  • l_ic [logical ,in]

Called from

maponefield_mpi(), maponefield(), mapdatahydro(), mapdatamag()

Call to

polynomial_interpolation()

subroutine readcheckpoints/finish_start_fields(time, minc_old, l_mag_old, omega_ic1old, omega_ma1old, z, s, xi, b, omega_ic, omega_ma)

– Input variables

Parameters
  • time [real ,in]

  • minc_old [integer ,in]

  • l_mag_old [logical ,in]

  • omega_ic1old [real ,in]

  • omega_ma1old [real ,in]

  • z (ulm-(llm)+1,n_r_max) [complex ,inout]

  • s (ulm-(llm)+1,n_r_max) [complex ,inout]

  • xi (ulm-(llm)+1,n_r_max) [complex ,inout]

  • b (ulmmag-(llmmag)+1,n_r_maxmag) [complex ,inout]

  • omega_ic [real ,out]

  • omega_ma [real ,out]

Called from

readstartfields_old(), readstartfields(), readstartfields_mpi()

subroutine readcheckpoints/print_info(ra_old, ek_old, pr_old, sc_old, raxi_old, pm_old, radratio_old, sigma_ratio_old, n_phi_tot_old, nalias_old, l_max_old)

– Input variables

Parameters
  • ra_old [real ,in]

  • ek_old [real ,in]

  • pr_old [real ,in]

  • sc_old [real ,in]

  • raxi_old [real ,in]

  • pm_old [real ,in]

  • radratio_old [real ,in]

  • sigma_ratio_old [real ,in]

  • n_phi_tot_old [integer ,in]

  • nalias_old [integer ,in]

  • l_max_old [integer ,in]

storeCheckPoints.f90

Description

This module contains several subroutines that can be used to store the checkpoint_#.tag files

Quick access

Routines

store(), store_mpi(), write_one_field(), write_one_field_mpi()

Needed modules

Variables

Subroutines and functions

subroutine storecheckpoints/store(time, tscheme, n_time_step, l_stop_time, l_new_rst_file, l_ave_file, w, z, p, s, xi, phi, b, aj, b_ic, aj_ic, dwdt, dzdt, dpdt, dsdt, dxidt, dphidt, dbdt, djdt, dbdt_ic, djdt_ic, domega_ma_dt, domega_ic_dt, lorentz_torque_ma_dt, lorentz_torque_ic_dt)

This subroutine stores the results in a checkpoint file. In addition to the magnetic field and velocity potentials we also store the time derivative terms djdt(lm,nR),dbdt(lm,nR), … to allow to restart with 2nd order Adams-Bashforth scheme. To minimize the memory imprint, a gather/write strategy has been adopted here. This implies that only one global array dimension(lm_max,n_r_max) is required.

Parameters
  • time [real ,in]

  • tscheme [real ]

  • n_time_step [integer ,in]

  • l_stop_time [logical ,in]

  • l_new_rst_file [logical ,in]

  • l_ave_file [logical ,in]

  • w (ulm-(llm)+1,n_r_max) [complex ,in]

  • z (ulm-(llm)+1,n_r_max) [complex ,in]

  • p (ulm-(llm)+1,n_r_max) [complex ,in]

  • s (ulm-(llm)+1,n_r_max) [complex ,in]

  • xi (ulm-(llm)+1,n_r_max) [complex ,in]

  • phi (ulm-(llm)+1,n_r_max) [complex ,in]

  • b (ulmmag-(llmmag)+1,n_r_maxmag) [complex ,in]

  • aj (ulmmag-(llmmag)+1,n_r_maxmag) [complex ,in]

  • b_ic (ulmmag-(llmmag)+1,n_r_ic_maxmag) [complex ,in]

  • aj_ic (ulmmag-(llmmag)+1,n_r_ic_maxmag) [complex ,in]

  • dwdt [type_tarray ,in]

  • dzdt [type_tarray ,in]

  • dpdt [type_tarray ,in]

  • dsdt [type_tarray ,in]

  • dxidt [type_tarray ,in]

  • dphidt [type_tarray ,in]

  • dbdt [type_tarray ,in]

  • djdt [type_tarray ,in]

  • dbdt_ic [type_tarray ,in]

  • djdt_ic [type_tarray ,in]

  • domega_ma_dt [type_tscalar ,in]

  • domega_ic_dt [type_tscalar ,in]

  • lorentz_torque_ma_dt [type_tscalar ,in]

  • lorentz_torque_ic_dt [type_tscalar ,in]

Call to

dble2str(), write_one_field(), gather_all_from_lo_to_rank0()

subroutine storecheckpoints/write_one_field(fh, tscheme, w, dwdt, work)
Parameters
  • fh [integer ,in] :: file unit

  • tscheme [real ] :: Time scheme

  • w (ulm-(llm)+1,n_r_max) [complex ,in] :: field

  • dwdt [type_tarray ,in]

  • work (*,*) [complex ,inout]

Called from

graphout_mpi(), graphout_ic(), store()

Call to

gather_all_from_lo_to_rank0()

subroutine storecheckpoints/store_mpi(time, tscheme, n_time_step, l_stop_time, l_new_rst_file, l_ave_file, w, z, p, s, xi, phi, b, aj, b_ic, aj_ic, dwdt, dzdt, dpdt, dsdt, dxidt, dphidt, dbdt, djdt, dbdt_ic, djdt_ic, domega_ma_dt, domega_ic_dt, lorentz_torque_ma_dt, lorentz_torque_ic_dt)

This subroutine stores the results in a checkpoint file. In addition to the magnetic field and velocity potentials we also store the time derivative terms djdt(lm,nR),dbdt(lm,nR), … to allow to restart with 2nd order Adams-Bashforth scheme. To minimize the memory imprint, a gather/write strategy has been adopted here. This implies that only one global array dimension(lm_max,n_r_max) is required.

Parameters
Called from

fields_average(), output()

Call to

dble2str(), mpiio_setup(), write_one_field_mpi(), gather_from_lo_to_rank0()

subroutine storecheckpoints/write_one_field_mpi(fh, info, datatype, tscheme, w, dwdt, work, size_tmp, disp, l_transp)

This subroutine is used to write one field and its associated possible help arrays (d?dt) which are required to time advance the solution. This is using MPI-IO with R-distributed arrays.

Parameters
  • fh [integer ,in] :: file unit

  • info [integer ,in] :: MPI file info

  • datatype [integer ,in] :: MPI file info

  • tscheme [real ]

  • w (ulm-(llm)+1,n_r_max) [complex ,in]

  • dwdt [type_tarray ,in] :: time advance arrays

  • work (ulm-(llm)+1,n_r_max) [complex ,inout]

  • size_tmp [integer ,in]

  • disp [integer ,inout]

  • l_transp [logical ,in] :: Do we need to transpose anything?

Called from

store_mpi()