Jpdiaz: fmriprep as job array example

2020-09-29T04:34:02Z

fmriprep as job array example

New page

'''running fmriprep-20.1.1 as a job array
'''
Job array is a type of batch mode. It makes it possible to process different subjects using the same script on multiple Hoffman2 working nodes at the same time (or as nodes become available).
It can be used to submit hundreds of subjects to be processed at once.
This can be done using two scripts, submit_fmriprep_jobarray.sh and run_fmriprep.sh.
The first script, submit_fmriprep_jobarray.sh, will (1). pass the job configuration to qsub, (2). initiate the job environment, and (3). schedule the job to run fmriprep on each of the 8 subject in this example:

#!/bin/bash
#$ -cwd -V # use current work dir and env for job
#$ -o joblog.$JOB_ID.$TASK_ID # write log (add Job/Index to name)
#$ -j y # merge error log with job log
#$ -l h_rt=22:00:00,h_data=20G # request 22 hours and 100GB total
#$ -pe shared 5 # request 5 cpus
#$ -m a # email about aborted runs only
#$ -t 1-8:1 # array index (adjust for N subjects)

# initiate job environment
. /u/local/Modules/default/init/modules.sh
module use /u/project/CCN/apps/modulefiles

module load fsl/5.0.9
module load freesurfer/6.0.0
module load ants/ants-2.3.1 # requires 2.3.1 or --use-syn-sdc will fail
module load afni/19.0.15
module load itksnap/3.6.0-RC1.QT4
module load hcp/1.3.2
module load python/3.7.0
module load ica-aroma/0.4.5 # requires 0.4.5 or --use-aroma will fail
module load fmriprep/20.1.1

# export paths and settings
export PATH=/u/project/CCN/apps/c3d/c3d-1.0.0-Linux-x86_64/bin/:$PATH
export NO_FSL_JOBS=true

# Declare the subjects list
declare -a sublist

SUBS[0]='01'
SUBS[1]='02'
SUBS[2]='03'
SUBS[3]='04'
SUBS[4]='05'
SUBS[5]='06'
SUBS[6]='07'
SUBS[7]='08'

## subjects can also be read from a file:
# readarray -t SUBS < subjects.txt

# initiate count for array submission
(( i=$SGE_TASK_ID - 1 ))
echo "This is sub-job $SGE_TASK_ID, --n-cpus $NSLOTS"
echo "This is subject ${SUBS[$i]}"

# pass subject ID and number of cpus to second script
./run_fmriprep.sh ${SUBS[$i]} $NSLOTS

# ~ done! ~

In total, 8 separate jobs will be created after running the script above.
Using the command qstat -u USERNAME will show a single job array until they begin running, as nodes become available.
The second script, run_fmriprep.sh, includes all the project-specific parameters that will be passed to the fmriprep command at the end.
#!/bin/sh

# # SET PARAMETERS ##############
SUB=$1 # a space delimited list of participant identifiers
CPUS=$2 # number of processors requested for each job
WORKPATH=${SCRATCH} # send intermediate files to scratch folder
PIPENAME='fmriprep-20.1.1' # name of output folder in derivatives
BIDSPATH='/u/project/usergroup/data/BIDS_PROJECT'

# project-specific flags for fmriprep
ADDFLAGS='--use-aroma --use-syn-sdc --output-spaces MNI152NLin6Asym MNIPediatricAsym:cohort-1:res-2 --resource-monitor --bids-filter-file bids.json'
freesurfer_license='/u/project/CCN/apps/freesurfer/6.0.0/LICENSE'

# # FMRIPREP COMMAND ############
fmriprep ${ADDFLAGS} -w ${WORKPATH} \
--participant_label ${SUB} --fs-license-file ${freesurfer_license} \
${BIDSPATH}/rawdata ${BIDSPATH}/derivatives/${PIPENAME} participant \
--n-cpus ${CPUS} --mem_mb 98000

# ~ done! ~

A few things to note:
(1). the module versions are different for the new version of fmrirpep
(2). $SCRATCH is a 2TB working directory for temporary files.
SCRATCH=/u/scratch/[u]/[username]
Sending fmriprep scratch files here helps save space in your group project directory.
These files will be automatically deleted after 7 days.
Call the CCN script, clear_scratch.sh, to clear those temporary files if necessary.
(3). fmriprep takes in environmental information to optimize its parallel processing.
Unfortunately, since it is not run in a container, fmriprep sometimes gets input about the entire cluster node and thinks it has access to more resources than requested.
The system will usually kill jobs that exceed the requested limit.
To prevent this, use --n-cpus and --mem_mb to limit fmriprep to the number of resources requested for the job.
For --mem_mb it’s recommended to specify an amount just below the total amount of memory requested as fmriprep sometimes spills over the limit.

Hoffman2:Software Tools: FMRIprep: Job Array Example-v20.2 - Revision history

Jpdiaz: fmriprep as job array example