Building Computational Pipelines with Nextflow

About the course  |  Intended audience  |  Prerequisites  |  Content details

About the course

High-throughput data analyses often involve multiple processing steps and the use of numerous command line tools and scripts to transform, filter, aggregate, and visualise data. As analyses become increasingly complex, managing software dependencies, execution environments, and reproducibility can become challenging.

This course provides a practical introduction to the Nextflow workflow management system for building reproducible and scalable computational pipelines. Participants will learn how to develop workflows that can be executed seamlessly on local computers, servers, high-performance computing (HPC) systems, and cloud environments without modifying the workflow definition.

The course covers the principles of workflow development using Nextflow and introduces advanced strategies for customising, automating, and scaling analyses. Participants will also learn how to use modules and workflow design principles from the nf-core initiative to build robust, high-quality pipelines that follow current community best practices.

By the end of the course, participants should be able to develop simple computational workflows, run and troubleshoot existing pipelines, and build nf-core compatible pipelines using reusable modules and community tooling.

Teaching is primarily hands-on, with short presentations introducing the concepts and methods needed to develop and manage computational workflows.

 Back to top

Intended audience

This course is suitable for:

• researchers who perform computational data analysis and wish to build reproducible and scalable pipelines
• participants who have experience running command line tools and scripting analyses but are new to workflow management systems
• researchers with limited experience of Nextflow who want practical experience developing and running workflows

This course is not intended for complete beginners in programming or the UNIX shell.

 Back to top

Prerequisites

Participants should have:

• competence with the UNIX command line, including navigating directories, managing files, inspecting file contents, and running command line tools with options
• familiarity with at least one scripting language, such as Python, R, or Bash
• experience editing scripts, using variables, and understanding simple control flow
• familiarity with tabular data formats such as CSV and TSV/TXT

The following experience is advantageous but not required:

• familiarity with common bioinformatic file formats such as FASTQ, BAM, and VCF
• experience using High Performance Computing (HPC) environments and job schedulers such as SLURM, PBS, or LSF
• familiarity with software package managers such as Conda or Mamba
• a basic understanding of container technologies such as Docker or Singularity/Apptainer
• experience organising computational projects or pipelines

 Back to top

Content details

The course covers the following topics:

• Introduction to workflow management
  Introduces the principles of workflow management and the challenges of developing reproducible and scalable computational analyses.
• Building workflows with Nextflow
  Covers the core components of Nextflow and demonstrates how to build simple multi-step workflows using processes, channels, and workflows.
• Running and troubleshooting workflows
  Introduces methods for launching workflows locally, interpreting results and log files, and troubleshooting common issues.
• Working with modules and configuration
  Covers strategies for modular workflow design, software provisioning, and workflow configuration using containers and configuration files.
• Scaling workflows across computing environments
  Introduces approaches for running workflows on local computers, servers, high-performance computing systems, and cloud environments.
• Introduction to nf-core
  Covers the structure and organisation of nf-core pipelines and demonstrates how to find, run, and interpret existing nf-core workflows.
• Building nf-core compatible pipelines
  Introduces the principles of nf-core pipeline development and demonstrates how to create pipelines from templates and incorporate reusable modules.
• Modules and input validation
  Covers the development and integration of nf-core compatible modules and introduces tools for validating workflow inputs and parameters.

 Back to top