Workflow Object

Orchestrates multiple Tool objects over a shared directory, handling file discovery, filtering, tidying, and writing in a single pipeline.

A Workflow object:

has a name (name);
has one or more paths to workflow results (path);
holds Tool subclass objects accessible via get_tools();
exposes matched files across all tools via list_files();
has a tibble of written output files, populated after write() (written_files);

The typical workflow is: optionally filter with filter_files(), tidy with tidy(), then write with write(). run() chains all three steps.

Public fields

name: (character(1))
Name of workflow.
path: (character(n))
Path(s) to workflow results.
metapkg: (character(n))
Package name(s) used for metadata version reporting.
written_files: (tibble(n))
Tibble of files written from self$write().

Methods

Method `new()`

Create a new Workflow object.

Usage

Workflow$new(name, path, tools, metapkg = "nemo")

Arguments

name: (character(1))
Name of workflow.
path: (character(n))
Path(s) to workflow results.
tools: (list(n))
Named list of Tool subclasses that compose a Workflow. List names serve as aliases and need not match each tool's own $name field.
metapkg: (character(n))
Package name(s) used for metadata version reporting.

Returns

(R6::R6Class())
R6 object.

Method `print()`

Print details about the Workflow.

Usage

Workflow$print(...)

Arguments

...: (ignored).

Returns

(R6::R6Class())
R6 object invisibly.

Method `get_tools()`

Get the list of Tool objects in this Workflow.

Usage

Workflow$get_tools()

Returns

(list(n))
Named list of instantiated Tool objects.

Method `filter_files()`

Filter files in given workflow directory.

Usage

Workflow$filter_files(include = NULL, exclude = NULL)

Arguments

include: (character(n))
tool_parser names to include (e.g. "tool1_table1").
exclude: (character(n))
tool_parser names to exclude (e.g. "tool1_table5").

Returns

(R6::R6Class())
R6 object invisibly.

Method `list_files()`

List only files of interest in given workflow directory, i.e. only those files that match the patterns listed in the individual tool config.

Usage

Workflow$list_files()

Returns

(tibble())
Bound files tibbles from all Tools, with a leading tool column.

Method `tidy()`

Tidy Workflow files.

Usage

Workflow$tidy(keep_raw = FALSE)

Arguments

keep_raw: (logical(1))
Should the raw parsed tibbles be kept in the final output?

Returns

(R6::R6Class())
R6 object invisibly.

Method `write()`

Write tidy tibbles.

Usage

Workflow$write(
  output_dir = ".",
  format = "tsv",
  input_id = NULL,
  output_id = NULL,
  prefix_include = FALSE,
  dbconn = NULL,
  write_metadata = TRUE
)

Arguments

output_dir: (character(1))
Directory path to output tidy files.
format: (character(1))
Format of output.
input_id: (character(1))
Input ID to use for the dataset (e.g. run123).
output_id: (character(1))
Output ID to use for the dataset (e.g. out1).
prefix_include: (logical(1))
If TRUE, prepend an input_prefix column to each tidy table.
dbconn: (DBIConnection)
Database connection object (see DBI::dbConnect).
write_metadata: (logical(1))
If TRUE (default), write a metadata.parquet file alongside the tidy outputs. Set to FALSE to suppress.

Returns

(R6::R6Class())
R6 object invisibly.

Method `run()`

Filter, tidy, and write files in one step.

Usage

Workflow$run(
  output_dir = ".",
  format = "tsv",
  input_id = NULL,
  output_id = NULL,
  prefix_include = FALSE,
  dbconn = NULL,
  write_metadata = TRUE,
  include = NULL,
  exclude = NULL
)

Arguments

output_dir: (character(1))
Directory path to output tidy files.
format: (character(1))
Format of output.
input_id: (character(1))
Input ID to use for the dataset (e.g. run123).
output_id: (character(1))
Output ID to use for the dataset (e.g. out1).
prefix_include: (logical(1))
If TRUE, prepend an input_prefix column to each tidy table.
dbconn: (DBIConnection)
Database connection object (see DBI::dbConnect).
write_metadata: (logical(1))
If TRUE (default), write a metadata.parquet file. Set to FALSE to suppress.
include: (character(n))
tool_parser names to include (e.g. "tool1_table1").
exclude: (character(n))
tool_parser names to exclude (e.g. "tool1_table5").

Returns

(R6::R6Class())
R6 object invisibly.

Method `get_schemas_raw()`

Get raw schemas for all Tools.

Usage

Workflow$get_schemas_raw()

Returns

(tibble())
Bound schemas_raw tibbles from all Tools, with a leading tool column.

Method `get_schemas_tidy()`

Get tidy schemas for all Tools.

Usage

Workflow$get_schemas_tidy()

Returns

(tibble())
Bound tidy schema tibbles from all Tools, with a leading tool column.

Method `get_tbls()`

Get tidy tibbles for all Tools.

Usage

Workflow$get_tbls()

Returns

(tibble())
Bound tbls tibbles from all Tools, with a leading tool column. When tidy(keep_raw = TRUE) was used, each tool's tibble also contains a raw list-column. Note: get_tbls() checks is.null() (not nrow()) because Tool$get_tbls() returns NULL when nothing was tidied, whereas Tool$list_files() always returns a zero-row tibble (never NULL).

Method `get_metadata()`

Get metadata for the workflow run.

Usage

Workflow$get_metadata(input_id, output_id, output_dir, pkgs = NULL)

Arguments

input_id: (character(1))
Input ID to use for the dataset (e.g. run123).
output_id: (character(1))
Output ID to use for the dataset (e.g. out1).
output_dir: (character(1))
Output directory.
pkgs: (character(n))
Which R packages to extract versions for.

Returns

(tibble())
Single-row tibble with columns input_id, output_id, input_dirs, output_dir, pkg_versions, and files.

Examples

fs::path(tempdir(), letters[1:5]) |>
  fs::file_temp_push() |>
  fs::dir_create()
path <- system.file("extdata/tool1", package = "nemo")
tools <- list(tool1 = Tool1)
wf <- Workflow$new(name = "wf1", path = path, tools = tools)
(lf_all <- wf$list_files())
#> # A tibble: 12 × 10
#>    tool  tool_parser parser bname  size lastmodified        path  pattern prefix
#>    <chr> <chr>       <chr>  <chr> <fs:> <dttm>              <chr> <chr>   <chr> 
#>  1 tool1 tool1_tabl… table1 samp…   133 2026-07-28 09:30:14 /hom… "\\.to… sampl…
#>  2 tool1 tool1_tabl… table1 samp…   113 2026-07-28 09:30:14 /hom… "\\.to… sampl…
#>  3 tool1 tool1_tabl… table1 samp…    93 2026-07-28 09:30:14 /hom… "\\.to… sampl…
#>  4 tool1 tool1_tabl… table2 samp…    70 2026-07-28 09:30:14 /hom… "\\.to… sampl…
#>  5 tool1 tool1_tabl… table2 samp…    47 2026-07-28 09:30:14 /hom… "\\.to… sampl…
#>  6 tool1 tool1_tabl… table3 samp…    83 2026-07-28 09:30:14 /hom… "\\.to… sampl…
#>  7 tool1 tool1_tabl… table3 samp…    48 2026-07-28 09:30:14 /hom… "\\.to… sampl…
#>  8 tool1 tool1_tabl… table4 samp…    52 2026-07-28 09:30:14 /hom… "\\.to… sampl…
#>  9 tool1 tool1_tabl… table4 samp…    34 2026-07-28 09:30:14 /hom… "\\.to… sampl…
#> 10 tool1 tool1_tabl… table6 samp…   100 2026-07-28 09:30:14 /hom… "\\.to… sampl…
#> 11 tool1 tool1_tabl… table6 samp…    78 2026-07-28 09:30:14 /hom… "\\.to… sampl…
#> 12 tool1 tool1_tabl… table5 samp…   994 2026-07-28 09:30:14 /hom… "\\.to… sampl…
#> # ℹ 1 more variable: prefix_suffix <chr>
wf$filter_files(exclude = "tool1_table5")
wf$tidy()
(tbls <- wf$get_tbls())
#> # A tibble: 11 × 11
#>    tool  tool_parser parser bname  size lastmodified        path  pattern prefix
#>    <chr> <chr>       <chr>  <chr> <fs:> <dttm>              <chr> <chr>   <chr> 
#>  1 tool1 tool1_tabl… table1 samp…   133 2026-07-28 09:30:14 /hom… "\\.to… sampl…
#>  2 tool1 tool1_tabl… table1 samp…   113 2026-07-28 09:30:14 /hom… "\\.to… sampl…
#>  3 tool1 tool1_tabl… table1 samp…    93 2026-07-28 09:30:14 /hom… "\\.to… sampl…
#>  4 tool1 tool1_tabl… table2 samp…    70 2026-07-28 09:30:14 /hom… "\\.to… sampl…
#>  5 tool1 tool1_tabl… table2 samp…    47 2026-07-28 09:30:14 /hom… "\\.to… sampl…
#>  6 tool1 tool1_tabl… table3 samp…    83 2026-07-28 09:30:14 /hom… "\\.to… sampl…
#>  7 tool1 tool1_tabl… table3 samp…    48 2026-07-28 09:30:14 /hom… "\\.to… sampl…
#>  8 tool1 tool1_tabl… table4 samp…    52 2026-07-28 09:30:14 /hom… "\\.to… sampl…
#>  9 tool1 tool1_tabl… table4 samp…    34 2026-07-28 09:30:14 /hom… "\\.to… sampl…
#> 10 tool1 tool1_tabl… table6 samp…   100 2026-07-28 09:30:14 /hom… "\\.to… sampl…
#> 11 tool1 tool1_tabl… table6 samp…    78 2026-07-28 09:30:14 /hom… "\\.to… sampl…
#> # ℹ 2 more variables: prefix_suffix <chr>, tidy <list>
(rs <- wf$get_schemas_raw())
#> # A tibble: 12 × 5
#>    tool  name   tbl_description                                 version schema  
#>    <chr> <chr>  <chr>                                           <chr>   <list>  
#>  1 tool1 table1 Table1 for tool1 (txt: header present, tab-del… v1.2.3  <tibble>
#>  2 tool1 table1 Table1 for tool1 (txt: header present, tab-del… v4.5.6  <tibble>
#>  3 tool1 table1 Table1 for tool1 (txt: header present, tab-del… latest  <tibble>
#>  4 tool1 table2 Table2 for tool1 (txt: header present, tab-del… v1.0.0  <tibble>
#>  5 tool1 table2 Table2 for tool1 (txt: header present, tab-del… latest  <tibble>
#>  6 tool1 table3 Table3 for tool1 (txt-keyvalue: no header, 2 c… v1.0.0  <tibble>
#>  7 tool1 table3 Table3 for tool1 (txt-keyvalue: no header, 2 c… latest  <tibble>
#>  8 tool1 table4 Table4 for tool1 (txt-nohead: no header, posit… v1.0.0  <tibble>
#>  9 tool1 table4 Table4 for tool1 (txt-nohead: no header, posit… latest  <tibble>
#> 10 tool1 table6 Table6 for tool1 (csv: header present, comma-d… v1.0.0  <tibble>
#> 11 tool1 table6 Table6 for tool1 (csv: header present, comma-d… latest  <tibble>
#> 12 tool1 table5 Table5 for tool1 (csv-nohead-long: no header, … latest  <tibble>
dir1 <- fs::file_temp(); dir2 <- fs::file_temp()
wf$write(output_dir = dir1, format = "parquet", input_id = "run1")
(lf1 <- list.files(dir1, pattern = "tool1.*parquet", full.names = TRUE))
#>  [1] "/tmp/RtmpLFaYvY/c/sampleA_2_tool1_table1.parquet"
#>  [2] "/tmp/RtmpLFaYvY/c/sampleA_2_tool1_table2.parquet"
#>  [3] "/tmp/RtmpLFaYvY/c/sampleA_2_tool1_table3.parquet"
#>  [4] "/tmp/RtmpLFaYvY/c/sampleA_2_tool1_table4.parquet"
#>  [5] "/tmp/RtmpLFaYvY/c/sampleA_2_tool1_table6.parquet"
#>  [6] "/tmp/RtmpLFaYvY/c/sampleA_3_tool1_table1.parquet"
#>  [7] "/tmp/RtmpLFaYvY/c/sampleA_tool1_table1.parquet"  
#>  [8] "/tmp/RtmpLFaYvY/c/sampleA_tool1_table2.parquet"  
#>  [9] "/tmp/RtmpLFaYvY/c/sampleA_tool1_table3.parquet"  
#> [10] "/tmp/RtmpLFaYvY/c/sampleA_tool1_table4.parquet"  
#> [11] "/tmp/RtmpLFaYvY/c/sampleA_tool1_table6.parquet"  
(meta <- wf$get_metadata(input_id = "run1", output_id = "out1", output_dir = dir1))
#> # A tibble: 1 × 6
#>   input_id output_id input_dirs output_dir        pkg_versions files        
#>   <chr>    <chr>     <list>     <fs::path>        <list>       <list>       
#> 1 run1     out1      <chr [1]>  /tmp/RtmpLFaYvY/c <df [1 × 2]> <df [11 × 4]>
wf2 <- Workflow$new(name = "wf2", path = path, tools = tools)
wf2$run(output_dir = dir2, format = "parquet", input_id = "run2")
(lf2 <- list.files(dir2, pattern = "tool1.*parquet", full.names = TRUE))
#>  [1] "/tmp/RtmpLFaYvY/d/sampleA_2_tool1_table1.parquet"
#>  [2] "/tmp/RtmpLFaYvY/d/sampleA_2_tool1_table2.parquet"
#>  [3] "/tmp/RtmpLFaYvY/d/sampleA_2_tool1_table3.parquet"
#>  [4] "/tmp/RtmpLFaYvY/d/sampleA_2_tool1_table4.parquet"
#>  [5] "/tmp/RtmpLFaYvY/d/sampleA_2_tool1_table6.parquet"
#>  [6] "/tmp/RtmpLFaYvY/d/sampleA_3_tool1_table1.parquet"
#>  [7] "/tmp/RtmpLFaYvY/d/sampleA_tool1_table1.parquet"  
#>  [8] "/tmp/RtmpLFaYvY/d/sampleA_tool1_table2.parquet"  
#>  [9] "/tmp/RtmpLFaYvY/d/sampleA_tool1_table3.parquet"  
#> [10] "/tmp/RtmpLFaYvY/d/sampleA_tool1_table4.parquet"  
#> [11] "/tmp/RtmpLFaYvY/d/sampleA_tool1_table5.parquet"  
#> [12] "/tmp/RtmpLFaYvY/d/sampleA_tool1_table6.parquet"

Public fields

Methods

Public methods

Method new()

Usage

Arguments

Returns

Method print()

Usage

Arguments

Returns

Method get_tools()

Usage

Returns

Method filter_files()

Usage

Arguments

Returns

Method list_files()

Usage

Returns

Method tidy()

Usage

Arguments

Returns

Method write()

Usage

Arguments

Returns

Method run()

Usage

Arguments

Returns

Method get_schemas_raw()

Usage

Returns

Method get_schemas_tidy()

Usage

Returns

Method get_tbls()

Usage

Returns

Method get_metadata()

Usage

Arguments

Returns

Examples

Method `new()`

Method `print()`

Method `get_tools()`

Method `filter_files()`

Method `list_files()`

Method `tidy()`

Method `write()`

Method `run()`

Method `get_schemas_raw()`

Method `get_schemas_tidy()`

Method `get_tbls()`

Method `get_metadata()`