Skip to contents

Structure

Source: vignettes/structure.Rmd
structure.Rmd

nemo is built on top of R’s R6 encapsulated object-oriented programming implementation, which helps with code organisation. It consists of several base classes (Config, Tool, and Workflow) which we describe below. Each R6 class can contain public and private functions and non-functions (fields).

Other R packages like tidywigits and tidydragen can create their own Tool and Workflow children classes that inherit (or override) functions from the nemo parent classes. This allows for custom parsers and tidiers for specific bioinformatic tools and workflows.

Here we use the Tool1 and Workflow1 nemo classes as examples to illustrate the structure of the package.

Config

A Config object reads a single schema.yaml file from inst/config/tools/<tool>/. Each table entry defines its file pattern, file type, description, and a list of columns — each column carrying both its raw name (as produced by the tool) and its tidy name (snake_case), type, description, and the versions array listing every tool version the column appears in. Config derives versioned raw and tidy views from this unified source on demand. See ?Config.

Each tool has a schema.yaml under inst/config/tools/<tool>/. It is a flat tables map with one entry per output file:

tables:
  <table_name>:
    description: '<human-readable description>'
    pattern: "<regex matching the file basename>"
    ftype: '<file type — see below>'
    columns:
      - raw: '<raw column name or positional label>'
        tidy: '<tidy snake_case name>'
        type: 'char | int | float'
        description: '<human-readable description>'
        versions: ['<version1>', '<version2>']

The ftype field signals how the file is parsed:

ftype Description raw: names Parser
tsv Header present; tab-delimited actual column names parse_file
csv Header present; comma-delimited actual column names parse_file (delim=,)
tsv-nohead No header; positional columns X1, X2, …, XN parse_file_nohead
tsv-keyvalue No header; 2 cols (key, value); pivoted wide actual key values parse_file_keyvalue
csv-nohead-long No header; long format; custom pivot metric names (values in the key column) custom tidy_* method

Child packages can register additional ftypes by overriding private$extra_ftypes() — return a named list of ftype -> function(x, table_name). These are checked before the built-in switch.

The versions array on each column lists every tool version that column appears in, giving a full picture of additions and removals:

# table1 columns
SampleID:   versions: ['v1.2.3', 'v4.5.6', 'latest']  # present in all versions
metricX:    versions: ['v1.2.3']                        # dropped in v4.5.6
metricY:    versions: ['latest']                        # added in latest

The effective schema for version V is all columns whose versions array contains V. schema_guess uses this to automatically match a file’s columns against the correct versioned snapshot.

tool <- params$tool
workflow <- params$workflow
conf <- nemo::Config$new(tool, pkg = "nemo")
conf
## #--- Config nemo::tool1 ---#
## 
## |var   |value |
## |:-----|:-----|
## |tool  |tool1 |
## |pkg   |nemo  |
## |ntbls |6     |

Table-level metadata — patterns, file types, and descriptions for all tables:

conf$get_patterns() |> knitr::kable(caption = glue("{tool} patterns."))
Tool1 patterns.
name pattern
table1 .tool1.table1.tsv$
table2 .tool1.table2.tsv$
table3 .tool1.table3.tsv$
table4 .tool1.table4.tsv$
table6 .tool1.table6.csv$
table5 .tool1.table5.csv$ 
conf$get_ftypes() |> knitr::kable(caption = glue("{tool} file types."))
Tool1 file types.
name ftype
table1 tsv
table2 tsv
table3 tsv-keyvalue
table4 tsv-nohead
table6 csv
table5 csv-nohead-long 
conf$get_descriptions() |> knitr::kable(caption = glue("{tool} descriptions."))
Tool1 descriptions.
name description
table1 Table1 for tool1 (txt: header present, tab-delimited).
table2 Table2 for tool1 (txt: header present, tab-delimited).
table3 Table3 for tool1 (txt-keyvalue: no header, 2 cols, col1=key col2=value).
table4 Table4 for tool1 (txt-nohead: no header, positional cols X1..XN).
table6 Table6 for tool1 (csv: header present, comma-delimited).
table5 Table5 for tool1 (csv-nohead-long: no header, long format with metric name col).

Per-table lookups:

conf$get_pattern("table1")
## [1] "\\.tool1\\.table1\\.tsv$"
conf$get_ftype("table1")
## [1] "tsv"
conf$get_description("table1")
## [1] "Table1 for tool1 (txt: header present, tab-delimited)."

All versioned raw and tidy schemas:

conf$get_schemas_raw() |> dplyr::select("name", "version", "schema")
## # A tibble: 12 × 3
##    name   version schema          
##    <chr>  <chr>   <list>          
##  1 table1 v1.2.3  <tibble [5 × 2]>
##  2 table1 v4.5.6  <tibble [4 × 2]>
##  3 table1 latest  <tibble [6 × 2]>
##  4 table2 v1.0.0  <tibble [2 × 2]>
##  5 table2 latest  <tibble [3 × 2]>
##  6 table3 v1.0.0  <tibble [3 × 2]>
##  7 table3 latest  <tibble [5 × 2]>
##  8 table4 v1.0.0  <tibble [3 × 2]>
##  9 table4 latest  <tibble [5 × 2]>
## 10 table6 v1.0.0  <tibble [3 × 2]>
## 11 table6 latest  <tibble [4 × 2]>
## 12 table5 latest  <tibble [4 × 2]>
conf$get_schemas_tidy() |> dplyr::select("name", "version", "schema")
## # A tibble: 12 × 3
##    name   version schema          
##    <chr>  <chr>   <list>          
##  1 table1 v1.2.3  <tibble [5 × 2]>
##  2 table1 v4.5.6  <tibble [4 × 2]>
##  3 table1 latest  <tibble [6 × 2]>
##  4 table2 v1.0.0  <tibble [2 × 2]>
##  5 table2 latest  <tibble [3 × 2]>
##  6 table3 v1.0.0  <tibble [3 × 2]>
##  7 table3 latest  <tibble [5 × 2]>
##  8 table4 v1.0.0  <tibble [3 × 2]>
##  9 table4 latest  <tibble [5 × 2]>
## 10 table6 v1.0.0  <tibble [3 × 2]>
## 11 table6 latest  <tibble [4 × 2]>
## 12 table5 latest  <tibble [4 × 2]>
conf$get_schemas_both()
## # A tibble: 12 × 4
##    name   tbl_description                                                           version schema  
##    <chr>  <chr>                                                                     <chr>   <list>  
##  1 table1 Table1 for tool1 (txt: header present, tab-delimited).                    v1.2.3  <tibble>
##  2 table1 Table1 for tool1 (txt: header present, tab-delimited).                    v4.5.6  <tibble>
##  3 table1 Table1 for tool1 (txt: header present, tab-delimited).                    latest  <tibble>
##  4 table2 Table2 for tool1 (txt: header present, tab-delimited).                    v1.0.0  <tibble>
##  5 table2 Table2 for tool1 (txt: header present, tab-delimited).                    latest  <tibble>
##  6 table3 Table3 for tool1 (txt-keyvalue: no header, 2 cols, col1=key col2=value).  v1.0.0  <tibble>
##  7 table3 Table3 for tool1 (txt-keyvalue: no header, 2 cols, col1=key col2=value).  latest  <tibble>
##  8 table4 Table4 for tool1 (txt-nohead: no header, positional cols X1..XN).         v1.0.0  <tibble>
##  9 table4 Table4 for tool1 (txt-nohead: no header, positional cols X1..XN).         latest  <tibble>
## 10 table6 Table6 for tool1 (csv: header present, comma-delimited).                  v1.0.0  <tibble>
## 11 table6 Table6 for tool1 (csv: header present, comma-delimited).                  latest  <tibble>
## 12 table5 Table5 for tool1 (csv-nohead-long: no header, long format with metric na… latest  <tibble>

Per-table, per-version:

conf$get_schema_raw("table1", version = "v1.2.3")
## # A tibble: 5 × 3
##   version field      type 
##   <chr>   <chr>      <chr>
## 1 v1.2.3  SampleID   c    
## 2 v1.2.3  Chromosome c    
## 3 v1.2.3  Start      i    
## 4 v1.2.3  End        i    
## 5 v1.2.3  metricX    d
conf$get_schema_tidy("table1", version = "v1.2.3")
## # A tibble: 5 × 3
##   version field      type 
##   <chr>   <chr>      <chr>
## 1 v1.2.3  sample_id  c    
## 2 v1.2.3  chromosome c    
## 3 v1.2.3  start      i    
## 4 v1.2.3  end        i    
## 5 v1.2.3  metric_x   d

For csv-nohead-long tables, get_col_map() returns the raw-to-tidy metric-name mapping used inside the tool’s tidy_* method:

conf$get_col_map("table5")
## # A tibble: 4 × 4
##   raw            tidy        type  description   
##   <chr>          <chr>       <chr> <chr>         
## 1 Total reads    reads_total d     total reads   
## 2 Mapped reads   reads_map   d     mapped reads  
## 3 Unmapped reads reads_unmap d     unmapped reads
## 4 Total bases    bases_total d     total bases

Tool

Tool is the main organisation class for all file parsers and tidiers. It contains functions for parsing and tidying typical CSV/TSV files (with column names), and TXT files where the column names are missing. Currently it utilises the very simple readr::read_delim function from the readr package that reads all the data into memory. See ?Tool.

These simple parsers are used in 80-90% of cases, so in the future we can optimise the parsing if needed with faster packages such as data.table, duckdb-r/duckplyr or r-polars.

We can have different Tool children classes that inherit (or override) functions and fields from the Tool parent class. For example, we can create a Tool object for Tool1 as follows:

  • Initialise a Tool1 object:
tool1_path <- system.file("extdata/tool1", package = "nemo")
t1 <- nemo::Tool1$new(path = tool1_path)
# each class comes with a print function
t1
## #--- Tool nemo::tool1 ---#
## 
## |var     |value                                                                     |
## |:-------|:-------------------------------------------------------------------------|
## |name    |tool1                                                                     |
## |path    |/home/runner/miniconda3/envs/pkgdown_env/lib/R/library/nemo/extdata/tool1 |
## |files   |12                                                                        |
## |tidied  |false                                                                     |
## |written |false                                                                     |
  • Its Config object is also constructed based on the name supplied - this is used internally to find files of interest and infer their schemas:
t1$config
## #--- Config nemo::tool1 ---#
## 
## |var   |value |
## |:-----|:-----|
## |tool  |tool1 |
## |pkg   |nemo  |
## |ntbls |6     |
t1$config$get_patterns()
## # A tibble: 6 × 2
##   name   pattern                   
##   <chr>  <chr>                     
## 1 table1 "\\.tool1\\.table1\\.tsv$"
## 2 table2 "\\.tool1\\.table2\\.tsv$"
## 3 table3 "\\.tool1\\.table3\\.tsv$"
## 4 table4 "\\.tool1\\.table4\\.tsv$"
## 5 table6 "\\.tool1\\.table6\\.csv$"
## 6 table5 "\\.tool1\\.table5\\.csv$"
t1$config$get_schema_raw("table1", version = "v1.2.3")
## # A tibble: 5 × 3
##   version field      type 
##   <chr>   <chr>      <chr>
## 1 v1.2.3  SampleID   c    
## 2 v1.2.3  Chromosome c    
## 3 v1.2.3  Start      i    
## 4 v1.2.3  End        i    
## 5 v1.2.3  metricX    d
# t1$config$get_schema_raw("table1", version = "latest") # default
t1$config$get_schema_tidy("table1", version = "v1.2.3")
## # A tibble: 5 × 3
##   version field      type 
##   <chr>   <chr>      <chr>
## 1 v1.2.3  sample_id  c    
## 2 v1.2.3  chromosome c    
## 3 v1.2.3  start      i    
## 4 v1.2.3  end        i    
## 5 v1.2.3  metric_x   d
# t1$config$get_schema_tidy("table1", version = "latest") # default

We can list files that can be parsed with list_files():

lf <- t1$list_files()
lf |> dplyr::slice(1) |> str()
## tibble [1 × 9] (S3: tbl_df/tbl/data.frame)
##  $ tool_parser  : chr "tool1_table1"
##  $ parser       : chr "table1"
##  $ bname        : chr "sampleA.tool1.table1.tsv"
##  $ size         : 'fs_bytes' num 133
##  $ lastmodified : POSIXct[1:1], format: "2026-07-28 09:30:14"
##  $ path         : chr "/home/runner/miniconda3/envs/pkgdown_env/lib/R/library/nemo/extdata/tool1/latest/sampleA.tool1.table1.tsv"
##  $ pattern      : chr "\\.tool1\\.table1\\.tsv$"
##  $ prefix       : chr "sampleA"
##  $ prefix_suffix: chr ""

We can parse and tidy files of interest using the tidy function. Note that this function is called on the object and not assigned anywhere:

# this will create a new field tbls containing the tidy data (and optionally
# the 'raw' parsed data)
t1$tidy(keep_raw = TRUE)
tbls <- t1$get_tbls()
tbls
## # A tibble: 12 × 11
##    tool_parser  parser bname    size lastmodified        path  pattern prefix prefix_suffix raw     
##    <chr>        <chr>  <chr>   <fs:> <dttm>              <chr> <chr>   <chr>  <chr>         <list>  
##  1 tool1_table1 table1 sample…   133 2026-07-28 09:30:14 /hom… "\\.to… sampl… ""            <tibble>
##  2 tool1_table1 table1 sample…   113 2026-07-28 09:30:14 /hom… "\\.to… sampl… "_2"          <tibble>
##  3 tool1_table1 table1 sample…    93 2026-07-28 09:30:14 /hom… "\\.to… sampl… "_3"          <tibble>
##  4 tool1_table2 table2 sample…    70 2026-07-28 09:30:14 /hom… "\\.to… sampl… ""            <tibble>
##  5 tool1_table2 table2 sample…    47 2026-07-28 09:30:14 /hom… "\\.to… sampl… "_2"          <tibble>
##  6 tool1_table3 table3 sample…    83 2026-07-28 09:30:14 /hom… "\\.to… sampl… ""            <tibble>
##  7 tool1_table3 table3 sample…    48 2026-07-28 09:30:14 /hom… "\\.to… sampl… "_2"          <tibble>
##  8 tool1_table4 table4 sample…    52 2026-07-28 09:30:14 /hom… "\\.to… sampl… ""            <tibble>
##  9 tool1_table4 table4 sample…    34 2026-07-28 09:30:14 /hom… "\\.to… sampl… "_2"          <tibble>
## 10 tool1_table6 table6 sample…   100 2026-07-28 09:30:14 /hom… "\\.to… sampl… ""            <tibble>
## 11 tool1_table6 table6 sample…    78 2026-07-28 09:30:14 /hom… "\\.to… sampl… "_2"          <tibble>
## 12 tool1_table5 table5 sample…   994 2026-07-28 09:30:14 /hom… "\\.to… sampl… ""            <tibble>
## # ℹ 1 more variable: tidy <list>
tbls$raw[[1]] |> dplyr::glimpse()
## Rows: 3
## Columns: 6
## $ SampleID   <chr> "sampleA", "sampleA", "sampleA"
## $ Chromosome <chr> "chr1", "chr2", "chr3"
## $ Start      <int> 10, 100, 1000
## $ End        <int> 50, 500, 5000
## $ metricY    <dbl> 0.4, 0.5, 0.6
## $ metricZ    <dbl> 0.7, 0.8, 0.9
# the tidy tibbles are nested to allow for more than one tidy tibble per file
tbls$tidy[[1]][["data"]][[1]] |> dplyr::glimpse()
## Rows: 3
## Columns: 6
## $ sample_id  <chr> "sampleA", "sampleA", "sampleA"
## $ chromosome <chr> "chr1", "chr2", "chr3"
## $ start      <int> 10, 100, 1000
## $ end        <int> 50, 500, 5000
## $ metric_y   <dbl> 0.4, 0.5, 0.6
## $ metric_z   <dbl> 0.7, 0.8, 0.9

We can also focus on a subset of files to tidy using the filter_files() function. The include and exclude arguments can specify which tool_parsers to include or exclude in the analysis:

# create new Tool1 object
t2 <- nemo::Tool1$new(path = tool1_path)
t2$list_files()
## # A tibble: 12 × 9
##    tool_parser  parser bname             size lastmodified        path  pattern prefix prefix_suffix
##    <chr>        <chr>  <chr>            <fs:> <dttm>              <chr> <chr>   <chr>  <chr>        
##  1 tool1_table1 table1 sampleA.tool1.t…   133 2026-07-28 09:30:14 /hom… "\\.to… sampl… ""           
##  2 tool1_table1 table1 sampleA.tool1.t…   113 2026-07-28 09:30:14 /hom… "\\.to… sampl… "_2"         
##  3 tool1_table1 table1 sampleA.tool1.t…    93 2026-07-28 09:30:14 /hom… "\\.to… sampl… "_3"         
##  4 tool1_table2 table2 sampleA.tool1.t…    70 2026-07-28 09:30:14 /hom… "\\.to… sampl… ""           
##  5 tool1_table2 table2 sampleA.tool1.t…    47 2026-07-28 09:30:14 /hom… "\\.to… sampl… "_2"         
##  6 tool1_table3 table3 sampleA.tool1.t…    83 2026-07-28 09:30:14 /hom… "\\.to… sampl… ""           
##  7 tool1_table3 table3 sampleA.tool1.t…    48 2026-07-28 09:30:14 /hom… "\\.to… sampl… "_2"         
##  8 tool1_table4 table4 sampleA.tool1.t…    52 2026-07-28 09:30:14 /hom… "\\.to… sampl… ""           
##  9 tool1_table4 table4 sampleA.tool1.t…    34 2026-07-28 09:30:14 /hom… "\\.to… sampl… "_2"         
## 10 tool1_table6 table6 sampleA.tool1.t…   100 2026-07-28 09:30:14 /hom… "\\.to… sampl… ""           
## 11 tool1_table6 table6 sampleA.tool1.t…    78 2026-07-28 09:30:14 /hom… "\\.to… sampl… "_2"         
## 12 tool1_table5 table5 sampleA.tool1.t…   994 2026-07-28 09:30:14 /hom… "\\.to… sampl… ""
t2$filter_files(include = c("tool1_table2", "tool1_table3"))
t2$list_files()
## # A tibble: 4 × 9
##   tool_parser  parser bname              size lastmodified        path  pattern prefix prefix_suffix
##   <chr>        <chr>  <chr>             <fs:> <dttm>              <chr> <chr>   <chr>  <chr>        
## 1 tool1_table2 table2 sampleA.tool1.ta…    70 2026-07-28 09:30:14 /hom… "\\.to… sampl… ""           
## 2 tool1_table2 table2 sampleA.tool1.ta…    47 2026-07-28 09:30:14 /hom… "\\.to… sampl… "_2"         
## 3 tool1_table3 table3 sampleA.tool1.ta…    83 2026-07-28 09:30:14 /hom… "\\.to… sampl… ""           
## 4 tool1_table3 table3 sampleA.tool1.ta…    48 2026-07-28 09:30:14 /hom… "\\.to… sampl… "_2"

After tidying the data of interest, we can write the tidy tibbles to various formats, like Apache Parquet, PostgreSQL, CSV/TSV and R’s RDS. Below we can see that the (optional) input_id specified is added to the written files in an additional input_id column. This can be used e.g. to distinguish results from different samples in a data pipeline. An optional output_id column can also be added (e.g. to distinguish between runs for the same input sample). An input_prefix column can be included via prefix_include = TRUE, which would use the prefix of the file basename. For more details about output naming conventions, see the output naming vignette.

t2$tidy() # first need to tidy
outdir1 <- file.path(tempdir(), "tool1_write")
fmt <- "csv"
t2$write(
  output_dir = outdir1,
  format = fmt,
  input_id = "run123",
  output_id = "out123",
  prefix_include = TRUE
)
wfiles <- fs::dir_info(outdir1) |>
  dplyr::mutate(bname = basename(.data$path)) |>
  dplyr::select("bname", "size", "type")
wfiles
## # A tibble: 5 × 3
##   bname                                size type 
##   <chr>                         <fs::bytes> <fct>
## 1 metadata_tool1.parquet              4.71K file 
## 2 sampleA_2_tool1_table2.csv.gz          83 file 
## 3 sampleA_2_tool1_table3.csv.gz          98 file 
## 4 sampleA_tool1_table2.csv.gz            99 file 
## 5 sampleA_tool1_table3.csv.gz           111 file

The run function is a convenient wrapper for the process of filtering, tidying, and writing.

t3 <- nemo::Tool1$new(path = tool1_path)
outdir2 <- file.path(tempdir(), "t3")
t3$list_files()
## # A tibble: 12 × 9
##    tool_parser  parser bname             size lastmodified        path  pattern prefix prefix_suffix
##    <chr>        <chr>  <chr>            <fs:> <dttm>              <chr> <chr>   <chr>  <chr>        
##  1 tool1_table1 table1 sampleA.tool1.t…   133 2026-07-28 09:30:14 /hom… "\\.to… sampl… ""           
##  2 tool1_table1 table1 sampleA.tool1.t…   113 2026-07-28 09:30:14 /hom… "\\.to… sampl… "_2"         
##  3 tool1_table1 table1 sampleA.tool1.t…    93 2026-07-28 09:30:14 /hom… "\\.to… sampl… "_3"         
##  4 tool1_table2 table2 sampleA.tool1.t…    70 2026-07-28 09:30:14 /hom… "\\.to… sampl… ""           
##  5 tool1_table2 table2 sampleA.tool1.t…    47 2026-07-28 09:30:14 /hom… "\\.to… sampl… "_2"         
##  6 tool1_table3 table3 sampleA.tool1.t…    83 2026-07-28 09:30:14 /hom… "\\.to… sampl… ""           
##  7 tool1_table3 table3 sampleA.tool1.t…    48 2026-07-28 09:30:14 /hom… "\\.to… sampl… "_2"         
##  8 tool1_table4 table4 sampleA.tool1.t…    52 2026-07-28 09:30:14 /hom… "\\.to… sampl… ""           
##  9 tool1_table4 table4 sampleA.tool1.t…    34 2026-07-28 09:30:14 /hom… "\\.to… sampl… "_2"         
## 10 tool1_table6 table6 sampleA.tool1.t…   100 2026-07-28 09:30:14 /hom… "\\.to… sampl… ""           
## 11 tool1_table6 table6 sampleA.tool1.t…    78 2026-07-28 09:30:14 /hom… "\\.to… sampl… "_2"         
## 12 tool1_table5 table5 sampleA.tool1.t…   994 2026-07-28 09:30:14 /hom… "\\.to… sampl… ""
t3$run(
  output_dir = outdir2,
  format = "tsv",
  input_id = "run_t3",
  output_id = "out456",
  prefix_include = TRUE
)
wfiles2 <- fs::dir_info(outdir2) |>
  dplyr::mutate(bname = basename(.data$path)) |>
  dplyr::select("path", "bname", "size", "type")
wfiles2 |> dplyr::select(-"path")
## # A tibble: 13 × 3
##    bname                                size type 
##    <chr>                         <fs::bytes> <fct>
##  1 metadata_tool1.parquet              4.85K file 
##  2 sampleA_2_tool1_table1.tsv.gz         127 file 
##  3 sampleA_2_tool1_table2.tsv.gz          86 file 
##  4 sampleA_2_tool1_table3.tsv.gz         100 file 
##  5 sampleA_2_tool1_table4.tsv.gz         103 file 
##  6 sampleA_2_tool1_table6.tsv.gz         129 file 
##  7 sampleA_3_tool1_table1.tsv.gz         112 file 
##  8 sampleA_tool1_table1.tsv.gz           137 file 
##  9 sampleA_tool1_table2.tsv.gz           102 file 
## 10 sampleA_tool1_table3.tsv.gz           112 file 
## 11 sampleA_tool1_table4.tsv.gz           122 file 
## 12 sampleA_tool1_table5.tsv.gz           231 file 
## 13 sampleA_tool1_table6.tsv.gz           142 file
tbl1_path <- wfiles2 |>
  dplyr::filter(stringr::str_detect(.data$bname, "tool1_table1")) |>
  dplyr::pull("path") |>
  head(1)
readr::read_tsv(tbl1_path, show_col_types = FALSE)
## # A tibble: 3 × 8
##   input_id input_prefix output_id sample_id chromosome start   end metric_x
##   <chr>    <chr>        <chr>     <chr>     <chr>      <dbl> <dbl>    <dbl>
## 1 run_t3   sampleA_2    out456    sampleA   chr1          10    50      0.1
## 2 run_t3   sampleA_2    out456    sampleA   chr2         100   500      0.2
## 3 run_t3   sampleA_2    out456    sampleA   chr3        1000  5000      0.3

Workflow

A Workflow consists of a list of one or more Tools. We can construct a certain Workflow with different Tools, which would allow parsing and writing tidy tables from a variety of bioinformatic tools. See ?Workflow.

For example, nemo contains a Workflow1 class as a Workflow child (containing only a single Tool1 for simplicity). Similarly to Tool, a Workflow object contains functions such as filter_files, list_files, tidy, write, run, and get_tools() (to access the instantiated Tool objects):

w <- system.file("extdata/tool1", package = "nemo") |>
  nemo::Workflow1$new()
outdir3 <- file.path(tempdir(), "w1")
w$get_tools()
## $tool1
## #--- Tool nemo::tool1 ---#
## 
## |var     |value     |
## |:-------|:---------|
## |name    |tool1     |
## |path    |<ignored> |
## |files   |12        |
## |tidied  |false     |
## |written |false     |
w$list_files()
## # A tibble: 12 × 10
##    tool  tool_parser  parser bname       size lastmodified        path  pattern prefix prefix_suffix
##    <chr> <chr>        <chr>  <chr>      <fs:> <dttm>              <chr> <chr>   <chr>  <chr>        
##  1 tool1 tool1_table1 table1 sampleA.t…   133 2026-07-28 09:30:14 /hom… "\\.to… sampl… ""           
##  2 tool1 tool1_table1 table1 sampleA.t…   113 2026-07-28 09:30:14 /hom… "\\.to… sampl… "_2"         
##  3 tool1 tool1_table1 table1 sampleA.t…    93 2026-07-28 09:30:14 /hom… "\\.to… sampl… "_3"         
##  4 tool1 tool1_table2 table2 sampleA.t…    70 2026-07-28 09:30:14 /hom… "\\.to… sampl… ""           
##  5 tool1 tool1_table2 table2 sampleA.t…    47 2026-07-28 09:30:14 /hom… "\\.to… sampl… "_2"         
##  6 tool1 tool1_table3 table3 sampleA.t…    83 2026-07-28 09:30:14 /hom… "\\.to… sampl… ""           
##  7 tool1 tool1_table3 table3 sampleA.t…    48 2026-07-28 09:30:14 /hom… "\\.to… sampl… "_2"         
##  8 tool1 tool1_table4 table4 sampleA.t…    52 2026-07-28 09:30:14 /hom… "\\.to… sampl… ""           
##  9 tool1 tool1_table4 table4 sampleA.t…    34 2026-07-28 09:30:14 /hom… "\\.to… sampl… "_2"         
## 10 tool1 tool1_table6 table6 sampleA.t…   100 2026-07-28 09:30:14 /hom… "\\.to… sampl… ""           
## 11 tool1 tool1_table6 table6 sampleA.t…    78 2026-07-28 09:30:14 /hom… "\\.to… sampl… "_2"         
## 12 tool1 tool1_table5 table5 sampleA.t…   994 2026-07-28 09:30:14 /hom… "\\.to… sampl… ""
x <- w$run(
  output_dir = outdir3,
  format = "tsv",
  input_id = "wf1_run1",
  output_id = "out1",
  prefix_include = TRUE
)
wfiles3 <- fs::dir_info(outdir3) |>
  dplyr::select(1:5) |>
  dplyr::mutate(bname = basename(.data$path))
wfiles3 |>
  dplyr::select("bname", "size", "type")
## # A tibble: 13 × 3
##    bname                                size type 
##    <chr>                         <fs::bytes> <fct>
##  1 metadata.parquet                    4.85K file 
##  2 sampleA_2_tool1_table1.tsv.gz         126 file 
##  3 sampleA_2_tool1_table2.tsv.gz          85 file 
##  4 sampleA_2_tool1_table3.tsv.gz          99 file 
##  5 sampleA_2_tool1_table4.tsv.gz         102 file 
##  6 sampleA_2_tool1_table6.tsv.gz         129 file 
##  7 sampleA_3_tool1_table1.tsv.gz         112 file 
##  8 sampleA_tool1_table1.tsv.gz           137 file 
##  9 sampleA_tool1_table2.tsv.gz           101 file 
## 10 sampleA_tool1_table3.tsv.gz           112 file 
## 11 sampleA_tool1_table4.tsv.gz           121 file 
## 12 sampleA_tool1_table5.tsv.gz           232 file 
## 13 sampleA_tool1_table6.tsv.gz           142 file

Metadata

After each Workflow$write() or Workflow$run() call (for non-db formats), a metadata.parquet file is written directly to the output directory alongside the tidy tables. It is a single-row tibble capturing the run context: IDs, input/output paths, package versions, and the list of files written.

meta <- arrow::read_parquet(file.path(outdir3, "metadata.parquet"))
meta |> dplyr::glimpse()
## Rows: 1
## Columns: 6
## $ input_id     <chr> "wf1_run1"
## $ output_id    <chr> "out1"
## $ input_dirs   <list<character>> "/home/runner/miniconda3/envs/pkgdown_env/lib/R/library/nemo/extdata/tool1"…
## $ output_dir   <chr> "/tmp/RtmpZKCdcB/w1"
## $ pkg_versions <list<
##   tbl_df<
##     name   : character
##     version: character
##   >
## >> [<tbl_df[1 x 2]>]
## $ files        <list<
##   tbl_df<
##     tbl   : character
##     prefix: character
##     fout  : character
##     fin   : character
##   >
## >> [<tbl_df[12 x 4]>]…

The nested list columns can be unpacked with tidyr::unnest():

meta |> dplyr::select("pkg_versions") |> tidyr::unnest("pkg_versions")
## # A tibble: 1 × 2
##   name  version
##   <chr> <chr>  
## 1 nemo  0.1.0
meta |> dplyr::select("files") |> tidyr::unnest("files")
## # A tibble: 12 × 4
##    tbl          prefix    fout                          fin                                         
##    <chr>        <chr>     <chr>                         <chr>                                       
##  1 tool1_table1 sampleA   sampleA_tool1_table1.tsv.gz   /home/runner/miniconda3/envs/pkgdown_env/li…
##  2 tool1_table1 sampleA_2 sampleA_2_tool1_table1.tsv.gz /home/runner/miniconda3/envs/pkgdown_env/li…
##  3 tool1_table1 sampleA_3 sampleA_3_tool1_table1.tsv.gz /home/runner/miniconda3/envs/pkgdown_env/li…
##  4 tool1_table2 sampleA   sampleA_tool1_table2.tsv.gz   /home/runner/miniconda3/envs/pkgdown_env/li…
##  5 tool1_table2 sampleA_2 sampleA_2_tool1_table2.tsv.gz /home/runner/miniconda3/envs/pkgdown_env/li…
##  6 tool1_table3 sampleA   sampleA_tool1_table3.tsv.gz   /home/runner/miniconda3/envs/pkgdown_env/li…
##  7 tool1_table3 sampleA_2 sampleA_2_tool1_table3.tsv.gz /home/runner/miniconda3/envs/pkgdown_env/li…
##  8 tool1_table4 sampleA   sampleA_tool1_table4.tsv.gz   /home/runner/miniconda3/envs/pkgdown_env/li…
##  9 tool1_table4 sampleA_2 sampleA_2_tool1_table4.tsv.gz /home/runner/miniconda3/envs/pkgdown_env/li…
## 10 tool1_table6 sampleA   sampleA_tool1_table6.tsv.gz   /home/runner/miniconda3/envs/pkgdown_env/li…
## 11 tool1_table6 sampleA_2 sampleA_2_tool1_table6.tsv.gz /home/runner/miniconda3/envs/pkgdown_env/li…
## 12 tool1_table5 sampleA   sampleA_tool1_table5.tsv.gz   /home/runner/miniconda3/envs/pkgdown_env/li…